@studnicky/geo-artifact-codec

Zero-dependency generic binary codec primitives (LEB128, RLE, bit-packing, field interning, geohash-cell RLE tables, multi-level override trees) for building geohash-indexed lookup artifacts.

This package provides the low-level binary building blocks for encoding and decoding compact, geohash-indexed lookup artifacts. Each primitive is independent and general-purpose — LEB128 varints, run-length encoding, fixed-width bit-packing, and string-field interning — plus two composite artifact formats (GAC1 cell tables and GACO override trees) built from those primitives for storing per-cell tuple lookups at scale.

Install

pnpm add @studnicky/geo-artifact-codec

Requires @studnicky:registry=https://npm.pkg.github.com in .npmrc.

LEB128 varint round-trip

Leb128Codec encodes, sizes, and decodes variable-length integers. Try it live — edit the code and press Execute:

LEB128 varint encode/decode round-trip

/** leb128Roundtrip — demonstrates Leb128Codec encode, decode, and size. Run: npx tsx examples/leb128Roundtrip.ts */

import assert from 'node:assert/strict';

// #region usage
import { Leb128Codec } from '../src/index.js';

function demonstrateLeb128Roundtrip(): { 'decoded': ReturnType<typeof Leb128Codec.decode>; 'encoded': Buffer } {
  // encode a value into a compact varint buffer
  const encoded = Leb128Codec.encode(300);
  console.log('encoded bytes:', encoded);

  // size predicts the encoded length without allocating
  console.log('predicted size:', Leb128Codec.size(300));

  // decode reads a varint back out starting at a given offset
  const decoded = Leb128Codec.decode(encoded, 0);
  console.log('decoded value:', decoded.value);
  console.log('bytes read:', decoded.bytesRead);

  return { 'decoded': decoded, 'encoded': encoded };
}
// #endregion usage

const result = demonstrateLeb128Roundtrip();

assert.equal(result.decoded.value, 300);
assert.equal(result.decoded.bytesRead, result.encoded.length);
assert.equal(Leb128Codec.size(300), result.encoded.length);

console.log('leb128Roundtrip: all assertions passed');

Output

Press Execute to run this example against the real library.

Bit-packed round-trip

BitPackedWriter/BitPackedReader pack and unpack fixed-width entries into a contiguous bitstream:

import { BitPackedReader, BitPackedWriter } from '../src/index.js';

function demonstrateBitPackedRoundtrip(): { 'packed': Buffer; 'unpacked': number[]; 'values': number[] } {
  const values = [1, 5, 12, 0, 9];
  const bitsPerValue = 4;

  // pack fixed-width entries into a contiguous bitstream
  const packed = BitPackedWriter.pack(values, bitsPerValue);
  console.log('packed bytes:', packed);

  // unpack reads the same entries back given the byte offset, count, and width
  const unpacked = BitPackedReader.unpack(packed, 0, values.length, bitsPerValue);
  console.log('unpacked values:', unpacked);

  return { 'packed': packed, 'unpacked': unpacked, 'values': values };
}

Cell table round-trip

CellTableWriter/CellTableReader RLE-compress and bit-pack per-cell tuple indices into a GAC1 artifact, including mixed cells marked null:

import type { CellTableType } from '../src/index.js';

import { CellTableReader, CellTableWriter } from '../src/index.js';

function demonstrateCellTableRoundtrip(): { 'artifact': Buffer; 'decoded': CellTableType; 'expected': number[]; 'tupleCount': number } {
  // per-cell tuple indices, in cell-index order; null marks a mixed cell
  const cellTupleIndices = [0, 0, 0, 1, 1, null, 2];
  const tupleCount = 3;

  // write RLE-compresses and bit-packs the indices into a GAC1 artifact
  const artifact = CellTableWriter.write({ 'cellTupleIndices': cellTupleIndices, 'tupleCount': tupleCount });
  console.log('artifact byte length:', artifact.length);

  // read decodes the artifact back into a dense per-cell tuple index array
  const decoded = CellTableReader.read(artifact);
  console.log('decoded tuple indices:', Array.from(decoded.tupleIndices));
  console.log('mixed-cell sentinel:', decoded.mixedSentinel);

  const expected = cellTupleIndices.map((tupleIndex) => { return tupleIndex ?? decoded.mixedSentinel; });

  return { 'artifact': artifact, 'decoded': decoded, 'expected': expected, 'tupleCount': tupleCount };
}

Source on GitHub