lake
Browse this documentation section

O(1) external manifest latest pointer

Issue: #41

Outcome

Opening a dataset reads one fixed latest key instead of scanning immutable manifest history. Existing deployments migrate lazily, and exact historical version reads remain available.

Protocol

  1. Latest is a fixed {version, path} record and the atomic claim for the current version.
  2. To claim the next version, CAS-create an immutable archive of current latest (or verify the existing identical archive), then CAS latest from exact old bytes to the new staging pointer.
  3. Lance writes the staging manifest before calling this protocol, preserving manifest-before-pointer. put_if_exists exact-CASes staging to final.
  4. Explicit reads first check immutable history, then exact-match fixed latest.
  5. A missing fixed pointer triggers one legacy max scan and CAS migration.
  6. Delete fences fixed latest as deleting, removes immutable history, and leaves a durable deleted marker that recreate replaces. This prevents stale legacy migration from reviving a pointer through an ABA None CAS.
  7. Every history create is atomically guarded by the exact latest bytes read by the writer, preventing pre-fence writers from publishing after drop.
  8. Latest pointers and delete markers carry an incarnation preserved within a dataset lifetime and regenerated by recreate, eliminating cross-cycle ABA.
  9. History records carry that incarnation too; legacy path-only history is upgraded lazily, and finalize convergence compares path plus incarnation.

Rollout

Pre-#41 binaries write only per-version records and therefore cannot coexist as committers after a fixed pointer becomes authoritative. Drain writes, upgrade every metadata node that may become leader, and only then resume commits. This is a protocol rollout fence, not a data migration window: existing datasets remain readable and install their pointer lazily on first open.

Verification