lake · Iceberg federation

Read-only external metadata federation: synchronous and durable Flight paths share one exact snapshot rule; the Iceberg catalog remains the metadata and commit authority.

Lake read-only Iceberg federation topology A Flight client is admitted by Query using its namespace grant. Query resolves and caches an exact Iceberg table snapshot through the external REST catalog. Interactive DoGet and durable PollFlightInfo both carry that encrypted snapshot identity; a worker point-loads the same ID, reads Iceberg files directly, and fails closed if retention removed it. Credentials remain in Query process memory, while the durable job state contains neither credentials nor source-object bytes. Lake Metasrv and the Lake registry are not part of this path. authenticated Flight SQL namespace grant exact table lookup bounded HTTPS REST interactive: direct object reads renew once after failed read PollFlightInfo seals job claim worker: retained snapshot ID worker: direct external reads manifest sealed ticket / completed endpoint Lake Query deployment — one optional Iceberg REST catalog External Iceberg authority Flight SQL client SDK / training reader principal map finite namespace grants Query replica DataFusion + Flight SQL admission + ticket encryption sync DoGet / durable submission snapshot cache 10,000 keys 5s fresh / 60s stale per-key single-flight durable async job state independent CAS store + result prefix sealed snapshot identity only no credentials / source-object bytes async worker bounded lease same snapshot or fail Iceberg REST catalog external metadata + commits no catalog/table enumeration OAuth server (optional) called only by Query Iceberg table files Parquet + manifests in object storage intentionally absent from both paths Lake Metasrv / Lake registry / user-side REST credentials no Iceberg writes, registry mirroring, or source-object proxy Legend Lake Query process durable async state external object storage authorization / credential flow durable job lifecycle Source: docs/design/iceberg-federation.md · one metadata authority per table.

SQL admission

  • • Full `iceberg.namespace.table` references only.
  • • Deployment allowlist and principal grant both apply.
  • • External discovery is never a request-path feature.

Snapshot semantics

  • • Current loads use a bounded cache and per-key single-flight.
  • • DoGet and durable workers use the same sealed snapshot ID.
  • • Expired upstream snapshots fail; they never fall forward.

Authority boundary

  • • Iceberg owns its metadata, commits, and GC.
  • • Query reads data files directly with its workload identity.
  • • Lake registry and Metasrv stay outside the federation path.