lake architecture

Choose native Lake tables for managed writes and large objects; federate external Iceberg tables for read-only exact-snapshot SQL.

Lake control and object-data paths SDKs and clients use Query replicas for SQL. Query caches Lake metadata from Metasrv and reads immutable data directly from object storage. Large objects go directly from the SDK to its managed stage. A separate read-only path resolves Iceberg snapshots at an external REST catalog and reads its Parquet files directly. Flight SQL FILE metadata only large video / model bytes — direct, streaming direct scans cache miss / refresh commit pointer registry / operations exact read-only catalog / snapshot lookup direct Parquet + manifest scan Lake query / metadata deployment Object storage and external authority Fleet / clients Flight SQL readers Rust SDK FILE upload / direct read Query replicas DataFusion + Flight SQL stateless / catalog cache Metasrv registry + write coordination bounded / leader-elected DynamoDB / RocksDB compact durable authority Lake datasets immutable Lance snapshots object-storage backed manifest authority engine-private metadata Iceberg REST catalog external snapshot authority Iceberg table files external immutable Parquet data managed object stage SDK-owned video / model bytes Legend Lake service durable metadata object storage read-only Iceberg path direct large-object bytes Source: docs/architecture.md · Iceberg remains an external read-only authority.

Read fan-out

  • • Query replicas scale horizontally.
  • • Catalog cache keeps routine reads off Metasrv.
  • • Queries read immutable snapshots directly.

Large objects

  • • FILE rows store metadata, not video/model bytes.
  • • SDK uploads and reads directly from its managed stage.
  • • Query and Metasrv never become data proxies.

External Iceberg

  • • `iceberg` is a separate read-only SQL catalog.
  • • The external catalog owns Iceberg snapshots and commits.
  • • Lake registry and Iceberg metadata never merge.