Skip to content
lake
Browse this documentation section

Query memory and spill budgets

Issue: #39

Outcome

Every Query replica builds DataFusion with a finite, process-wide execution memory pool and a finite spill manager rooted in an operator-selected local directory. Large spilling operators degrade to disk or a typed resource error instead of relying on the process OOM killer.

Implementation

  1. Add QueryResources with validated memory bytes, spill bytes, and spill root. Build a RuntimeEnv with FairSpillPool and DiskManagerBuilder.
  2. Add a fallible QueryEngine constructor that injects the runtime into SessionContext; retain bounded defaults for library callers.
  3. Parse LAKE_QUERY_MEMORY_BYTES, LAKE_QUERY_SPILL_BYTES, and LAKE_QUERY_SPILL_DIR at the CLI boundary before starting the server.
  4. Add focused runtime, spill/cleanup, and CLI validation tests.
  5. Backport an atomic spill-quota reservation fix for DataFusion 53.1 so a rejected write never leaks global disk accounting; cover recovery on the same shared runtime.
  6. Document sizing, local disk ownership, and the distinction between execution memory and streamed result backpressure.

Verification