Query memory and spill budgets
Issue: #39
Outcome
Every Query replica builds DataFusion with a finite, process-wide execution memory pool and a finite spill manager rooted in an operator-selected local directory. Large spilling operators degrade to disk or a typed resource error instead of relying on the process OOM killer.
Implementation
- Add
QueryResourceswith validated memory bytes, spill bytes, and spill root. Build aRuntimeEnvwithFairSpillPoolandDiskManagerBuilder. - Add a fallible
QueryEngineconstructor that injects the runtime intoSessionContext; retain bounded defaults for library callers. - Parse
LAKE_QUERY_MEMORY_BYTES,LAKE_QUERY_SPILL_BYTES, andLAKE_QUERY_SPILL_DIRat the CLI boundary before starting the server. - Add focused runtime, spill/cleanup, and CLI validation tests.
- Backport an atomic spill-quota reservation fix for DataFusion 53.1 so a rejected write never leaks global disk accounting; cover recovery on the same shared runtime.
- Document sizing, local disk ownership, and the distinction between execution memory and streamed result backpressure.
Verification
mise run spec-lifecycle specs/issue-39-query-memory-spill.spec.mdcargo test -p lake-query query_resources_reject_invalid_budgetscargo test -p lake-query query_engine_uses_bounded_fair_spill_runtimecargo test -p lake-query memory_intensive_sort_spills_and_cleans_upcargo test -p lake-query spill_budget_error_does_not_poison_runtimecargo test -p lake-cli query_resource_values_are_validated_before_servingcargo clippy -p lake-query -p lake-cli --all-targets -- -D warningsmise run gate