Prometheus runtime metrics implementation plan
Goal: Make Query and Metasrv saturation, authority, and background health measurable through a bounded, privately scraped Prometheus surface.
Architecture: Domain crates emit low-cardinality metrics through the
metrics facade. The CLI owns one recorder, a loopback-only Axum endpoint, and
an upkeep task under the same cancellation lifecycle as the selected server.
Task 1: Establish the owned exporter
- Add failing configuration and endpoint lifecycle tests.
- Parse optional
LAKE_METRICS_ADDRand reject non-loopback/hostname values. - Install the recorder, bind before service startup, and own HTTP/upkeep tasks.
- Refactor CLI server shutdown to cancel and join metrics on every exit path.
Task 2: Instrument Query
- Wrap admission permits so the inflight gauge follows the real permit lifetime, including streaming responses.
- Count admitted, saturated, shutting-down, and SQL-size rejection outcomes.
- Count initial/background catalog refresh results and publish readiness.
- Test rendered exposition with a task-local recorder.
Task 3: Instrument Metasrv
- Wrap append permits and reserved-byte accounting through commit/forwarding.
- Count bounded campaign outcomes and publish exact write readiness.
- Count bounded maintenance pages/items after each stage.
- Test rendered exposition without any user- or data-derived labels.
Task 4: Document and ship
- Document enablement, sidecar collection, metric names, and label policy.
- Run guarded lifecycle, strict clippy, gate, ship, independent review, and independent verification.
- Merge one reviewed PR and verify main.