OTLP distributed tracing implementation plan
Goal: Follow one Flight operation across Lake tiers while preserving bounded telemetry cardinality and process-owned shutdown.
Task 1: Lock the metadata contract
- Add W3C trace-context inject/extract helpers for tonic metadata.
- Prove only
traceparentandtracestateround-trip. - Reject malformed remote context without weakening authentication.
Task 2: Own exporter lifecycle in the CLI
- Compose JSON/pretty logging with an optional OpenTelemetry layer.
- Build a bounded OTLP trace batch exporter from validated environment.
- Flush and shut down inside a finite process deadline.
Task 3: Instrument tier boundaries
- Create bounded server spans for public Query and Metasrv Flight paths.
- Inject current context into Query-to-Metasrv requests.
- Preserve the same context through follower-to-leader forwarding.
- Inject context from the Rust SDK without changing Flight payloads.
Task 4: Operate and verify
- Document collector configuration, service names, sampling, redaction, and failure behavior.
- Wire optional collector endpoint/service-name values in the Kubernetes reference without exposing collector credentials.
- Run the guarded spec lifecycle, strict clippy, full gate, integration, independent review, and independent verification before merge.