gRPC health readiness implementation plan
Goal: Expose scheduler-usable liveness and real traffic readiness without bypassing the existing Flight security boundary.
Architecture: Use the standard gRPC health service on each tier’s existing Tonic server. Query readiness follows strict catalog warmup. Metasrv readiness is derived from the same local lease/remote-forward route used by writes and is woken by leadership state changes plus the exact local deadline.
Task 1: Add failing protocol tests
- Add secured Query health Check/Watch tests for auth, serving after warmup, and NOT_SERVING before shutdown drain.
- Add Metasrv route-state tests covering no leader, remote leader, live local lease, and expired remembered-self state.
- Add Metasrv Watch/shutdown ownership coverage.
- Confirm RED because neither server registers the health protocol.
Task 2: Wire Query health
- Register
tonic-healthbehind the existing server interceptor. - Mark overall and Flight services serving after strict refresh.
- Withdraw both statuses before server cancellation.
Task 3: Wire Metasrv route-aware health
- Add leadership change subscription and exact local-expiry inspection.
- Run an owned readiness monitor that publishes Flight status.
- Withdraw health before drain and join the monitor with maintenance and campaign under the total deadline.
Task 4: Document and verify
- Document service names, auth/TLS requirements, readiness semantics, and shutdown ordering.
- Run spec lifecycle, strict clippy, full gate, independent verification, and independent review.
- Ship one PR and merge after approval.