Add OTel metrics dual export alongside Prometheus

[shuheiktgw]

Summary

• Adds OpenTelemetry metrics instrumentation alongside the existing Prometheus metrics, enabling dual export to both backends.
• Each metric type (IntCounter, IntGauge, Gauge, Histogram, and their Vec variants) now records to both Prometheus and an OTel instrument.
• OTel instrument creation is lazy/on-demand: instruments are not built at metric registration time. Instead, an OtelState holds a builder closure and a OnceLock. The instrument is created on first use after install_otel_meter() has been called. This avoids ordering issues at startup (metrics are registered as LazyLock statics, often before the OTel meter provider is configured).

Known limitation: IntGauge relative operations

OTel gauges only support absolute record(value) — there is no add/sub API. For relative operations (inc, dec, add, sub), the implementation reads the Prometheus gauge value after mutation and records that to OTel. This is not atomic: under concurrent updates the OTel side may briefly observe a stale value. This is acceptable for now because gauges are inherently point-in-time approximations and the next update self-corrects. A TODO is left in the code to manage a single AtomicI64 as the source of truth if strict correctness is needed later.

🤖 Generated with Claude Code

[shuheiktgw]

I also pushed a branch, refactor/otel-metrics-dual-export-resolved, where I experimented with solving the IntGauge issue described above by using a callback. The idea was to store the gauge value within the struct and let the exporter scrape it via a callback each time metrics are exported, avoiding race conditions.

This was still a very early-stage POC, and I’ve put it aside for now since it started to feel too complex—but sharing it for context in case we want to revisit this approach.