Correlate Rust logger lines to the active OTel trace + tee to tracing#173
Open
brentrager wants to merge 1 commit into
Open
Correlate Rust logger lines to the active OTel trace + tee to tracing#173brentrager wants to merge 1 commit into
brentrager wants to merge 1 commit into
Conversation
Two-part correlation fix (th-de3805) so logger lines join to the traces they belong to and reach the @smooai/observability OTLP logs pipeline. 1. Real trace/span ids. The logger seeded `traceId` from a fabricated uuid and never carried a span id. Now, at log-build time, when an OpenTelemetry span is active it overrides `traceId` with the span's real W3C trace_id and adds `spanId`; with no span active it falls back to the seeded uuid (prior behavior). Reads the OpenTelemetry **API only** (`opentelemetry`, trace feature) — never `@smooai/observability`, which would be circular. 2. Tee to the tracing facade. `emit` writes stdout-direct, so lines never reached the observability appender (a `tracing_subscriber::Layer`). Each line is now also emitted as a `tracing` event at the matching level (message → body, full payload → `log` field). The event fires while the same OTel span is active, so the appender's SDK logger stamps the record's trace_id/span_id from the current context — matching the ids already in the payload. Gated by `otel_bridge`, defaulting to the SAME env enablement as the observability logs pipeline (`SMOOAI_OBSERVABILITY_ENDPOINT` set and not disabled); off = stdout-direct with zero tracing overhead. Tests (tests/otel_correlation.rs): a log within an active span carries the span's real trace_id/span_id (asserted against the real ids — mutating the expected id fails the test); no span → uuid fallback and no spanId; enabled bridge tees one event at the matching level with the message as body; disabled bridge tees nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2bM94GAnVjYSSv1x7HKRB
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Rust logger fabricated a uuid for
traceIdand carried no span id, so its lines couldn't be joined to the OpenTelemetry traces they belong to. Andemitwrites stdout-direct, so lines never reached@smooai/observability's OTLP logs appender (atracing_subscriber::Layer) — even once that pipeline exists. Correlation fix, pearl th-de3805.Solution
Two halves, depending on the OpenTelemetry API only (
opentelemetry,tracefeature) — never@smooai/observability(that would be circular):Real trace/span ids. At log-build time, when an OTel span is active, override
traceIdwith the span's real W3C trace_id and attachspanId(newspanIdcontext key). No active span → the seeded uuid stays (prior behavior). Newcontext::active_otel_trace_context()readsopentelemetry::Context::current().Tee to the
tracingfacade. Each emitted line is also fired as atracingevent at the matching level (message → body, full JSON payload →logfield), so the observability appender turns it into an OTLP log record. The event fires while the same OTel span is still active, so the appender's SDK logger stamps the record's trace_id/span_id from the current context — matching the ids this crate already put in the payload. Gated by a newotel_bridgeoption that defaults to the same env enablement as the observability logs pipeline (SMOOAI_OBSERVABILITY_ENDPOINTset andSMOOAI_OBSERVABILITY_DISABLEDnot truthy). Off →emitstays stdout-direct with zerotracingoverhead.Verification
cargo build,cargo clippy --all-targets(clean),cargo fmt --check(clean),cargo test— 20 passed (16 existing + 4 new). Newtests/otel_correlation.rs:log_within_active_span_carries_real_trace_and_span_id— assertstraceId/spanIdequal the active span's real ids. Actually verified, not just compiled: mutating the expected trace_id fails the test (real captured id595170f2…vs the zeroed value).log_without_active_span_falls_back_to_uuid— no span → uuidtraceId, nospanId.enabled_bridge_tees_line_to_tracing— one line → onetracingevent at the matching level with the message as body (captured via a test subscriber layer).disabled_bridge_does_not_tee— off → notracingevents.Branched off
main(the repo's default work branch is a feature branch). No deploy. End-to-end join through the real observability appender was not exercised here — that lives in@smooai/observability(companion PR adds the logs signal) and the follow-up dogfood wiring.🤖 Generated with Claude Code