fix(eve): skip span writes after the span has ended - #1233
Open
NicoMoli wants to merge 1 commit into
Open
Conversation
Signed-off-by: Nicolas Molina <n_m09@hotmail.com>
Contributor
|
@NicoMoli is attempting to deploy a commit to the Vercel Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #776.
On long-running self-hosted (non-Vercel) deployments, every turn that makes tool calls emits repeated
Operation attempted on ended Spanerrors at error level. It is non-fatal — the turn finishes and replies send — but it fires several times per turn, buries real errors, and trips log-based alerting.Root cause: eve writes to an OpenTelemetry span after it has been
.end()-ed.harness/tool-loop.tscreates theai.eve.turnspan, installs it as the active span viaotelContext.with(...), and ends it in afinallywhile it is still active.internal/logging.tswrites to the active span on every error log with no guard. Tool-call turns widen the async window (model → tool → model), so an error log emitted on a late microtask continuation can resolve after thefinallyended the span.Fix: guard both span-mutating helpers in
internal/logging.tswithisRecording(), which the OTel spec defines as returningfalseonce a span has ended.recordErrorOnSpan— early return when the span is not recording. This is the load-bearing guard; the function is exported and called directly by the harness, so it must be self-protecting.recordOnActiveSpan— tightened fromspan === undefinedto also check!span.isRecording(), short-circuiting all three mutation branches (setStatus/recordException/addEvent) before any write.isRecordingwas also added to the vendored@opentelemetry/apiSpandeclaration, which did not previously expose it.No change to
harness/tool-loop.ts: thesetAttributethere is synchronous inside the awaited body and provably runs beforeend(), and the directrecordErrorOnSpan(turnSpan, ...)call sits inside the model-calltrywhere the span is still recording, so the guard is a no-op there.Behavior while a span is live is unchanged — this only drops writes that would otherwise be errors.
How did you test your changes?
Added a
span recording guardblock to the existing unit-tiersrc/internal/logging.test.ts. It mocks#compiled/@opentelemetry/api/index.js(importOriginal, realSpanStatusCode, overriddentrace.getActiveSpan) in the style ofexecution/subagent-usage-span.test.ts, and uses amakeFakeSpan(isRecording)factory whose mutators throw when not recording — mirroring the SDK — so any unguarded write fails the test loudly. Cases cover: ended active span (no throw, no mutation), ended span passed directly torecordErrorOnSpan, and three no-regression cases on a recording span.PR Checklist
CONTRIBUTING.mdevepackagegit commit --signoff)🤖 Generated with Claude Code
https://claude.ai/code/session_013RApsUy4qVAeVCWwmCph2R