Skip to content

Add reliable Codex and Pi usage telemetry - #70

Merged
Qian-Cheng-nju merged 4 commits into
specula-org:mainfrom
munimthahmid:feat/codex-pi-usage-telemetry
Jul 29, 2026
Merged

Add reliable Codex and Pi usage telemetry#70
Qian-Cheng-nju merged 4 commits into
specula-org:mainfrom
munimthahmid:feat/codex-pi-usage-telemetry

Conversation

@munimthahmid

@munimthahmid munimthahmid commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Add provider-native usage telemetry for Codex turn.completed events and Pi message_end events, normalized into the structured usage schema while preserving the existing top-level token fields.
  • Keep Codex child agents enabled and aggregate their native rollout token deltas without double-counting inherited parent usage.
  • Aggregate usage across retries and continuation rounds, while retaining trustworthy lower bounds for incomplete or damaged streams.
  • Prevent retries when a request may already have reached the provider, and retain Pi's native per-request metadata and estimated USD costs.
  • Pin the validated Codex 0.144.6, Pi 0.80.10, and Kiro provider 0.8.1 versions.

Testing

  • uv run pytest -q (889 passed, 3 skipped)
  • uv run ruff check .
  • uv run ruff format --check .
  • docker buildx build --check --file docker/base.Dockerfile .
  • Verified live requests through Codex and Pi, including a gpt-5.6-sol high-reasoning Codex session with a real child agent and complete aggregated usage.

@munimthahmid
munimthahmid marked this pull request as ready for review July 21, 2026 04:22
@munimthahmid
munimthahmid force-pushed the feat/codex-pi-usage-telemetry branch from e5dc252 to 9fe1871 Compare July 21, 2026 14:11
@munimthahmid

munimthahmid commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

@Saadmrp1038, if possible, can you review this PR as well? Thanks a lot.

@Qian-Cheng-nju

Copy link
Copy Markdown
Member

@sukanya1426 Could you review that? I think Pial is very busy recently

@sukanya1426

Copy link
Copy Markdown
Contributor

@Qian-Cheng-nju sure!

@sukanya1426 sukanya1426 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, could you resolve the merge conflicts ?

Comment thread src/evaluator/runner.py Outdated
Comment thread src/evaluator/usage.py Outdated
@munimthahmid
munimthahmid force-pushed the feat/codex-pi-usage-telemetry branch from 9fe1871 to 9771ee4 Compare July 29, 2026 08:30
@munimthahmid
munimthahmid requested a review from sukanya1426 July 29, 2026 08:43
@Qian-Cheng-nju
Qian-Cheng-nju self-requested a review July 29, 2026 09:11
@Qian-Cheng-nju
Qian-Cheng-nju merged commit 3aa080c into specula-org:main Jul 29, 2026
1 check passed
@Qian-Cheng-nju

Copy link
Copy Markdown
Member

Fixed a small issue, thanks! LGTM

@munimthahmid
munimthahmid deleted the feat/codex-pi-usage-telemetry branch July 29, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants