Skip to content

Refine benchmark cost and time accounting - #91

Merged
Qian-Cheng-nju merged 5 commits into
mainfrom
fix/benchmark-cost-time
Jul 31, 2026
Merged

Refine benchmark cost and time accounting#91
Qian-Cheng-nju merged 5 commits into
mainfrom
fix/benchmark-cost-time

Conversation

@Qian-Cheng-nju

@Qian-Cheng-nju Qian-Cheng-nju commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

  • record formal agent wall time and equivalent public API cost for every benchmark backend
  • use native agent telemetry where available and maintained public pricing otherwise
  • keep infra and quota attempts in separate diagnostic accounting instead of formal benchmark totals
  • report missing or incomplete cost as unavailable rather than zero

@Qian-Cheng-nju Qian-Cheng-nju changed the title Add benchmark cost and time accounting Refine benchmark cost and time accounting Jul 30, 2026
@Qian-Cheng-nju
Qian-Cheng-nju marked this pull request as ready for review July 30, 2026 14:28
@Qian-Cheng-nju
Qian-Cheng-nju marked this pull request as draft July 30, 2026 14:29
@Qian-Cheng-nju
Qian-Cheng-nju marked this pull request as ready for review July 30, 2026 16:35
Comment thread src/evaluator/runner.py

@munimthahmid munimthahmid left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Qian-Cheng-nju Reviewed the implementation thoroughly. Also I ran Codex, Claude Code, and one shot OpenAI calls on a small proof, including complete and timeout paths . Wall time, token/request accounting, fail-closed behavior, and headline equivalent costs were reported correctly.

I found no major issue. Approving.

@Qian-Cheng-nju

Copy link
Copy Markdown
Member Author

Thanks a lot! @munimthahmid Merging.

@Qian-Cheng-nju
Qian-Cheng-nju merged commit cceeb09 into main Jul 31, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants