Skip to content

Fix perf gate crash when report uses nested metrics schema#69

Merged
Navy10021 merged 1 commit into
mainfrom
codex/fix-typeerror-in-check_perf_gate.py
Mar 6, 2026
Merged

Fix perf gate crash when report uses nested metrics schema#69
Navy10021 merged 1 commit into
mainfrom
codex/fix-typeerror-in-check_perf_gate.py

Conversation

@Navy10021

Copy link
Copy Markdown
Owner

Motivation

  • The perf gate script crashed with a TypeError when the evaluation report used the newer JSON schema that nests metrics under metrics, because the script expected flat top-level fields and tried to format None values.
  • The goal is to make scripts/check_perf_gate.py robust to both the v1 nested schema and legacy flat schema and to avoid formatting errors for missing metrics.

Description

  • Change _extract_metrics to read from report.get("metrics", report) so the v1 nested metrics block is used when present while falling back to legacy flat keys.
  • Add a small helper _fmt_metric(value, spec) that returns "N/A" for None and otherwise formats the value, and use it for verbose output to prevent NoneType formatting errors.
  • Update verbose printing to use _fmt_metric for blue_win_rate and cvar_10, while preserving gate logic that treats missing metrics as failures with user-friendly messages.

Testing

  • Ran python scripts/check_perf_gate.py /tmp/perf_gate_sample.json --min-win-rate 0.45 --max-cvar10 40.0 with a v1-style report and observed normal evaluation output and expected gate failure due to thresholds (script ran without crashing).
  • Ran python scripts/check_perf_gate.py /tmp/perf_gate_empty.json --min-win-rate 0.45 --max-cvar10 40.0 with a report missing metrics and observed N/A printed and clean failure messages instead of an exception.
  • Ran the end-to-end flow python evaluate.py --fast --no-progress --seed 42 --output-json /tmp/perf_gate.json followed by python scripts/check_perf_gate.py /tmp/perf_gate.json --min-win-rate 0.45 --max-cvar10 40.0 and confirmed the perf gate runs to completion (no TypeError) and reports gate result (failed on CVaR threshold)

Codex Task

@Navy10021
Navy10021 merged commit cb1372f into main Mar 6, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant