Context
cal2_erag.py resolves its default golden set as os.path.join(HERE, "example_golden.jsonl") in main() (03-evals/src/cal2_erag.py:75), where HERE is the script's own directory, 03-evals/src/. The golden file actually ships at 03-evals/example_golden.jsonl — one directory up. Run the script with no arguments today and it raises FileNotFoundError instead of scoring anything. This is a known, already-documented gap (docs/EVALS_AND_TRUST.md:138 calls it out explicitly) — it just hasn't been fixed yet.
What to change
Point the default golden-set path at 03-evals/example_golden.jsonl instead of 03-evals/src/example_golden.jsonl. The file already resolves one other path the same way one directory up (03-evals/src/cal2_erag.py:20-21, used to reach 01-context/src for imports) — follow that same repo-relative style rather than introducing a new path-resolution mechanism. There's no existing CLI/env override for the golden path today, so this is purely a default-path fix, not new configuration surface.
Acceptance
From repo root, after the Quickstart pipeline (README.md Quickstart section — Neo4j up, constraints/indexes applied, etl.py + demo_seed.py run):
python 03-evals/src/cal2_erag.py | tee /tmp/cal2.log
grep -q CAL2_OK /tmp/cal2.log
Before the fix this fails with FileNotFoundError before it ever gets to scoring. After the fix it should read the golden set and print CAL2_OK (or CAL2_FAIL: [...] with a real scoring reason — either is fine, the point is it no longer dies on the path).
Pointers
03-evals/src/cal2_erag.py — the file to fix (see main(), and the HERE/sys.path pattern at the top for the resolve style already in use)03-evals/example_golden.jsonl — where the golden set actually livesdocs/EVALS_AND_TRUST.md:138 — the existing "Known gap" note; update or remove it once this landsCONTRIBUTING.md — read before opening a PR (issue-before-PR, CLA, determinism-check convention)
Size: S — one path expression changes; no logic, no new config.
Comment here to claim this before opening a PR (see CONTRIBUTING.md — issue-before-PR). CLA: one-time signature via the automated check on your first PR.
Context
cal2_erag.pyresolves its default golden set asos.path.join(HERE, "example_golden.jsonl")inmain()(03-evals/src/cal2_erag.py:75), whereHEREis the script's own directory,03-evals/src/. The golden file actually ships at03-evals/example_golden.jsonl— one directory up. Run the script with no arguments today and it raisesFileNotFoundErrorinstead of scoring anything. This is a known, already-documented gap (docs/EVALS_AND_TRUST.md:138calls it out explicitly) — it just hasn't been fixed yet.What to change
Point the default golden-set path at
03-evals/example_golden.jsonlinstead of03-evals/src/example_golden.jsonl. The file already resolves one other path the same way one directory up (03-evals/src/cal2_erag.py:20-21, used to reach01-context/srcfor imports) — follow that same repo-relative style rather than introducing a new path-resolution mechanism. There's no existing CLI/env override for the golden path today, so this is purely a default-path fix, not new configuration surface.Acceptance
From repo root, after the Quickstart pipeline (
README.mdQuickstart section — Neo4j up, constraints/indexes applied,etl.py+demo_seed.pyrun):Before the fix this fails with
FileNotFoundErrorbefore it ever gets to scoring. After the fix it should read the golden set and printCAL2_OK(orCAL2_FAIL: [...]with a real scoring reason — either is fine, the point is it no longer dies on the path).Pointers
03-evals/src/cal2_erag.py— the file to fix (seemain(), and theHERE/sys.pathpattern at the top for the resolve style already in use)03-evals/example_golden.jsonl— where the golden set actually livesdocs/EVALS_AND_TRUST.md:138— the existing "Known gap" note; update or remove it once this landsCONTRIBUTING.md— read before opening a PR (issue-before-PR, CLA, determinism-check convention)Size: S — one path expression changes; no logic, no new config.
Comment here to claim this before opening a PR (see CONTRIBUTING.md — issue-before-PR). CLA: one-time signature via the automated check on your first PR.