Skip to content

Add strict proof-completion evaluator contract - #89

Merged
Qian-Cheng-nju merged 2 commits into
specula-org:mainfrom
munimthahmid:feat/proof-completion-evaluator
Jul 30, 2026
Merged

Add strict proof-completion evaluator contract#89
Qian-Cheng-nju merged 2 commits into
specula-org:mainfrom
munimthahmid:feat/proof-completion-evaluator

Conversation

@munimthahmid

Copy link
Copy Markdown
Contributor

What changed

  • Reuse the manifest and proof-marker code from proof-from-scratch instead of adding a second copy.
  • For marked tasks, use manifest.json to give the agent exactly the supporting files listed for that task and keep them read-only. The agent can edit only the marked proof, and the checker rejects any other changes before grading.
  • Add separate prompts for regular and one-shot agents so both follow the new proof boundary.
  • Keep the current proof-completion tasks working even though they do not have markers or a manifest yet.

Scope

This PR only adds evaluator and checker support. We can keep the legacy fallback and prompts for now so the current dataset still works. After the generator is merged and we regenerate the dataset with markers and manifest.json, we can remove them.

Testing

  • All 1,080 tests pass, including regressions for direct marked-task checks and legacy benchmark dependencies.
  • The rebuilt standalone checker passes the real proof-completion and proof-from-scratch boundary tests.
  • Three live proof-completion tasks pass with gpt-5.6-sol at medium reasoning.

@munimthahmid

Copy link
Copy Markdown
Contributor Author

@Qian-Cheng-nju can you review the PR?

@Qian-Cheng-nju

Copy link
Copy Markdown
Member

LGTM! Thanks a lot!

@Qian-Cheng-nju
Qian-Cheng-nju merged commit c3bb3b3 into specula-org:main Jul 30, 2026
1 check passed
@munimthahmid
munimthahmid deleted the feat/proof-completion-evaluator branch July 30, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants