Add layered proof-completion task generation - #92
Conversation
|
@Qian-Cheng-nju , Hi, can you please review this pr? |
|
I'm reviewing that. @munimthahmid Since you have implemented evaluator, could you also help with reviewing this? |
munimthahmid
left a comment
There was a problem hiding this comment.
@sukanya1426 Thanks for the work on this. I found a few issues during testing and left inline comments. Could you please check and fix them?
| dropped = [] | ||
| if run_gates: | ||
| sm.layered_sany_gate(output_root, manifest, audit_writer) | ||
| dropped = sm.layered_triviality_gate(output_root, manifest, audit_writer) |
There was a problem hiding this comment.
I ran full generation and got 474 tasks instead of the existing 473. The extra Data/SequencesTheorems_AppendDef.tla task passed unchanged with PROOF OBVIOUS. The triviality gate treats a tlapm timeout as if the task needs a proof. We should retry or stop generation instead of keeping it.
| os.makedirs(out_dir, exist_ok=True) | ||
| written = audit_state.setdefault("written", {}) | ||
|
|
||
| model_set, _main_specs = sm.compute_model_set(dump, targets) |
There was a problem hiding this comment.
The shared model is built from all targets, so it moves some definitions and assumptions that appear later in the source ahead of earlier theorems. I found 52 tasks with this scope change. For example, Voting_QuorumNonEmpty can now use Ballot, even though Ballot is defined after it. BY QuorumAssumption DEF Ballot fails in the old task with Ballot not found, but passes in the layered task. We should preserve each target’s original scope and only include model content available before that theorem.
|
|
||
| dropped = [] | ||
| if run_gates: | ||
| sm.layered_sany_gate(output_root, manifest, audit_writer) |
There was a problem hiding this comment.
Generation still reports success when a source or SANY check fails. I tested an invalid task: the gate reported 1/1 FAILED, but finalization kept it in manifest.json. I also passed a nonexistent source file; generation logged the SANY error, exited 0, and wrote an empty manifest. We should keep the audit output, but return nonzero and not write the final manifest when generation has errors.
What changed
<base>Model.tlafor the model, assumptions, spec, and fairness.<task>Scaffold.tlafor the provided definitions and earlier lemmas.<task>.tlafor the target theorem and editable proof region.Only
<task>.tlais submitted, and only its proof region can be edited.manifest.jsonso every task carries its exact context. This reuses the existing marker and manifest logic from proof-from-scratch instead of duplicating it. Every generated task is validated before the manifest is written.--layered(473 tasks).VARIABLESdeclarations shadowing theorem variables.USE/HIDEstatements now live in the task file, since they are not inherited throughEXTENDS.parse_theoremsnow handles standalonePROOF OBVIOUS.scripts/dataset_table.pynow counts tasks from the manifest instead of counting scaffold files.Scope
This PR covers the generator and regenerated dataset only, the evaluator changes are in #89. Now that every task has a manifest, the legacy fallback and prompts in #89 are no longer needed for this suite. It also adds
source/OpenAddressing/OpenAddressing_proof.tla. The task was already in the dataset, but its source module was missing fromsource/, so regeneration would have removed it. This restores the missing source.Testing
PROOF OMITTED). The reference proof still passes.--filterrun only regenerates its own group and leaves the rest of the manifest unchanged.Issue
find_proof_endstill misses theorem lines that end right after theTHEOREMkeyword. This can causevalidate.pyto include the next theorem in the task and report a false "1 missing" for someSets_*andGraphTheorem_*tasks. This is a pre existing issue and is unchanged by this PR.