What happened
On PR #4075, the review agent correctly identified that the per-repo chain description in the new documentation was inaccurate — reusable-dispatch.yml on main had already inlined stages per ADR 62, making the documented two-hop chain (reusable-dispatch.yml -> reusable-<agent>.yml) incorrect. In fix iteration 1 (run 29900226359, Jul 22), the fix agent disagreed with this finding, stating: "The reviewer's claim is factually incorrect. reusable-dispatch.yml currently DOES call reusable-.yml via uses: references — grep for 'uses:.*reusable-' returns 6 matches." This was true on the PR branch but false on main — the PR branch was 507 commits behind main and still contained the pre-ADR-62 architecture. The fix agent verified its claim against the PR branch's stale copy of the file rather than against main.
A human reviewer (waynesun09) then posted a detailed recheck on the same day, confirming via 4 independent reviewer agents and a direct branch-divergence check that main had already inlined the stages. The fix agent's second run (same day) partially fixed the issue but still treated inlining as future work. It took a third fix run on Jul 27 — 5 days later, with explicit human instruction to "rebase and move the guidance" — before the fix agent rebased, checked main, and acknowledged its error: "Previous iterations disagreed with findings by checking the stale PR branch instead of main. This iteration rebased first, then verified all claims against current main — confirming the reviewers were correct about the inlined architecture."
What could go better
The fix agent should distinguish between claims about the PR's own code (check the PR branch) and claims about the current state of the codebase (check the base branch / main). When documentation describes existing architecture, the source of truth is main, not the PR branch — especially for docs-only PRs that don't modify the files being described.
This is related to but distinct from #4953, which proposes requiring the fix agent to read files before disagreeing. In this case, the fix agent DID read the file — but from the wrong branch. Implementing fullsend-ai#4953's proposed fix (require reading and quoting the file) would not have prevented this error, because the agent would have quoted the stale PR-branch version and still disagreed incorrectly.
Confidence: High — the fix agent's own retrospective statement confirms the root cause. The pattern is reproducible: any long-lived PR branch that doesn't include recent main changes will have stale copies of files, and the fix agent will verify claims against those stale copies.
Proposed change
Update the fix agent definition (in the agents repo, fullsend-ai/.fullsend) to add an instruction for the 'Disagreed' disposition: when evaluating a review finding that makes claims about the current state of files NOT modified by the PR (e.g., the reviewer says file X has property Y), the fix agent must verify against the base branch (main / the PR's merge target), not against the PR branch. Concretely:
-
Add to the fix agent's prompt or instructions: "When a review finding references a file that is NOT in the PR diff, verify the finding's claim by reading that file from the base branch (origin/main or the PR's base ref), not from the PR branch. The PR branch may be significantly behind main. For documentation PRs that describe current codebase behavior, the base branch is the source of truth."
-
Consider adding a pre-evaluation step to the fix agent harness: if the PR branch is more than N commits behind main (e.g., 50), log a warning that the branch is stale and that claims about codebase state should be verified against main.
This proposal targets fullsend-ai/fullsend for tracking, but implementation requires changes to the fix agent definition in the agents repo (fullsend-ai/.fullsend), consistent with where fullsend-ai#4953 is tracked.
Validation criteria
On the next 5 fix agent runs where the agent uses the 'Disagreed' disposition on a finding about files not in the PR diff: (1) the justification references the base branch version of the file, not the PR branch version, (2) zero incorrect disagreements caused by reading stale PR-branch copies of files that have been updated on main. Measurable proxy: no human corrections of 'Disagreed' claims that cite stale branch state in the next 20 fix agent runs that include disagreements.
Generated by retro agent from fullsend-ai#4075
What happened
On PR #4075, the review agent correctly identified that the per-repo chain description in the new documentation was inaccurate —
reusable-dispatch.ymlonmainhad already inlined stages per ADR 62, making the documented two-hop chain (reusable-dispatch.yml->reusable-<agent>.yml) incorrect. In fix iteration 1 (run 29900226359, Jul 22), the fix agent disagreed with this finding, stating: "The reviewer's claim is factually incorrect. reusable-dispatch.yml currently DOES call reusable-.yml via uses: references — grep for 'uses:.*reusable-' returns 6 matches." This was true on the PR branch but false onmain— the PR branch was 507 commits behindmainand still contained the pre-ADR-62 architecture. The fix agent verified its claim against the PR branch's stale copy of the file rather than againstmain.A human reviewer (waynesun09) then posted a detailed recheck on the same day, confirming via 4 independent reviewer agents and a direct branch-divergence check that
mainhad already inlined the stages. The fix agent's second run (same day) partially fixed the issue but still treated inlining as future work. It took a third fix run on Jul 27 — 5 days later, with explicit human instruction to "rebase and move the guidance" — before the fix agent rebased, checkedmain, and acknowledged its error: "Previous iterations disagreed with findings by checking the stale PR branch instead of main. This iteration rebased first, then verified all claims against current main — confirming the reviewers were correct about the inlined architecture."What could go better
The fix agent should distinguish between claims about the PR's own code (check the PR branch) and claims about the current state of the codebase (check the base branch /
main). When documentation describes existing architecture, the source of truth ismain, not the PR branch — especially for docs-only PRs that don't modify the files being described.This is related to but distinct from #4953, which proposes requiring the fix agent to read files before disagreeing. In this case, the fix agent DID read the file — but from the wrong branch. Implementing fullsend-ai#4953's proposed fix (require reading and quoting the file) would not have prevented this error, because the agent would have quoted the stale PR-branch version and still disagreed incorrectly.
Confidence: High — the fix agent's own retrospective statement confirms the root cause. The pattern is reproducible: any long-lived PR branch that doesn't include recent main changes will have stale copies of files, and the fix agent will verify claims against those stale copies.
Proposed change
Update the fix agent definition (in the agents repo,
fullsend-ai/.fullsend) to add an instruction for the 'Disagreed' disposition: when evaluating a review finding that makes claims about the current state of files NOT modified by the PR (e.g., the reviewer says file X has property Y), the fix agent must verify against the base branch (main/ the PR's merge target), not against the PR branch. Concretely:Add to the fix agent's prompt or instructions: "When a review finding references a file that is NOT in the PR diff, verify the finding's claim by reading that file from the base branch (origin/main or the PR's base ref), not from the PR branch. The PR branch may be significantly behind main. For documentation PRs that describe current codebase behavior, the base branch is the source of truth."
Consider adding a pre-evaluation step to the fix agent harness: if the PR branch is more than N commits behind main (e.g., 50), log a warning that the branch is stale and that claims about codebase state should be verified against main.
This proposal targets
fullsend-ai/fullsendfor tracking, but implementation requires changes to the fix agent definition in the agents repo (fullsend-ai/.fullsend), consistent with where fullsend-ai#4953 is tracked.Validation criteria
On the next 5 fix agent runs where the agent uses the 'Disagreed' disposition on a finding about files not in the PR diff: (1) the justification references the base branch version of the file, not the PR branch version, (2) zero incorrect disagreements caused by reading stale PR-branch copies of files that have been updated on main. Measurable proxy: no human corrections of 'Disagreed' claims that cite stale branch state in the next 20 fix agent runs that include disagreements.
Generated by retro agent from fullsend-ai#4075