Fix agent.extract_code() fence extraction to match its documented contract - #95
Merged
Merged
Conversation
…tract agent.extract_code examined only the FIRST markdown fence and, on a language mismatch, returned the entire response (prose + fences) — its own comment admitted it fell back to returning the response as-is. This leaked non-code text to callers (living-script feeds the result straight to pytest/syntax), contradicting the CLAUDE.md contract: prefers matching lang hint, returns longest block if multiple found. Rewrite agentExtractCode as a multi-fence scanner: - Enumerate ALL fenced blocks, prefer the longest whose language matches the hint, else fall back to the longest block of any language (a model that mislabeled or omitted the tag still yields usable code). - Fix the bare-fence guard: a no-language fence put line_end exactly at fence_start+3, which the old <= check wrongly rejected. - Preserve no-fence -> return input unchanged (keeps the agent.send auto-strip -> extract_code chain intact; stripMarkdownFences untouched). Pure function (no handle/governance/API), one shared dispatch for VM and tree-walker, so the single change covers both engines. New deterministic test tests/governance_v4/test_extract_code.sh (11 assertions, no API keys — runs under --no-governance) covers each traced bug: leading non-target fence, bare fence, longest-of-multiple, json hint, no-match fallback, no-fence passthrough, plus anti-leak assertions. Tests: test_extract_code.sh 11/11; test_error_msg_leaks.sh 874/874; run-all-tests.sh 441 tests, 0 unexpected failures. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018L3RcVMf1tLNsGeJtofWmS
b-macker
marked this pull request as ready for review
July 23, 2026 09:47
b-macker
enabled auto-merge
July 23, 2026 09:47
NAAb Governance Report
All governance checks passed! Generated by NAAb Governance Engine v4.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent.extract_code()sometimes returned the model's raw prose + fences instead of the code, which then failed pytest/syntax downstream in the living-script. Deep-tracing the function against its CLAUDE.md contract ("prefers matching lang hint, returns longest block if multiple found") showed the code never implemented its own documented behavior — its inline comment even admitted "For now, fall back to returning the response as-is if lang doesn't match." This is a contract-conformance fix.Verified bugs (each traced to a line + failing input)
```block was ever considered; "returns longest block if multiple found" was unimplemented.```text…```then```python…```with hintpython: the first fence didn't match, so the entire response (prose + both fences) came back and the python block was never reached.```\ncode\n```puts the newline exactly atfence_start+3, which the oldline_end <= fence_start+3guard wrongly treated as invalid, so bare fences with surrounding prose were never stripped.Change
Rewrote
agentExtractCode(src/stdlib/agent_impl.cpp) as a multi-fence scanner: enumerate all blocks, prefer the longest whose language matches the hint, else fall back to the longest block of any language, else (no fence) return input unchanged.Regression surface (traced, not assumed additive):
stripMarkdownFences(theagent.sendauto-strip) is a separate function, left untouched; the auto-strip → extract_code chain is preserved because the rewrite keeps "no fence → return input unchanged."--no-governanceand no keys.vm.cpp/interpreter, so the fix covers VM and tree-walker identically.agent.extract_codecaller is a live-API gorilla test that skips without keys; nothing codified the old behavior.Test plan
tests/governance_v4/test_extract_code.sh— 11/11, runs under--no-governance(no API keys): single block, prose-stripping, leading non-target fence (bug 2), bare fence (bug 3), longest-of-multiple (bug 1), json hint, no-match fallback, no-fence passthrough, plus explicit anti-leak assertions. Registered inrun-all-tests.sh.bash tests/security/test_error_msg_leaks.sh— 874/874 clean (no error strings changed)bash run-all-tests.sh— 441 tests, 0 unexpected failures; new test ALL PASSED inside the sweepRelated
Follow-up to the living-script forensics series (#87–#94).
🤖 Generated with Claude Code
https://claude.ai/code/session_018L3RcVMf1tLNsGeJtofWmS
Generated by Claude Code