Skip to content

Fix agent.extract_code() fence extraction to match its documented contract - #95

Merged
b-macker merged 1 commit into
masterfrom
claude/living-script-run-analysis-s66x3t
Jul 23, 2026
Merged

Fix agent.extract_code() fence extraction to match its documented contract#95
b-macker merged 1 commit into
masterfrom
claude/living-script-run-analysis-s66x3t

Conversation

@b-macker

Copy link
Copy Markdown
Owner

Summary

agent.extract_code() sometimes returned the model's raw prose + fences instead of the code, which then failed pytest/syntax downstream in the living-script. Deep-tracing the function against its CLAUDE.md contract ("prefers matching lang hint, returns longest block if multiple found") showed the code never implemented its own documented behavior — its inline comment even admitted "For now, fall back to returning the response as-is if lang doesn't match." This is a contract-conformance fix.

Verified bugs (each traced to a line + failing input)

  1. First-fence-only — only the first ``` block was ever considered; "returns longest block if multiple found" was unimplemented.
  2. Language mismatch → whole response returned```text…``` then ```python…``` with hint python: the first fence didn't match, so the entire response (prose + both fences) came back and the python block was never reached.
  3. Bare fence rejected — a no-language ```\ncode\n``` puts the newline exactly at fence_start+3, which the old line_end <= fence_start+3 guard wrongly treated as invalid, so bare fences with surrounding prose were never stripped.

Change

Rewrote agentExtractCode (src/stdlib/agent_impl.cpp) as a multi-fence scanner: enumerate all blocks, prefer the longest whose language matches the hint, else fall back to the longest block of any language, else (no fence) return input unchanged.

Regression surface (traced, not assumed additive):

  • stripMarkdownFences (the agent.send auto-strip) is a separate function, left untouched; the auto-strip → extract_code chain is preserved because the rewrite keeps "no fence → return input unchanged."
  • Only behavior change: mismatch/multi-block input now returns the best/longest block instead of the raw response — strictly better for every real caller (all feed pytest/syntax/convergence; none relied on the raw-response fallback).
  • Pure function (no handle/governance/locks/globals — grep-confirmed): zero threading impact, callable with --no-governance and no keys.
  • One shared dispatch — no inline copy in vm.cpp/interpreter, so the fix covers VM and tree-walker identically.
  • No test breakage: every existing agent.extract_code caller is a live-API gorilla test that skips without keys; nothing codified the old behavior.

Test plan

  • New tests/governance_v4/test_extract_code.sh11/11, runs under --no-governance (no API keys): single block, prose-stripping, leading non-target fence (bug 2), bare fence (bug 3), longest-of-multiple (bug 1), json hint, no-match fallback, no-fence passthrough, plus explicit anti-leak assertions. Registered in run-all-tests.sh.
  • bash tests/security/test_error_msg_leaks.sh — 874/874 clean (no error strings changed)
  • bash run-all-tests.sh — 441 tests, 0 unexpected failures; new test ALL PASSED inside the sweep
  • Live run on the originating device — expect fewer spurious pytest/syntax failures from wrong-tag / bare-fence model responses

Related

Follow-up to the living-script forensics series (#87#94).

🤖 Generated with Claude Code

https://claude.ai/code/session_018L3RcVMf1tLNsGeJtofWmS


Generated by Claude Code

…tract

agent.extract_code examined only the FIRST markdown fence and, on a
language mismatch, returned the entire response (prose + fences) — its own
comment admitted it fell back to returning the response as-is. This leaked
non-code text to callers (living-script feeds the result straight to
pytest/syntax), contradicting the CLAUDE.md contract: prefers matching lang
hint, returns longest block if multiple found.

Rewrite agentExtractCode as a multi-fence scanner:
- Enumerate ALL fenced blocks, prefer the longest whose language matches the
  hint, else fall back to the longest block of any language (a model that
  mislabeled or omitted the tag still yields usable code).
- Fix the bare-fence guard: a no-language fence put line_end exactly at
  fence_start+3, which the old <= check wrongly rejected.
- Preserve no-fence -> return input unchanged (keeps the agent.send
  auto-strip -> extract_code chain intact; stripMarkdownFences untouched).

Pure function (no handle/governance/API), one shared dispatch for VM and
tree-walker, so the single change covers both engines.

New deterministic test tests/governance_v4/test_extract_code.sh (11
assertions, no API keys — runs under --no-governance) covers each traced
bug: leading non-target fence, bare fence, longest-of-multiple, json hint,
no-match fallback, no-fence passthrough, plus anti-leak assertions.

Tests: test_extract_code.sh 11/11; test_error_msg_leaks.sh 874/874;
run-all-tests.sh 441 tests, 0 unexpected failures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L3RcVMf1tLNsGeJtofWmS
@b-macker
b-macker marked this pull request as ready for review July 23, 2026 09:47
@b-macker
b-macker enabled auto-merge July 23, 2026 09:47
@github-actions

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

@b-macker
b-macker merged commit a60f1b1 into master Jul 23, 2026
21 checks passed
@b-macker
b-macker deleted the claude/living-script-run-analysis-s66x3t branch July 23, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants