feat: content-specific extract-code + marker_mode + live examples#7
Merged
Conversation
added 4 commits
July 13, 2026 12:03
…+ summarize Split runStarlark into generate + execStarlark so an example can run the exact Starlark source the model wrote (deterministic, no behavior change). Add CG_LIVE- gated tests that drive the real incoming model on realistic tool output: - extract-code on JSON search hits (22 recs/576 tok -> 2 recs/55 tok, ~90%, containment-validated) and on a raw-text pytest log (line-subsequence path). - summarize compressing a mid-trajectory span into one grounded summary. Skipped unless CG_LIVE=1 (+ CG_BASE/CG_TOKEN), so CI and normal test runs are unaffected and no API calls are made. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…on build Reviewed + verified the marker_mode / cg_skeleton work for the AuthBridge integration (see docs/HANDOVER-context-guru-marker-mode.md). - marker_mode (full|summary|off) per offload component: full stashes + leaves a resolvable <<cg:HASH>> marker; summary leaves a non-resolvable ⟪cg⟫ sentinel, no stash; off leaves nothing. summary/off set Report.Irreversible so the pipeline's dropped-without-stash guard permits the deliberate lossy drop (it still reverts a full-mode component that forgets to stash). - components/offload/marker.go centralizes the three modes (mark()); cmdfilter inlines them for its pre-commit never-worse check. expand.HasPlaceholder matches both marker kinds for cross-turn skip-detection. - cg_skeleton build tag isolates skeleton + internal/treesitter so the DEFAULT build is pure-Go (CGO_ENABLED=0), as AuthBridge requires; opt in with -tags cg_skeleton (+cgo). - Added TestMarkerModeSummarySurvivesWire (⟪cg⟫ survives the JSON wire round-trip through apply.Body — HANDOVER review item 5). Build/test/vet green for both the pure-Go default and the cg_skeleton opt-in. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…=code extract-code wrote blind generic filters because the prompt showed the model only a 1500-char sample; on a raw-text pytest log it reduced ~1.3%. Now the model sees the full output and writes code specific to it. - Prompt shows the FULL tool output (bounded ~32k chars; head+tail beyond) so the model targets the actual noise. New deletion-only contract + examples. - Regex helpers in the sandbox (re_sub/re_findall/re_split/re_match, RE2, pure-Go) so it can trim words/sentences/parts within lines, not just whole lines. - Containment generalized to a character subsequence (isCharSubsequence): the result must be obtainable by DELETING characters — trims anything, provably cannot fabricate, reorder, or reword. Keeps the "never corrupts data" guarantee. - Opt-in rewrite:true drops containment for free rewrite (lossy, unverified — pair with a non-full marker_mode); default false stays verified deletion-only. - Split runStarlark into generate + execStarlark so examples run the exact source. - Tests: char-subsequence accept/reject, regex builtins + fail-open, rewrite skips containment; live CG_LIVE examples (pytest 3487->157 tokens ~95%, JSON ~90%). Docs: components.md + RESULTS.md. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
The inbound marker_mode handover has been reviewed and acted on; the current
state is documented in docs/{components,RESULTS}.md, and a fresh outbound
handover for the AuthBridge track lives outside the repo.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on the merged LLM-components work (PR #6) with three things:
marker_mode+cg_skeleton(reviewed for AuthBridge) — a per-offload reversibility knob(
full|summary|off) so an in-process host (AuthBridge) can compact without the expand-stash,plus a
cg_skeletonbuild tag that keeps the default build pure-Go (CGO_ENABLED=0). Reviewedagainst
docs/HANDOVER-context-guru-marker-mode.md; all checklist items hold; addedTestMarkerModeSummarySurvivesWirefor the flagged⟪cg⟫-over-the-wire risk.extract-code: content-specific deletion codegen (the main fix) — extract-code used to write
blind generic filters because the prompt showed the model only a 1.5k-char sample (on a pytest
log it reduced ~1.3%). Now the model sees the full output and writes code specific to it:
re_sub/re_findall/re_split/re_match, RE2, pure-Go),words/sentences/parts within a line, but provably cannot fabricate, reorder, or reword —
the "never corrupts data" guarantee holds.
rewrite: truedrops containment for free rewrite (lossy, unverified; pair with anon-
fullmarker_mode).Reproducible live examples —
CG_LIVE-gated tests that drive the real model (skipped in CI).Validation
CGO_ENABLED=0 go build/test/vet ./...green (pure-Go default);-tags cg_skeletonbuild + tests green.claude-sonnet-4-6):rewrite:true+marker_mode:summary64.5% — both reward-preserved.