feat: LLM-based context components (summarize + extract code strategy)#6
Merged
Conversation
added 6 commits
July 12, 2026 18:44
…tegy)
Land the two model-calling components, general + configurable, largely restored
from git history.
- internal/cheapmodel: Anthropic+OpenAI Complete(ctx,prompt) clients (stdlib).
- components.Model + ModelSpec{Incoming,Static}.For(source); Ctx.Model populated
by apply.BodyWithModel. proxy builds the per-request 'incoming' client (route
upstream + gateway key + request model); cmd builds a static 'config' cheap
model from CHEAP_MODEL* env; AuthBridge offers config-only.
- summarize (CE-Manager port): whole-transcript summary -> [msg0, <summary>,
last-K], replaced span stashed for expand. apply gains a lossless count-change
rebuild (retained messages kept byte-identical).
- extract: strategy deterministic|code|rlm; code runs an LLM-written Starlark
filter in a sandbox (no imports/IO, step+2s limits) accepted only if a
containment check proves a lossless subset, else falls back to deterministic;
content-hash cache. Starlark chosen over sandboxed Python (safer, pure-Go).
- NeedsModel components degrade gracefully when no model is available.
Tests for summarize shape/stash/no-model, extract code + fallback, and the
apply count-change lossless invariant. Live-verified: extract code had
claude-sonnet-4-6 write a Starlark filter that reduced a tool output 65%.
Assisted-By: Claude (Anthropic AI)
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> # Conflicts: # config/config.go # docs/components.md
…n, edge cases) - internal/extract: RunExtraction accepts a contained Starlark result, REJECTS a fabricated (non-contained) one (falls back to deterministic); ContentKey marker-insensitivity; HarvestIdentifiers. - components: ModelSpec.For resolution (incoming/config/nil/fallback). - summarize: model-error fail-open (transcript untouched), empty-response skip. - extract: rlm strategy maps to the code path (RLM deferred). Assisted-By: Claude (Anthropic AI) Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…sweep - 30s per-call timeout on extract-code / summarize model calls (fail-open if the model is slow), so an in-request LLM call can't stall the agent. - gateway start accepts CONTEXT_GURU_CONFIG_YAML (full config doc) so a sweep can pin per-component config (strategy: code, model.source); forwarded via the compose override. - sweep.py: cg-extract-code / cg-summarize configs (model.source: incoming) with a higher LLM floor (min_tokens 1500 — only large outputs pay for a call). Assisted-By: Claude (Anthropic AI) Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…pactions
Make the LLM-based components (summarize, extract strategy: code) run only when
worthwhile and reuse prior work, and make extract-code apply to any output shape.
- Trigger: new components.Trigger{min_request_tokens,min_messages,min_output_tokens}
embedded in summarize/extract so they fire on request shape (tokens/steps/large
output), not every turn. Zero fields = fire always (back-compat).
- State reuse: extract caches each reduced output by content hash (re-sent output
reuses it, no new model call, byte-identical); summarize checkpoints its summary
per session and reuses it until the tail grows past resummarize_tokens.
- Generalize extract-code: IsContained string case now accepts an in-order
whole-line subsequence, so logs/source/tracebacks reduce, not only JSON; prompt
rewritten domain-agnostic (JSON + raw-text examples).
- Full context: both components use conversationGoal (task + recent turns), not
just the last message; summarizer is told to summarize toward the task.
- Reliability: raise call timeouts (extract 60s, summarize 150s) and cap the
summarizer trajectory to fit the model context instead of failing open.
- Tests: Trigger.Fires, line-subsequence containment, extract result-cache reuse,
summarize checkpoint reuse. Docs: components.md + RESULTS.md (tuned 10-task sweep;
resolved 6/6/6, zero reward regression; extract-code ~193k, summarize ~15.6k saved).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Preserve the full 10-task sweep behind docs/RESULTS.md as a dated snapshot (the live sweep-results.csv is gitignored scratch). One row per (task, config) with reward + gateway token-savings stats; LLM configs use the final tuned triggers. Adds a README documenting columns, config, and headline (resolved 6/6/6, extract-code ~193k / summarize ~15.6k tokens saved). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the two model-calling context-engineering components (the deterministic ones shipped in #4), general + configurable, largely restored from git history.
Components
summarize— CE-Manager ReSum-style whole-transcript summary →[msg0, <summary>, last-K]; the replaced span is stashed under a<<cg:HASH>>marker forexpand. Config:summary_level,keep_last,start_from_message,min_tokens,include_tool_calls,model.source.extractstrategy: code— a cheap LLM writes a Starlarkextract_relevant_data-style filter, run in a sandbox (no imports/IO, step + 2s limits); the output is accepted only if a containment check proves a lossless subset, else falls back to deterministic. Content-hash cache. (rlmmaps tocodefor now.)Plumbing
internal/cheapmodel(Anthropic+OpenAIComplete, stdlib) +components.Model/ModelSpec{Incoming,Static}.For(source).incoming(default — reuse the request's own model + gateway key, built inproxy.chat) orconfig(a dedicated cheap model fromCHEAP_MODEL*). AuthBridge = config-only.apply.BodyWithModelsetsCtx.Model; a new lossless count-change rebuild keeps retained messages byte-identical whensummarizerestructures the transcript.NeedsModelcomponents degrade gracefully with no model (extract→deterministic, summarize→no-op) and are fail-open (any error reverts the component).Why Starlark (not sandboxed Python)
Researched: Starlark is expressive enough for these select/subset filters and decisively safer (no import/reflection, guaranteed halting) with zero external runtime; the language-agnostic containment validator is the real safety net.
Tests + live verification
Unit: summarize shape/stash/no-model-skip, extract code uses-model + nil-fallback, apply count-change lossless invariant. Full suite + vet + gofmt green. Live:
extract codehadclaude-sonnet-4-6write a Starlark filter that reduced a tool output 267→92 tokens (65%), containment-validated.Follow-ups
Expand-tool injection (the reversibility unlock for these lossy components) and the RLM recursive-REPL variant remain follow-ups.