From 0495e705ec5184d33675fa35844e6ca6628f6283 Mon Sep 17 00:00:00 2001 From: Osher-Elhadad Date: Sun, 12 Jul 2026 17:00:04 +0300 Subject: [PATCH 1/2] docs: add per-component SWE-bench results + agent preset Full per-component eval on SWE-bench (Claude Code, claude-sonnet-4-6) through the eval-containers gateway: zero task-reward regression across every component; mask offloads the growing file-read transcript for ~27% content-token savings with no reward loss, the biggest lever on agentic traffic. Adds docs/RESULTS.md (table + methodology + why the inert components are inert on this traffic) and a mask-centric 'agent' preset [format, dedup, failed_run, mask, extract, cacheinject]. Assisted-By: Claude (Anthropic AI) Signed-off-by: Osher-Elhadad --- README.md | 1 + config/config.go | 8 ++++++ docs/RESULTS.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++ docs/components.md | 4 ++- 4 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 docs/RESULTS.md diff --git a/README.md b/README.md index 18efb09..7a8c6cb 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,7 @@ Details in [docs/integrations.md](docs/integrations.md). - [docs/components.md](docs/components.md) — every registered component: how it works, before→after, lossiness, config, best use. - [docs/integrations.md](docs/integrations.md) — proxy gateway vs AuthBridge plugin, with request paths. - [docs/setup.md](docs/setup.md) — setup + a concrete SWE-bench run through the eval-containers gateway. +- [docs/RESULTS.md](docs/RESULTS.md) — per-component SWE-bench benchmark (Claude Code, claude-sonnet-4-6): `mask` ≈27% token savings, no reward loss. ## License diff --git a/config/config.go b/config/config.go index 0826713..2da8d03 100644 --- a/config/config.go +++ b/config/config.go @@ -78,6 +78,14 @@ var presets = map[string][]string{ "aggressive": {"format", "dedup", "failed_run", "cmdfilter", "smartcrush", "extract", "cacheinject"}, "coding": {"format", "skeleton", "cmdfilter", "cacheinject"}, "mcp": {"format", "smartcrush", "cacheinject"}, + // agent: tuned for long agentic sessions (e.g. Claude Code on SWE-bench), + // where the dominant cost is the transcript of tool outputs (file reads) + // re-sent every turn. mask (drop old tool outputs) is the biggest lever + // there — ~27% content-token savings with no task-reward loss in the + // eval-containers SWE-bench sweep (see docs/RESULTS.md); extract + failed_run + // + dedup add relevance/supersession/dup wins; cacheinject keeps the prefix + // cacheable. Order: lossless first, then offload old-then-large, cache last. + "agent": {"format", "dedup", "failed_run", "mask", "extract", "cacheinject"}, } // Build constructs the ordered pipeline from the config, wiring each named diff --git a/docs/RESULTS.md b/docs/RESULTS.md new file mode 100644 index 0000000..6ac73e8 --- /dev/null +++ b/docs/RESULTS.md @@ -0,0 +1,70 @@ +# Benchmark results + +Per-component evaluation of context-guru on **SWE-bench** with the **Claude Code** +agent and model **`claude-sonnet-4-6`**, run through the context-guru gateway in +[eval-containers](setup.md). Baseline = the same gateway with an empty pipeline +(passthrough); each `cg-*` row = that one component alone (default config, **no +task-specific tuning**); `cg-balanced` / `cg-agent` = presets. + +## Headline + +- **Zero task-reward regression, every component.** On the tasks the baseline + resolved, every configuration resolved them too (6/6). Even the lossy + offloaders (`mask`, `extract`, `collapse`) did not break a task — the offloaded + old tool outputs were not re-needed. +- **`mask` is the biggest lever: ~27% content-token reduction, no reward loss.** + Claude Code's context is dominated by file-read tool outputs re-sent every turn; + dropping the old ones (keep-recent window) attacks that directly. + +## Savings on the 6 baseline-resolved tasks (honest reward-parity basis) + +| config | mean content-token savings | reward kept | mechanism | +|---|---:|:---:|---| +| `cg-mask` | **26.8%** | 6/6 | offload tool outputs older than the keep-recent window | +| `cg-balanced` | 12.2% | 6/6 | format + dedup + failed_run + cmdfilter + cacheinject | +| `cg-extract` | 11.1% | 6/6 | project each large tool output to query-relevant lines | +| `cg-failed_run` | 5.8% | 6/6 | collapse superseded test/build runs | +| `cg-collapse` | ~0% | 6/6 | fires only on >2000-token outputs (rare here) | +| `cg-dedup` | ~0.2% | 6/6 | exact within-request duplicates (rare here) | +| `cg-format`, `cg-cmdfilter`, `cg-cacheinject`, `cg-skeleton`, `cg-smartcrush`, `cg-phi_evict` | 0% | 6/6 | inert on this traffic — see below | + +The **`agent` preset** (`format, dedup, failed_run, mask, extract, cacheinject`) +combines the winners; use it for long agentic sessions. + +## Why the 0% components are inert here (not bugs) + +A debug capture of real Claude Code traffic showed the tool outputs are +**line-numbered file reads** (source code) and git — repeated across turns as the +transcript grows. So: + +| component | needs | this traffic | +|---|---|---| +| `format` | pretty-printed JSON | file reads, not JSON | +| `cmdfilter` | shell-command banners (pytest/npm/make) | file reads, not command output | +| `skeleton` | fenced ` ```lang ` code blocks | Read output is line-numbered, unfenced | +| `smartcrush` | JSON arrays | none | +| `phi_evict` | transcript over `budget_tokens` (120k) | under budget | +| `cacheinject` | (saves provider **KV-cache** tokens, not content tokens) | works, but invisible to content-token metric | + +## Methodology & caveats + +- Harness: [`deploy/eval-containers/sweep.py`](../deploy/eval-containers/sweep.py) + (resumable) → `aggregate.py`. One cell = one `(task, config)` run through the + compose stack; reward from `output/task/result.json`, savings from the gateway + `/stats` (within-run `tokens_before → tokens_after`). +- **Within-run savings %** is the honest metric — before/after are the same + request stream, immune to run-to-run agent nondeterminism. Cumulative token + totals across a growing transcript overstate per-task savings and are not used + for the headline. +- Reward parity is measured only on tasks the **baseline resolved** (6 of 10); + the others neither baseline nor context-guru resolved (SWE-bench resolve rates + vary), so they can't show "did reduction hurt correctness". +- Grading requires **offline-gradable** tasks; network-dependent tasks (e.g. + `psf/requests`) grade as 0 under the runner's isolated network and are excluded. +- Host was arm64 (SWE-bench images run under emulation); results are functional, + not latency benchmarks. + +## Tasks + +`sympy-13647/16766/20438`, `sphinx-7910/9320`, `scikit-learn-12973/25931`, +`xarray-4629`, `django-11820/14089`. diff --git a/docs/components.md b/docs/components.md index abc2d40..275422c 100644 --- a/docs/components.md +++ b/docs/components.md @@ -23,7 +23,9 @@ messages (`role:"tool"`; for Anthropic, `tool_result` blocks normalized to that Presets (`config`): `off` `[]` · `safe` `[format, cacheinject]` · `balanced` `[format, dedup, failed_run, cmdfilter, cacheinject]` · `aggressive` adds `smartcrush, extract` · -`coding` `[format, skeleton, cmdfilter, cacheinject]` · `mcp` `[format, smartcrush, cacheinject]`. +`coding` `[format, skeleton, cmdfilter, cacheinject]` · `mcp` `[format, smartcrush, cacheinject]` · +**`agent`** `[format, dedup, failed_run, mask, extract, cacheinject]` — for long agentic sessions; +`mask` is the biggest lever there (~27% content-token savings, no reward loss — see [RESULTS.md](RESULTS.md)). Common gates every Offload respects: skip non-text (`Rewritable`) messages, skip content already carrying a marker (no double-offload), and skip if the rewrite (marker + hint included) isn't From ebbb700eff0a62079b080a5ce6f0970e4175f4e6 Mon Sep 17 00:00:00 2001 From: Osher-Elhadad Date: Sun, 12 Jul 2026 17:44:56 +0300 Subject: [PATCH 2/2] docs: expand RESULTS.md with full per-task metrics + real message examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All-task tables (savings%, reward, agent wall-clock across 10 tasks × 13 configs), per-component summary with measured compute time (mask ~2ms, failed_run/extract ~6ms), and real before->after message rewrites captured via CONTEXT_GURU_DUMP on sphinx-7910 (mask 606->27, failed_run 423->33, extract relevance projection). Notes the marker-overhead per-message follow-up for the offloaders. Assisted-By: Claude (Anthropic AI) Signed-off-by: Osher-Elhadad --- docs/RESULTS.md | 255 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 196 insertions(+), 59 deletions(-) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 6ac73e8..95ae81f 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1,70 +1,207 @@ # Benchmark results -Per-component evaluation of context-guru on **SWE-bench** with the **Claude Code** -agent and model **`claude-sonnet-4-6`**, run through the context-guru gateway in -[eval-containers](setup.md). Baseline = the same gateway with an empty pipeline -(passthrough); each `cg-*` row = that one component alone (default config, **no -task-specific tuning**); `cg-balanced` / `cg-agent` = presets. +Per-component evaluation of context-guru on **SWE-bench** with the **Claude Code** agent and model +**`claude-sonnet-4-6`**, through the context-guru gateway in [eval-containers](setup.md). Baseline = the +same gateway with an empty pipeline (passthrough); each `cg-*` = that one component alone (default config, +**no task-specific tuning**); `cg-balanced` = the preset. 10 tasks × 13 configs = 130 runs. ## Headline -- **Zero task-reward regression, every component.** On the tasks the baseline - resolved, every configuration resolved them too (6/6). Even the lossy - offloaders (`mask`, `extract`, `collapse`) did not break a task — the offloaded - old tool outputs were not re-needed. -- **`mask` is the biggest lever: ~27% content-token reduction, no reward loss.** - Claude Code's context is dominated by file-read tool outputs re-sent every turn; - dropping the old ones (keep-recent window) attacks that directly. - -## Savings on the 6 baseline-resolved tasks (honest reward-parity basis) - -| config | mean content-token savings | reward kept | mechanism | -|---|---:|:---:|---| -| `cg-mask` | **26.8%** | 6/6 | offload tool outputs older than the keep-recent window | -| `cg-balanced` | 12.2% | 6/6 | format + dedup + failed_run + cmdfilter + cacheinject | -| `cg-extract` | 11.1% | 6/6 | project each large tool output to query-relevant lines | -| `cg-failed_run` | 5.8% | 6/6 | collapse superseded test/build runs | -| `cg-collapse` | ~0% | 6/6 | fires only on >2000-token outputs (rare here) | -| `cg-dedup` | ~0.2% | 6/6 | exact within-request duplicates (rare here) | -| `cg-format`, `cg-cmdfilter`, `cg-cacheinject`, `cg-skeleton`, `cg-smartcrush`, `cg-phi_evict` | 0% | 6/6 | inert on this traffic — see below | - -The **`agent` preset** (`format, dedup, failed_run, mask, extract, cacheinject`) -combines the winners; use it for long agentic sessions. - -## Why the 0% components are inert here (not bugs) - -A debug capture of real Claude Code traffic showed the tool outputs are -**line-numbered file reads** (source code) and git — repeated across turns as the -transcript grows. So: - -| component | needs | this traffic | -|---|---|---| -| `format` | pretty-printed JSON | file reads, not JSON | -| `cmdfilter` | shell-command banners (pytest/npm/make) | file reads, not command output | -| `skeleton` | fenced ` ```lang ` code blocks | Read output is line-numbered, unfenced | -| `smartcrush` | JSON arrays | none | -| `phi_evict` | transcript over `budget_tokens` (120k) | under budget | -| `cacheinject` | (saves provider **KV-cache** tokens, not content tokens) | works, but invisible to content-token metric | +- **Zero task-reward change, every component.** Of the 10 tasks, the baseline resolved 6; on those 6 every + config still resolved (6/6), and none of the 4 the baseline failed were rescued. context-guru changes + tokens, not agent capability — no task broken, none fixed. +- **`mask` is the biggest lever** (drop tool outputs older than the keep-recent window): **26.8% mean + content-token savings on the resolved tasks, up to 93.5% on a long session**, reward preserved. +- **Component overhead is single-digit milliseconds** — the token savings cost effectively no compute. + +## Per-component summary + +| component | fires on | mean save% (all 10) | mean save% (resolved 6) | reward kept | compute/req | +|---|---|---:|---:|:---:|---:| +| `cg-format` | pretty-printed JSON | 0.0 | 0.0 | 6/6 | ~1–7 ms | +| `cg-dedup` | exact dup tool outputs | 0.3 | 0.2 | 6/6 | ~1–7 ms | +| `cg-cmdfilter` | shell-cmd banners | 0.0 | 0.0 | 6/6 | ~1–7 ms | +| `cg-cacheinject` | provider cache (not content) | 0.0 | 0.0 | 6/6 | ~1–7 ms | +| `cg-failed_run` | superseded test/build runs | 7.3 | 5.8 | 6/6 | ~6 ms | +| `cg-skeleton` | fenced code blocks | 0.0 | 0.0 | 6/6 | ~1–7 ms | +| `cg-collapse` | >2000-tok outputs | 0.6 | 0.0 | 6/6 | ~1–7 ms | +| `cg-mask` | old tool outputs | 30.9 | 26.8 | 6/6 | ~2 ms | +| `cg-smartcrush` | JSON arrays | 0.0 | 0.0 | 6/6 | ~1–7 ms | +| `cg-extract` | large outputs (relevance) | 11.4 | 11.1 | 6/6 | ~6 ms | +| `cg-phi_evict` | transcript > budget | 0.0 | 0.0 | 6/6 | ~1–7 ms | +| `cg-balanced` | combined preset | 9.0 | 12.2 | 6/6 | ~1–7 ms | + +Compute/req measured directly for mask/failed_run/extract; the rest are the same order of magnitude +(regex / JSON / hashing over the message array). All negligible vs. the agent's seconds-to-minutes per step. + +## Content-token savings % per task (within-run before→after) + +| config | sympy-13647 | sympy-16766 | sympy-20438 | sphinx-7910 | sphinx-9320 | scikit-learn-12973 | scikit-learn-25931 | xarray-4629 | django-11820 | django-14089 | +|---|---|---|---|---|---|---|---|---|---|---| +| `baseline` | — | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | — | 0.0 | 0.0 | +| `cg-format` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-dedup` | 1.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.5 | 0.0 | +| `cg-cmdfilter` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-cacheinject` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-failed_run` | 9.0 | 19.7 | 1.4 | 5.4 | 0.0 | 0.0 | 0.8 | 0.0 | 36.8 | 0.0 | +| `cg-skeleton` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-collapse` | 0.0 | 0.0 | 6.2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-mask` | 19.6 | 42.9 | 93.5 | 19.0 | 39.0 | 0.0 | 40.6 | 0.0 | 50.0 | 4.2 | +| `cg-smartcrush` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-extract` | 14.9 | 5.0 | 28.8 | 19.3 | 3.8 | 11.2 | 12.1 | 0.0 | 19.1 | 0.0 | +| `cg-phi_evict` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| `cg-balanced` | 14.6 | 14.1 | 2.6 | 21.0 | 0.0 | 0.0 | 23.4 | 0.0 | 14.1 | 0.0 | + +## Reward per task (1 = resolved) + +| config | sympy-13647 | sympy-16766 | sympy-20438 | sphinx-7910 | sphinx-9320 | scikit-learn-12973 | scikit-learn-25931 | xarray-4629 | django-11820 | django-14089 | +|---|---|---|---|---|---|---|---|---|---|---| +| `baseline` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | — | 0 | 0 | +| `cg-format` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-dedup` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-cmdfilter` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-cacheinject` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-failed_run` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-skeleton` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-collapse` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-mask` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-smartcrush` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-extract` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-phi_evict` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | +| `cg-balanced` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | + +## Agent wall-clock (seconds) — single run each; **dominated by agent nondeterminism, not component cost** + +| config | sympy-13647 | sympy-16766 | sympy-20438 | sphinx-7910 | sphinx-9320 | scikit-learn-12973 | scikit-learn-25931 | xarray-4629 | django-11820 | django-14089 | +|---|---|---|---|---|---|---|---|---|---|---| +| `baseline` | 67 | 59 | 333 | 56 | 53 | 28 | 138 | — | 200 | 22 | +| `cg-format` | 74 | 60 | 342 | 60 | 52 | 31 | 129 | 25 | 113 | 23 | +| `cg-dedup` | 71 | 59 | 461 | 108 | 38 | 35 | 154 | 21 | 172 | 34 | +| `cg-cmdfilter` | 82 | 70 | 251 | 74 | 47 | 33 | 96 | 22 | 93 | 25 | +| `cg-cacheinject` | 58 | 49 | 1352 | 119 | 37 | 32 | 130 | 22 | 118 | 28 | +| `cg-failed_run` | 86 | 95 | 375 | 126 | 53 | 32 | 91 | 22 | 219 | 23 | +| `cg-skeleton` | 80 | 82 | 253 | 60 | 45 | 36 | 106 | 24 | 233 | 22 | +| `cg-collapse` | 76 | 82 | 1047 | 106 | 46 | 30 | 121 | 29 | 168 | 25 | +| `cg-mask` | 63 | 87 | 1833 | 109 | 59 | 33 | 155 | 33 | 116 | 27 | +| `cg-smartcrush` | 65 | 89 | 705 | 125 | 38 | 30 | 132 | 24 | 240 | 23 | +| `cg-extract` | 70 | 96 | 517 | 109 | 61 | 42 | 123 | 23 | 218 | 22 | +| `cg-phi_evict` | 79 | 93 | 276 | 111 | 37 | 35 | 140 | 26 | 191 | 38 | +| `cg-balanced` | 91 | 71 | 633 | 129 | 59 | 36 | 156 | 29 | 454 | 22 | + +## Real message rewrites (captured on `sphinx-doc__sphinx-7910`) + +Actual before→after content the components produced (via `CONTEXT_GURU_DUMP`). Offloaded originals are +recoverable from the store by the `<>` marker. + +### `mask` — drops tool outputs older than the keep-recent window + +**606 → 27 tokens (saved 579):** + +```text +BEFORE: +407- `obj` will be `A.__init__`. +408- skip : bool +409- A boolean indicating if autodoc will skip this member if `_skip_member` +410- does not override the decision +411- options : sphinx.ext.autodoc.Options +412- … +AFTER: +[older tool output masked] <> [full output: call context_guru_expand] +``` + +**223 → 31 tokens (saved 192):** + +```text +BEFORE: +426 if name != '__weakref__' and has_doc and is_member: +427 cls_is_owner = False +428 if what == 'class' or what == 'exception': +429 qualname = getattr(obj, '__qualname__', '') +430 cls_path, _, _ = … +AFTER: +[older tool output masked] <> [full output: call context_guru_expand] +``` + +### `failed_run` — collapses a run superseded by a later one + +**423 → 33 tokens (saved 390):** + +```text +BEFORE: +Exit code 1 +WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establi … +AFTER: +[superseded by a later run] <> [full output: call context_guru_expand] +``` + +**201 → 32 tokens (saved 169):** + +```text +BEFORE: +__wrapped__: True +__qualname__: MyClass.__init__ +cls_path: MyClass +wrapper __globals__ keys: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__builtins__', 'functools', 'my_decorator'] +Old way FAILED with KeyError - BUG CO … +AFTER: +[superseded by a later run] <> [full output: call context_guru_expand] +``` + +### `extract` — keeps query-relevant lines, elides the rest + +**455 → 434 tokens (saved 21):** + +```text +BEFORE: +415 directive. +416 +417 Returns +418 ------- +419 bool +420 True if the member should be skipped during creation of the docs, +421 False if it should be included in the docs. +422 +423 """ +424 has_doc … +AFTER: +415 directive. +416 +417 Returns +418 ------- +419 bool +420 True if the member should be skipped during creation of the docs, +421 False if it should be included in the docs. +… +424 has_doc = getattr(obj, '__doc__', False) +425 is_member = (what == 'class' or wh +``` ## Methodology & caveats -- Harness: [`deploy/eval-containers/sweep.py`](../deploy/eval-containers/sweep.py) - (resumable) → `aggregate.py`. One cell = one `(task, config)` run through the - compose stack; reward from `output/task/result.json`, savings from the gateway - `/stats` (within-run `tokens_before → tokens_after`). -- **Within-run savings %** is the honest metric — before/after are the same - request stream, immune to run-to-run agent nondeterminism. Cumulative token - totals across a growing transcript overstate per-task savings and are not used - for the headline. -- Reward parity is measured only on tasks the **baseline resolved** (6 of 10); - the others neither baseline nor context-guru resolved (SWE-bench resolve rates - vary), so they can't show "did reduction hurt correctness". -- Grading requires **offline-gradable** tasks; network-dependent tasks (e.g. - `psf/requests`) grade as 0 under the runner's isolated network and are excluded. -- Host was arm64 (SWE-bench images run under emulation); results are functional, - not latency benchmarks. +- Harness: [`deploy/eval-containers/sweep.py`](../deploy/eval-containers/sweep.py) (resumable) → `aggregate.py`. + One cell = one `(task, config)` run through the compose stack; reward from `output/task/result.json`, + savings from the gateway `/stats` (within-run `tokens_before → tokens_after`), component time from the + gateway's per-component `duration_ms` logs, example rewrites from `CONTEXT_GURU_DUMP`. +- **Within-run savings %** is the honest metric — before/after are the same request stream, immune to + run-to-run agent nondeterminism. Cumulative token totals across a growing transcript overstate per-task + savings and are not used here. +- **Single run per cell.** Agent trajectories are stochastic, so both wall-clock and per-task savings vary + run-to-run (e.g. `extract` on sphinx-7910 was 19% in the sweep, 0.7% on the capture re-run). Robust + numbers need N repeats per cell; treat single-task figures as indicative, the per-component means as the + signal. +- **Reward parity** is judged only on the 6 tasks the baseline resolved; the other 4 neither baseline nor + context-guru resolved (SWE-bench resolve rates vary). +- **$ cost** is not measured directly — Claude Code streams (SSE) and the gateway would need to parse the + streaming `usage` for exact provider-billed tokens. Savings are content tokens; at a representative + ~$3/M Sonnet input price the message-portion input cost drops roughly in proportion to the savings % + each turn, compounding across a session. +- Host was arm64 (SWE-bench images run under emulation); functional results, not latency benchmarks. +- **Known follow-up:** `extract`/`collapse`/`mask`/`failed_run`/`skeleton` check their per-message size + win *before* appending the `<>` marker, so an individual message can occasionally grow by the + marker's ~10–15 tokens (the pipeline's overall never-worse guard still reverts any net regression). The + marker-inclusive check already added to `cmdfilter` should be applied to the other offloaders too. ## Tasks -`sympy-13647/16766/20438`, `sphinx-7910/9320`, `scikit-learn-12973/25931`, -`xarray-4629`, `django-11820/14089`. +`sympy-13647/16766/20438`, `sphinx-7910/9320`, `scikit-learn-12973/25931`, `xarray-4629`, +`django-11820/14089`.