diff --git a/README.md b/README.md index ad02991..be84eb9 100644 --- a/README.md +++ b/README.md @@ -32,25 +32,29 @@ runs, projecting large reads down to what's relevant — and every reduction is ## Benchmark: the cheapest & highest-reward arm on SWE-bench Verified Evaluated **live, end-to-end**, with the **claude-code** agent on **`aws/claude-sonnet-5`**, against a -no-compaction baseline and against [**headroom**](https://pypi.org/project/headroom-ai/) -(`headroom-ai` v0.32.1). All 50 tasks scored under all three arms. - -| dimension | baseline | **context-guru** | headroom | -|---|--:|--:|--:| -| tasks solved | 86% | **88%** | 80% | -| **total billed cost** vs baseline | — | **−13.2%** | −5.3% | -| cache-read tokens vs baseline | — | **−17.8%** | −6.3% | -| cache-write tokens vs baseline | — | −0.4% | −0.9% | -| mean steps / task vs baseline | — | **−13.9%** | −2.8% | -| added latency / req | — | 117 ms | 63 ms | +no-compaction baseline, against the [**headroom**](https://pypi.org/project/headroom-ai/) request-stream +proxy, and against [**rtk**](https://github.com/rtk-ai/rtk) (Rust Token Killer, a shell-level Bash-output +hook). All 50 tasks scored under all **four** arms. + +| dimension | baseline | **context-guru** | headroom | rtk | +|---|--:|--:|--:|--:| +| tasks solved | 86% | **88%** | 80% | 86% | +| **total billed cost** vs baseline | — | **−13.2%** | −5.3% | −9.0% | +| cache-read tokens vs baseline | — | **−17.8%** | −6.3% | −10.8% | +| cache-write tokens vs baseline | — | −0.4% | −0.9% | −1.1% | +| mean steps / task vs baseline | — | **−13.9%** | −2.8% | −8.0% | +| added latency / req | — | 117 ms | 63 ms | **0 ms** | +| tool's own LLM cost | — | $0.31 | $0 | $0 | **context-guru is the cheapest arm and solves the most tasks** — it cuts billed cost **13.2%** vs no -compaction (headroom cuts only **5.3%**), driven by an **17.8%** cache-read reduction, while keeping -cache-write within **1%** of baseline (it never busts the cache). It does this by *freezing each compaction -and replaying it byte-identically every turn*, so the saving compounds across the whole session. headroom -keeps an edge on added latency (it is fully deterministic — no model on the hot path). Full three-way study, -per-task/per-component breakdowns, real before→after examples, and how to reproduce: -**[docs/RESULTS.md](docs/RESULTS.md)**. +compaction, driven by an **17.8%** cache-read reduction, while keeping cache-write within **1%** of baseline +(it never busts the cache). It does this by *freezing each compaction and replaying it byte-identically every +turn*, so the saving compounds across the whole session. The surprise is **rtk**: a simple deterministic +shell filter is the **2nd-cheapest** arm (**−9.0%**), **reward-neutral** (86% = baseline), at **zero +request-path latency and $0 tool cost** — it **beats the headroom proxy on both cost and reward**. rtk's +ceiling is that it only compresses **Bash-tool** output (Claude Code's built-in `Read`/`Grep`/`Glob` bypass +its hook), which is why the whole-request proxy goes deeper. Full four-way study, per-task/per-component +breakdowns, real before→after examples, and how to reproduce: **[docs/RESULTS.md](docs/RESULTS.md)**. ## Architecture @@ -182,7 +186,7 @@ Details in [docs/integrations.md](docs/integrations.md). - [docs/components.md](docs/components.md) — every registered component: how it works, live before→after, lossiness, config, best use. - [docs/integrations.md](docs/integrations.md) — proxy gateway vs AuthBridge plugin, with request paths. - [docs/setup.md](docs/setup.md) — setup + a concrete SWE-bench run through the eval-containers gateway. -- [docs/RESULTS.md](docs/RESULTS.md) — the live three-way SWE-bench Verified benchmark (Claude Code, `aws/claude-sonnet-5`): context-guru is the cheapest arm (−13.2% billed cost vs baseline) and solves the most tasks (88% vs baseline 86%, headroom 80%). +- [docs/RESULTS.md](docs/RESULTS.md) — the live four-way SWE-bench Verified benchmark (Claude Code, `aws/claude-sonnet-5`): context-guru is the cheapest arm (−13.2% billed cost vs baseline) and solves the most tasks (88%); headroom −5.3%/80%; rtk (shell-level Bash-output hook) −9.0%/86% at $0 tool cost. ## License diff --git a/deploy/harbor/claude_code_rtk_agent.py b/deploy/harbor/claude_code_rtk_agent.py new file mode 100644 index 0000000..bef42af --- /dev/null +++ b/deploy/harbor/claude_code_rtk_agent.py @@ -0,0 +1,112 @@ +"""Claude Code + rtk (Rust Token Killer) agent. + +Identical to the stock ``claude-code`` agent, except it makes the ``rtk`` CLI +proxy active inside the task container so that **bash tool output is compressed +at the shell** before it ever enters the model context. + +rtk is architecturally different from request-stream compaction proxies +(context-guru, headroom): it is a Claude Code ``PreToolUse`` hook that rewrites +Bash commands (``pytest ...`` -> ``rtk pytest ...``, ``cat f`` -> ``rtk read f``, +``git status`` -> ``rtk git status``) *inside the container*. It is not on the +network path and needs no proxy. Model routing is therefore whatever the harness +already points ``ANTHROPIC_BASE_URL`` at — for an apples-to-apples comparison the +harness routes it exactly like the ``off`` baseline, so the ONLY difference from +baseline is the in-container bash compression. + +This subclass: + 1. ``install()`` — after the stock Claude Code install, uploads a host-built + static-musl ``rtk`` binary to ``/usr/local/bin/rtk`` (path from + ``RTK_BIN_HOST``, default ``/tmp/rtk-runs/rtk``) and also symlinks it into + ``~/.local/bin`` so it is on the same PATH Claude Code launches with. + 2. ``run()`` — installs the rtk ``PreToolUse`` hook into the exact + ``CLAUDE_CONFIG_DIR`` Claude Code will read (``rtk init -g --auto-patch``, + which honors ``CLAUDE_CONFIG_DIR``), runs the stock agent, then dumps + ``rtk gain --all --format json`` to ``/logs/agent/rtk-gain.json`` so the + harness can report rtk's own bash-output savings per trial. + +End-to-end trajectory metrics (reward, steps, cache-read/write, billed cost, +cache-hit, wall) are produced by the unchanged parent class, so they are +measured identically to the other benchmark arms. +""" + +import os +from typing import override + +from harbor.agents.installed.claude_code import ClaudeCode +from harbor.environments.base import BaseEnvironment +from harbor.models.agent.context import AgentContext +from harbor.models.agent.name import AgentName +from harbor.models.trial.paths import EnvironmentPaths + +# Host path to the static-musl rtk binary uploaded into each container. +RTK_BIN_HOST = os.environ.get("RTK_BIN_HOST", "/tmp/rtk-runs/rtk") + + +class ClaudeCodeRTK(ClaudeCode): + @staticmethod + @override + def name() -> str: + return AgentName.CLAUDE_CODE_RTK.value + + @override + async def install(self, environment: BaseEnvironment) -> None: + # Stock Claude Code install first (curl/npm bootstrap of the CLI). + await super().install(environment) + + # Deliver the rtk binary (static-musl, runs on any glibc/musl image). + await environment.upload_file(RTK_BIN_HOST, "/usr/local/bin/rtk") + await self.exec_as_root( + environment, + command="chmod 755 /usr/local/bin/rtk && /usr/local/bin/rtk --version", + ) + # Belt-and-suspenders: Claude Code launches with PATH prepending + # ~/.local/bin, and both the PreToolUse hook (`rtk hook claude`) and the + # rewritten commands (`rtk pytest`) must resolve `rtk`. Symlink it there + # too so it is found regardless of whether /usr/local/bin is on PATH. + await self.exec_as_agent( + environment, + command='mkdir -p "$HOME/.local/bin" && ln -sf /usr/local/bin/rtk "$HOME/.local/bin/rtk"', + ) + + @override + async def run( + self, instruction: str, environment: BaseEnvironment, context: AgentContext + ) -> None: + # Install the rtk PreToolUse hook into the SAME config dir Claude Code + # will read (claude_code.py sets CLAUDE_CONFIG_DIR to this path). rtk + # init honors CLAUDE_CONFIG_DIR; --auto-patch is REQUIRED for headless + # (piped stdin otherwise defaults the settings-patch to "N"). + config_dir = (EnvironmentPaths.agent_dir / "sessions").as_posix() + await self.exec_as_agent( + environment, + command=( + 'export PATH="/usr/local/bin:$HOME/.local/bin:$PATH"; ' + "mkdir -p /logs/agent " + f'"{config_dir}"; ' + f'CLAUDE_CONFIG_DIR="{config_dir}" ' + "rtk init -g --auto-patch " + "> /logs/agent/rtk-init.log 2>&1 || true; " + # Snapshot the installed hook so we can confirm it fired. + f'cp "{config_dir}/settings.json" /logs/agent/rtk-settings.json ' + "2>/dev/null || true" + ), + env={"RTK_TELEMETRY_DISABLED": "1"}, + ) + + # Run the stock Claude Code agent (its own setup_command only mkdir's + # under CLAUDE_CONFIG_DIR, so it does not clobber settings.json). + await super().run(instruction, environment, context) + + # Dump rtk's own savings ledger for this trial. The rewritten bash-tool + # commands ran as the agent user and tracked to the default per-user DB + # (~/.local/share/rtk/history.db); read it back the same way. + await self.exec_as_agent( + environment, + command=( + 'export PATH="/usr/local/bin:$HOME/.local/bin:$PATH"; ' + "rtk gain --all --format json > /logs/agent/rtk-gain.json " + "2>/dev/null || true; " + "rtk gain --history > /logs/agent/rtk-history.txt 2>/dev/null || true" + ), + env={"RTK_TELEMETRY_DISABLED": "1"}, + ) diff --git a/deploy/harbor/deep_analysis.py b/deploy/harbor/deep_analysis.py index 8fb7436..ed8a63e 100644 --- a/deploy/harbor/deep_analysis.py +++ b/deploy/harbor/deep_analysis.py @@ -20,7 +20,14 @@ "baseline": ("/tmp/cg-runs/final50/rows-off.json", None), "context-guru": ("/tmp/cg-runs/final50-v6/rows-codesmart.json", "/tmp/cg-runs/final50-v6/summary.json"), "headroom": ("/tmp/hd-runs/swe50/rows-hd-cache.json", None), + # rtk (Rust Token Killer): in-container bash-output compression via a Claude + # Code PreToolUse hook. A 4th arm; its rows carry an extra `rtk` sub-dict with + # rtk's own bash-output savings ledger. Optional — only included if present, so + # the published 3-way matched set is unchanged unless the rtk run is supplied. + "rtk": ("/tmp/rtk-runs/swe50/rows-rtk.json", "/tmp/rtk-runs/swe50/summary.json"), } +# Drop arms whose rows file is missing so the script still runs pre-rtk-run. +SRC = {k: v for k, v in SRC.items() if Path(v[0]).exists()} def billed(r): @@ -108,6 +115,41 @@ def agg(c, key): except Exception: pass + # rtk tool metrics: $0 own-LLM cost + deterministic; its native metric is + # bash-OUTPUT bytes saved (bytes/4 estimate), a DIFFERENT denominator than the + # proxies' whole-request content%, so it is reported separately. Reversibility + # is a tee-file on failure (no expand/retrieve bounce), so bounces=0. + if "rtk" in configs: + rrows = json.load(open(SRC["rtk"][0])) + bycmd = {} + for r in rrows: + for c in ((r.get("rtk") or {}).get("by_command") or []): + name = c.get("command") or c.get("name") or "?" + key = " ".join(str(name).split()[:2]) # e.g. "rtk pytest" + e = bycmd.setdefault(key, {"count": 0, "saved": 0}) + e["count"] += c.get("count") or 0 + e["saved"] += c.get("saved") or c.get("saved_tokens") or 0 + try: + rs = next(x for x in json.load(open(SRC["rtk"][1]))["configs"]) + except Exception: + rs = {} + # Prefer the per-command aggregate built from the rtk `gain --history` "By + # Command" tables (the --format json output has no per-command array); fall + # back to whatever the rows carried. + try: + bc = json.load(open("/tmp/rtk-runs/rtk_by_command.json")) + bycmd = {k: {"count": v["count"], "saved": v["saved"]} for k, v in bc.items()} + except Exception: + pass + tool["rtk"] = dict( + llm_cost=0.0, added_ms=0.0, bounces=0, + content_pct=rs.get("rtk_bash_savings_pct"), # NB: bash-output denom, not whole-request + bash_tokens_before=rs.get("rtk_bash_tokens_before"), + bash_tokens_after=rs.get("rtk_bash_tokens_after"), + bash_tokens_saved=rs.get("rtk_bash_tokens_saved"), + commands=rs.get("rtk_commands"), + per_command=dict(sorted(bycmd.items(), key=lambda x: -x[1]["saved"]))) + out = dict(matched_tasks=len(common), configs=configs, aggregate=aggregate, tool=tool, per_task=per_task) Path(f"{a.out}/deep_analysis.json").write_text(json.dumps(out, indent=1)) diff --git a/deploy/harbor/deep_plots.py b/deploy/harbor/deep_plots.py index 4af6d62..5e88c09 100644 --- a/deploy/harbor/deep_plots.py +++ b/deploy/harbor/deep_plots.py @@ -12,8 +12,8 @@ import matplotlib.pyplot as plt SURFACE = "#fcfcfb"; INK = "#0b0b0b"; INK2 = "#52514e"; GRID = "#e6e5e2" -# baseline grey, context-guru blue, headroom orange -COL = {"baseline": "#8a8985", "context-guru": "#2a78d6", "headroom": "#eb6834"} +# baseline grey, context-guru blue, headroom orange, rtk teal-green +COL = {"baseline": "#8a8985", "context-guru": "#2a78d6", "headroom": "#eb6834", "rtk": "#1baf7a"} COMP = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300"] plt.rcParams.update({ "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE, @@ -58,7 +58,7 @@ def fig_headline(A, out): ax[1][2].set_xticks(range(len(cfgs))); ax[1][2].set_xticklabels(cfgs, fontsize=9) ax[1][2].set_title("Compaction latency added / request"); ax[1][2].set_ylabel("ms") ax[1][2].grid(axis="x", visible=False); style(ax[1][2]) - fig.suptitle("baseline vs context-guru vs headroom — SWE-bench Verified (matched %d tasks)" % A["matched_tasks"], + fig.suptitle("%s — SWE-bench Verified (matched %d tasks)" % (" vs ".join(cfgs), A["matched_tasks"]), fontsize=13, fontweight="bold") fig.tight_layout(); fig.savefig(out); plt.close(fig) @@ -87,11 +87,14 @@ def fig_per_task_cost(A, out): rows = sorted(A["per_task"], key=lambda r: r["baseline"]["cost"]) n = len(rows) fig, ax = plt.subplots(figsize=(8, max(6, n * 0.22))) + lo = lambda r: min(r[c]["cost"] for c in A["configs"] if c != "baseline") + hi = lambda r: max(r[c]["cost"] for c in A["configs"] if c != "baseline") for i, r in enumerate(rows): - ax.plot([r["headroom"]["cost"], r["context-guru"]["cost"]], [i, i], color=GRID, lw=1.5, zorder=1) - for c, mk in [("baseline", "|"), ("headroom", "o"), ("context-guru", "o")]: - ax.scatter([r[c]["cost"] for r in rows], range(n), s=30, color=COL[c], zorder=3, label=c, - marker=mk if c != "baseline" else "D") + ax.plot([lo(r), hi(r)], [i, i], color=GRID, lw=1.5, zorder=1) + markers = {"baseline": "D", "headroom": "o", "context-guru": "o", "rtk": "s"} + for c in A["configs"]: + ax.scatter([r[c]["cost"] for r in rows], range(n), s=26, color=COL[c], zorder=3, label=c, + marker=markers.get(c, "o")) ax.set_yticks(range(n)); ax.set_yticklabels([r["task"] for r in rows], fontsize=6) ax.set_xlabel("billed input cost per task ($)"); ax.set_title("Per-task cost") ax.legend(loc="lower right", frameon=False, fontsize=8); ax.grid(axis="y", visible=False); style(ax) @@ -118,7 +121,9 @@ def fig_components(A, out): cg = A["tool"]["context-guru"] uq = (cg.get("unique") or {}).get("components", {}) hr = A["tool"]["headroom"].get("per_strategy", {}) - fig, axes = plt.subplots(1, 2, figsize=(11, 3.6)) + rtk = (A["tool"].get("rtk") or {}).get("per_command", {}) + npanel = 3 if rtk else 2 + fig, axes = plt.subplots(1, npanel, figsize=(5.5 * npanel, 3.6)) # context-guru: cumulative vs unique items = [(k, v.get("cum", 0), (uq.get(k, {}) or {}).get("uniq_saved", 0)) for k, v in cg["per_component"].items() if v.get("cum")] # map dump categories to component names best-effort @@ -138,6 +143,16 @@ def fig_components(A, out): ax.set_yticks(range(len(hs))); ax.set_yticklabels([k for k, _ in hs], fontsize=8) ax.set_xlabel("tokens removed"); ax.set_title("headroom — per compressor") ax.grid(axis="y", visible=False); style(ax) + # rtk per-command (bash-output tokens saved) + if rtk: + rs = sorted([(k, v.get("saved", 0)) for k, v in rtk.items() if v.get("saved")], key=lambda x: x[1]) + ax = axes[2] + ax.barh(range(len(rs)), [v for _, v in rs], color="#1baf7a", height=0.6, zorder=2) + for i, (k, v) in enumerate(rs): + ax.text(v, i, f" {int(v):,}", va="center", fontsize=8, color=INK) + ax.set_yticks(range(len(rs))); ax.set_yticklabels([k for k, _ in rs], fontsize=8) + ax.set_xlabel("bash-output tokens removed"); ax.set_title("rtk — per command") + ax.grid(axis="y", visible=False); style(ax) fig.tight_layout(); fig.savefig(out); plt.close(fig) diff --git a/deploy/harbor/swebench_rtk.py b/deploy/harbor/swebench_rtk.py new file mode 100644 index 0000000..6b23d23 --- /dev/null +++ b/deploy/harbor/swebench_rtk.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""SWE-bench Verified harness for RTK (Rust Token Killer) — a 4th arm for the +three-way study, using IDENTICAL trajectory accounting to swebench.py so results +are directly comparable to baseline / context-guru / headroom. + +rtk is NOT a request-stream proxy. It is a Claude Code ``PreToolUse`` hook that +rewrites Bash commands (``pytest`` -> ``rtk pytest``, ``cat`` -> ``rtk read``, +``git status`` -> ``rtk git status``) INSIDE the task container, compressing bash +output at the shell before it enters the model context. So there is nothing to +proxy for compaction — model routing is made IDENTICAL to the baseline by +running the same context-guru ``off`` passthrough proxy on :4000. The ONLY +difference from baseline is the in-container bash compression, delivered by the +custom ``claude-code-rtk`` Harbor agent (see +harbor/src/harbor/agents/installed/claude_code_rtk.py). + +Per config it: (1) starts cg-proxy-d1 with preset ``off`` (pure passthrough, +FORCE_MODEL=sonnet-5) — same routing as the baseline ``rows-off.json``; +(2) runs harbor with ``-a claude-code-rtk`` (uploads the rtk binary + installs +the PreToolUse hook in-container), passing RTK_BIN_HOST so the agent finds the +host-built static-musl binary; (3) parses each trial's result.json + trajectory +final_metrics for reward/steps/cache-aware tokens/cost (identical to swebench.py); +(4) reads each trial's /logs/agent/rtk-gain.json for rtk's OWN bash-output +savings ledger. Writes rows-.json + summary.json under the jobs-root. + +The ``off`` baseline is NOT re-run — reuse /tmp/cg-runs/final50/rows-off.json. + +Usage: + swebench_rtk.py --tasks /tmp/cg-runs/swe3-verify.txt --jobs-root /tmp/rtk-runs/swe3 --n 2 +""" +import argparse, glob, json, os, subprocess, sys, time, urllib.request +from pathlib import Path + +CG = Path("/home/vpcuser/projects/context-engineering/context-guru") +HB = Path("/home/vpcuser/projects/context-engineering/harbor") +BIN = "/tmp/cg-runs/cg-proxy-d1" # context-guru proxy (off = passthrough) +RTK_BIN = os.environ.get("RTK_BIN_HOST", "/tmp/rtk-runs/rtk") # host static-musl rtk +PORT = 4000 +LAN = "9.47.170.83" +PRICES_URL = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json" +MODEL = "aws/claude-sonnet-5" + + +def creds(): + e = json.load(open(Path("~/.claude/settings.json").expanduser()))["env"] + return e["ANTHROPIC_BASE_URL"], e["ANTHROPIC_AUTH_TOKEN"] + + +def price(model): + fb = (2e-6, 1e-5, 2e-7, 2.5e-6) # in, out, cache_read, cache_write + try: + d = json.load(urllib.request.urlopen(PRICES_URL, timeout=15)) + c = d.get(model) or d.get(model.split("/")[-1]) + if c: + return (c.get("input_cost_per_token") or fb[0], c.get("output_cost_per_token") or fb[1], + c.get("cache_read_input_token_cost") or fb[2], + c.get("cache_creation_input_token_cost") or (c.get("input_cost_per_token") or fb[0]) * 1.25) + except Exception as e: + print(f"[price] {e}; fallback", file=sys.stderr) + return fb + + +def stop_proxy(): + for _ in range(3): + subprocess.run("pkill -x cg-proxy-d1", shell=True) + time.sleep(1) + r = subprocess.run("pgrep -x cg-proxy-d1", shell=True, capture_output=True) + if not r.stdout.strip(): + return + time.sleep(2) + + +def start_proxy(base, token): + """Start cg-proxy-d1 as a pure passthrough (preset off) — identical routing + to the baseline, so the only variable is rtk's in-container compression.""" + stop_proxy() + env = dict(os.environ, ANTHROPIC_UPSTREAM=base, ANTHROPIC_API_KEY=token, + OPENAI_UPSTREAM=base, OPENAI_API_KEY=token, FORCE_MODEL=MODEL, + LISTEN_ADDR=f":{PORT}") + log = open("/tmp/rtk-runs/proxy-off.log", "w") + p = subprocess.Popen([BIN, "--preset", "off"], cwd=str(CG), stdout=log, stderr=log, + env=env, preexec_fn=os.setsid) + for _ in range(30): + try: + urllib.request.urlopen(f"http://localhost:{PORT}/healthz", timeout=3).read() + return p + except Exception: + time.sleep(0.5) + raise RuntimeError("off proxy did not come up") + + +def run_harbor(tasks, jobs_dir, n, setup_mult, build_mult, agent_mult): + proxy_url = f"http://{LAN}:{PORT}/anthropic" + inc = " ".join(f"-i {t}" for t in tasks) + home = os.path.expanduser("~") + abs_path = f"{home}/.local/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + # RTK_BIN_HOST tells the claude-code-rtk agent where the host rtk binary is. + cmd = (f"cd {HB} && ANTHROPIC_BASE_URL='{proxy_url}' ANTHROPIC_API_KEY='sk-proxy' " + f"ANTHROPIC_AUTH_TOKEN='sk-proxy' RTK_BIN_HOST='{RTK_BIN}' PATH='{abs_path}' HOME='{home}' " + f"{home}/.local/bin/uv run harbor run -y -d swebench-verified@1.0 -a claude-code-rtk -m '{MODEL}' " + f"--env docker {inc} -n {n} --jobs-dir '{jobs_dir}' --no-delete " + f"--agent-setup-timeout-multiplier {setup_mult} --environment-build-timeout-multiplier {build_mult} " + f"--agent-timeout-multiplier {agent_mult} --max-retries 2 " + f"--ae ANTHROPIC_BASE_URL='{proxy_url}' --ae ANTHROPIC_API_KEY='sk-proxy' --ae ANTHROPIC_AUTH_TOKEN='sk-proxy'") + log = f"/tmp/rtk-runs/run-{Path(jobs_dir).name}.log" + with open(log, "w") as f: + subprocess.run(["sg", "docker", "-c", cmd], stdout=f, stderr=f) + return log + + +def iso(s): + from datetime import datetime + return datetime.fromisoformat(s.replace("Z", "+00:00")).timestamp() + + +def read_rtk_gain(tdir): + """rtk's OWN bash-output savings ledger for this trial (bytes/4 estimate).""" + gp = tdir / "agent" / "rtk-gain.json" + if not gp.exists(): + return None + try: + g = json.load(open(gp)) + except Exception: + return None + s = g.get("summary") or {} + by_cmd = g.get("byCommand") or g.get("by_command") or [] + return dict(commands=s.get("total_commands"), input_tokens=s.get("total_input"), + output_tokens=s.get("total_output"), saved_tokens=s.get("total_saved"), + savings_pct=s.get("avg_savings_pct"), total_time_ms=s.get("total_time_ms"), + by_command=by_cmd) + + +def parse_trials(jobs_dir, pr): + rows = [] + for rf in glob.glob(f"{jobs_dir}/*/*/result.json"): + try: + d = json.load(open(rf)) + except Exception: + continue + if "verifier_result" not in d: + continue + tdir = Path(rf).parent + reward = ((d.get("verifier_result") or {}).get("rewards") or {}).get("reward") + fm = {} + traj = tdir / "agent" / "trajectory.json" + if traj.exists(): + try: + fm = (json.load(open(traj)) or {}).get("final_metrics") or {} + except Exception: + fm = {} + ex = fm.get("extra") or {} + pt = fm.get("total_prompt_tokens") or 0 + ct = fm.get("total_completion_tokens") or 0 + cached = fm.get("total_cached_tokens") or 0 + cwrite = ex.get("total_cache_creation_input_tokens") or 0 + cread = ex.get("total_cache_read_input_tokens") or cached + fresh = max(pt - cread - cwrite, 0) + norm_cost = fresh * pr[0] + ct * pr[1] + cread * pr[2] + cwrite * pr[3] + wall = None + try: + wall = iso(d["finished_at"]) - iso(d["started_at"]) + except Exception: + pass + agent_wall = None + try: + ae = d.get("agent_execution") or {} + agent_wall = iso(ae["finished_at"]) - iso(ae["started_at"]) + except Exception: + pass + rows.append(dict(task=d.get("task_name", tdir.parent.name), reward=reward, + steps=fm.get("total_steps"), prompt_tokens=pt, completion_tokens=ct, + cached_tokens=cached, cache_read=cread, cache_write=cwrite, fresh_input=fresh, + agent_cost=fm.get("total_cost_usd"), norm_cost=round(norm_cost, 5), + wall_s=round(wall, 1) if wall else None, + agent_wall_s=round(agent_wall, 1) if agent_wall else None, + exception=bool(d.get("exception_info")), + rtk=read_rtk_gain(tdir))) + return rows + + +def summarize(cfg, rows): + got = [r for r in rows if r["reward"] is not None] + solved = sum(1 for r in got if r["reward"] and r["reward"] >= 1) + def avg(k): + vs = [r[k] for r in rows if isinstance(r.get(k), (int, float))] + return round(sum(vs) / len(vs), 3) if vs else None + tot = lambda k: sum(r[k] for r in rows if isinstance(r.get(k), (int, float))) + cacheable = tot("cache_read") + tot("fresh_input") + tot("cache_write") + # aggregate rtk's own ledger across trials + rk = [r["rtk"] for r in rows if r.get("rtk")] + rtk_before = sum((x.get("input_tokens") or 0) for x in rk) + rtk_after = sum((x.get("output_tokens") or 0) for x in rk) + rtk_saved = sum((x.get("saved_tokens") or 0) for x in rk) + rtk_cmds = sum((x.get("commands") or 0) for x in rk) + return dict(config=cfg, trials=len(rows), scored=len(got), solved=solved, + solve_rate=round(solved / len(got), 3) if got else None, + exceptions=sum(1 for r in rows if r["exception"]), + mean_steps=avg("steps"), mean_prompt_tokens=avg("prompt_tokens"), + mean_completion_tokens=avg("completion_tokens"), + cache_hit_rate=round(tot("cache_read") / cacheable, 4) if cacheable else None, + total_fresh_input=tot("fresh_input"), total_cache_read=tot("cache_read"), + total_cache_write=tot("cache_write"), total_completion=tot("completion_tokens"), + total_norm_cost=round(tot("norm_cost"), 4), mean_norm_cost=avg("norm_cost"), + mean_agent_cost=avg("agent_cost"), mean_wall_s=avg("wall_s"), + mean_agent_wall_s=avg("agent_wall_s"), + rtk_trials_with_ledger=len(rk), rtk_commands=rtk_cmds, + rtk_bash_tokens_before=rtk_before, rtk_bash_tokens_after=rtk_after, + rtk_bash_tokens_saved=rtk_saved, + rtk_bash_savings_pct=round(100 * rtk_saved / rtk_before, 2) if rtk_before else None) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--tasks", required=True) + ap.add_argument("--config", default="rtk", help="label for this arm") + ap.add_argument("--jobs-root", default="/tmp/rtk-runs/swebench") + ap.add_argument("--n", type=int, default=2) + ap.add_argument("--setup-mult", type=float, default=4.0) + ap.add_argument("--build-mult", type=float, default=4.0) + ap.add_argument("--agent-mult", type=float, default=1.5) + a = ap.parse_args() + base, token = creds() + pr = price(MODEL) + tasks = [t.strip() for t in open(a.tasks) if t.strip()] + Path(a.jobs_root).mkdir(parents=True, exist_ok=True) + print(f"SWE-bench(rtk): {len(tasks)} tasks (n={a.n}) | rtk_bin={RTK_BIN} | " + f"price in=${pr[0]*1e6:.2f} out=${pr[1]*1e6:.2f} cread=${pr[2]*1e6:.2f} cwrite=${pr[3]*1e6:.2f}/M\n", flush=True) + if not Path(RTK_BIN).exists(): + raise SystemExit(f"rtk binary not found at {RTK_BIN}; set RTK_BIN_HOST") + cfg = a.config + jobs = f"{a.jobs_root}/{cfg}" + subprocess.run(f"rm -rf {jobs}", shell=True) + print(f"### config={cfg} (rtk in-container hook; off-proxy routing) ...", flush=True) + start_proxy(base, token) + t0 = time.time() + run_harbor(tasks, jobs, a.n, a.setup_mult, a.build_mult, a.agent_mult) + rows = parse_trials(jobs, pr) + Path(f"{a.jobs_root}/rows-{cfg}.json").write_text(json.dumps(rows, indent=1)) + stop_proxy() + s = summarize(cfg, rows) + s["wall_total_min"] = round((time.time() - t0) / 60, 1) + Path(f"{a.jobs_root}/summary.json").write_text(json.dumps( + dict(model=MODEL, price=pr, tasks=len(tasks), configs=[s]), indent=1)) + print(json.dumps(s, indent=1), flush=True) + print("\n==== SUMMARY ====") + print(f"{cfg:<12} solved={s['solved']}/{s['scored']} rate={s['solve_rate']} " + f"steps={s['mean_steps']} cache_hit={s['cache_hit_rate']} " + f"norm$/t={s['mean_norm_cost']} exceptions={s['exceptions']}") + print(f" rtk ledger: {s['rtk_trials_with_ledger']}/{len(rows)} trials, {s['rtk_commands']} cmds, " + f"bash tokens {s['rtk_bash_tokens_before']}->{s['rtk_bash_tokens_after']} " + f"(saved {s['rtk_bash_tokens_saved']} = {s['rtk_bash_savings_pct']}%)") + print(f"\nwrote {a.jobs_root}/summary.json + rows-{cfg}.json") + + +if __name__ == "__main__": + main() diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 2838e3a..ecfa385 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -2,43 +2,47 @@ **context-guru is the cheapest and highest-reward context-compaction layer on SWE-bench Verified** — evaluated live, end-to-end, with the **claude-code** agent on -**`aws/claude-sonnet-5`**, against a no-compaction baseline and against -[**headroom**](https://pypi.org/project/headroom-ai/) (`headroom-ai` v0.32.1). +**`aws/claude-sonnet-5`**, against a no-compaction baseline, against +[**headroom**](https://pypi.org/project/headroom-ai/) (a request-stream proxy), and against +[**rtk**](https://github.com/rtk-ai/rtk) (Rust Token Killer, a shell-level Bash-output hook). -50 tasks, all of which scored under all three arms (zero infrastructure exceptions). +50 tasks, all of which scored under all **four** arms (zero infrastructure exceptions). ![headline](img/benchmark/headline.png) -| dimension | baseline | **context-guru** | headroom | -|---|--:|--:|--:| -| reward (solved / 50) | 43 | **44** | 40 | -| **billed cost** (matched total) | $31.98 | **$27.77 (−13.2%)** | $30.30 (−5.3%) | -| cache-read tokens | 102.8M | **84.5M** | 96.4M | -| cache-write tokens | 1.855M | 1.847M | 1.839M | -| mean steps / task | 36.1 | **31.1** | 35.1 | -| added latency / req | — | 117 ms | 63 ms | -| tool LLM cost | $0 | $0.31 | $0 | +| dimension | baseline | **context-guru** | headroom | rtk | +|---|--:|--:|--:|--:| +| reward (solved / 50) | 43 | **44** | 40 | 43 | +| **billed cost** (matched total) | $31.98 | **$27.77 (−13.2%)** | $30.30 (−5.3%) | $29.09 (−9.0%) | +| cache-read tokens | 102.8M | **84.5M** | 96.4M | 91.7M | +| cache-write tokens | 1.855M | 1.847M | 1.839M | 1.835M | +| mean steps / task | 36.1 | **31.1** | 35.1 | 33.2 | +| added latency / req | — | 117 ms | 63 ms | **0 ms** | +| tool LLM cost | $0 | $0.31 | $0 | $0 |
-**context-guru wins on cost, cache usage, steps, and reward-vs-headroom.** headroom keeps -an edge on added latency and tool cost (it is fully deterministic). The reason -context-guru is cheaper despite removing less *raw* content per request: it **freezes -each compaction and re-applies it byte-identically every turn**, so the reduction -compounds across the whole session's cache-reads while never mutating the cached prefix. +**context-guru wins on cost, cache usage, steps, and reward.** It is cheaper despite +removing less *raw* content per request because it **freezes each compaction and re-applies +it byte-identically every turn**, so the reduction compounds across the session's +cache-reads while never mutating the cached prefix. The surprise is **rtk**: a simple +deterministic shell filter is the **2nd-cheapest** arm (−9.0%), **reward-neutral** (43 = 43), +at **zero request-path latency and $0 tool cost** — it **beats the headroom proxy on both +cost and reward**. Its ceiling is that it only compresses **Bash-tool** output (built-in +`Read`/`Grep`/`Glob` bypass its hook), which is why the whole-request proxy goes deeper. ## The results suite -- **[Full comparison](results/comparison.md)** — all dimensions, cost decomposition, +- **[Full comparison](results/comparison.md)** — all four arms, cost decomposition, per-task plots, per-component breakdown, and the honest caveats. - **[Component internals & real examples](results/components.md)** — how every - context-guru component and headroom compressor works, when it triggers, and real - before→after compactions from the run logs, side by side. + context-guru component, headroom compressor, and rtk command filter works, when it + triggers, and real before→after compactions from the run logs, side by side. - Per-config detail (per-task tables + totals): [baseline](results/baseline.md) · [context-guru](results/context-guru.md) · - [headroom](results/headroom.md). -- **[Reproduce](results/REPRODUCE.md)** — install and run all three arms yourself. + [headroom](results/headroom.md) · [rtk](results/rtk.md). +- **[Reproduce](results/REPRODUCE.md)** — install and run all four arms yourself. Method note: cache-aware billed **input** cost = fresh $2/M · cache-read $0.20/M · cache-write $2.50/M (recomputed from each trial's own token tiers) + output $10/M; diff --git a/docs/img/benchmark/components.png b/docs/img/benchmark/components.png index 1ad5c1d..07e61d7 100644 Binary files a/docs/img/benchmark/components.png and b/docs/img/benchmark/components.png differ diff --git a/docs/img/benchmark/cost_decomposition.png b/docs/img/benchmark/cost_decomposition.png index 9d4aa43..366c7ab 100644 Binary files a/docs/img/benchmark/cost_decomposition.png and b/docs/img/benchmark/cost_decomposition.png differ diff --git a/docs/img/benchmark/headline.png b/docs/img/benchmark/headline.png index c1fb31c..360796e 100644 Binary files a/docs/img/benchmark/headline.png and b/docs/img/benchmark/headline.png differ diff --git a/docs/img/benchmark/per_task_cost.png b/docs/img/benchmark/per_task_cost.png index 47cc19a..89a4ddd 100644 Binary files a/docs/img/benchmark/per_task_cost.png and b/docs/img/benchmark/per_task_cost.png differ diff --git a/docs/javascripts/charts.js b/docs/javascripts/charts.js index fc8f774..a1bbbd2 100644 --- a/docs/javascripts/charts.js +++ b/docs/javascripts/charts.js @@ -6,9 +6,9 @@ // deploy/eval-containers/results/*.csv if the numbers should track the sweep automatically. // Matched 50-task totals, from docs/results/comparison.md. -const CG_ARMS = ["baseline", "context-guru", "headroom"]; -const CG_COST = [31.98, 27.77, 30.30]; // total billed $ (lower is better) -const CG_CACHE_READ = [102.8, 84.5, 96.4]; // cache-read tokens, millions (lower is better) +const CG_ARMS = ["baseline", "context-guru", "headroom", "rtk"]; +const CG_COST = [31.98, 27.77, 30.30, 29.09]; // total billed $ (lower is better) +const CG_CACHE_READ = [102.8, 84.5, 96.4, 91.7]; // cache-read tokens, millions (lower is better) function cgTeal(alpha) { return `rgba(0, 150, 136, ${alpha})`; diff --git a/docs/results/REPRODUCE.md b/docs/results/REPRODUCE.md index 664d0da..f877fa3 100644 --- a/docs/results/REPRODUCE.md +++ b/docs/results/REPRODUCE.md @@ -93,10 +93,55 @@ python3 /tmp/hd-runs/swebench_headroom.py --tasks /tmp/cg-runs/swe50.txt \ Use a **different proxy port** (e.g. 4010) and jobs-root than context-guru so the two never collide; reuse the same authenticated Docker Hub + `--no-delete`. +## 4b. Run rtk (Rust Token Killer) + +rtk is **not** a request-stream proxy — it is a Claude Code **`PreToolUse` hook** that +rewrites Bash commands (`pytest …` → `rtk pytest …`, `cat f` → `rtk read f`, +`git status` → `rtk git status`) **inside the task container**, compressing bash output +at the shell before it enters the model context. So there is nothing to proxy for +compaction: model routing is made **identical to the baseline** by running the same +context-guru `off` passthrough proxy on `:4000`. The only difference from baseline is +the in-container bash compression. + +**1) Fetch the rtk binary** (static-musl, runs on any SWE-bench image): + +``` +mkdir -p /tmp/rtk-runs && cd /tmp/rtk-runs +curl -fsSL -o rtk.tgz https://github.com/rtk-ai/rtk/releases/download/v0.43.0/rtk-x86_64-unknown-linux-musl.tar.gz +tar xzf rtk.tgz && chmod +x rtk && ./rtk --version # rtk 0.43.0 +``` + +**2) Add the `claude-code-rtk` agent to the Harbor checkout** (one new file + two lines). +The agent subclasses the stock `claude-code` agent and, per trial: uploads the rtk binary +to `/usr/local/bin/rtk`, installs the rtk `PreToolUse` hook into the exact +`CLAUDE_CONFIG_DIR` Claude Code reads (`rtk init -g --auto-patch`, which honors that +env var — `--auto-patch` is **required** headless, else the piped-stdin patch defaults to +"no"), and dumps `rtk gain --all --format json` to `/logs/agent/rtk-gain.json`. + +- add `harbor/src/harbor/agents/installed/claude_code_rtk.py` (a copy lives at + [`deploy/harbor/claude_code_rtk_agent.py`](https://github.com/rossoctl/context-guru/blob/main/deploy/harbor/claude_code_rtk_agent.py)); +- register it: `CLAUDE_CODE_RTK = "claude-code-rtk"` in `harbor/src/harbor/models/agent/name.py` + and `AgentName.CLAUDE_CODE_RTK: "harbor.agents.installed.claude_code_rtk:ClaudeCodeRTK"` + in `harbor/src/harbor/agents/factory.py`. + +**3) Run the 50 tasks** (starts the `off` proxy, runs `-a claude-code-rtk`): + +``` +cd /home/vpcuser/projects/context-engineering/context-guru +RTK_BIN_HOST=/tmp/rtk-runs/rtk python3 -u deploy/harbor/swebench_rtk.py \ + --tasks /tmp/cg-runs/swe50.txt --jobs-root /tmp/rtk-runs/swe50 --n 2 +``` + +End-to-end metrics (reward, steps, cache-read/write, billed cost, cache-hit, wall) come +from the unchanged claude-code trajectory parser — measured **identically** to the other +three arms. Each trial's `agent/rtk-gain.json` also carries rtk's own bash-output savings +ledger (its native `bytes/4` estimate, a bash-output denominator — **not** the whole-request +content% the proxies report). + ## 5. Analyze & plot ``` -# Three-way matched analysis (baseline vs context-guru vs headroom) — every dimension, +# Four-way matched analysis (baseline vs context-guru vs headroom vs rtk) — every dimension, # per-task + aggregate + per-component, cumulative & unique tokens → deep_analysis.json: python3 deploy/harbor/deep_analysis.py --out /tmp/cg-runs/deep # Figures (validated CVD-safe palette) → docs/img/benchmark/: @@ -106,9 +151,12 @@ python3 deploy/harbor/gen_result_docs.py "