Skip to content

docs(benchmark): add rtk (Rust Token Killer) as a 4th SWE-bench arm #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
OsherElhadad merged 1 commit into from
Jul 27, 2026
Merged

docs(benchmark): add rtk (Rust Token Killer) as a 4th SWE-bench arm #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 22 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,25 +32,29 @@ runs, projecting large reads down to what's relevant — and every reduction is
## Benchmark: the cheapest & highest-reward arm on SWE-bench Verified

Evaluated **live, end-to-end**, with the **claude-code** agent on **`aws/claude-sonnet-5`**, against a
no-compaction baseline and against [**headroom**](https://pypi.org/project/headroom-ai/)
(`headroom-ai` v0.32.1). All 50 tasks scored under all three arms.

| dimension | baseline | **context-guru** | headroom |
|---|--:|--:|--:|
| tasks solved | 86% | **88%** | 80% |
| **total billed cost** vs baseline | — | **−13.2%** | −5.3% |
| cache-read tokens vs baseline | — | **−17.8%** | −6.3% |
| cache-write tokens vs baseline | — | −0.4% | −0.9% |
| mean steps / task vs baseline | — | **−13.9%** | −2.8% |
| added latency / req | — | 117 ms | 63 ms |
no-compaction baseline, against the [**headroom**](https://pypi.org/project/headroom-ai/) request-stream
proxy, and against [**rtk**](https://github.com/rtk-ai/rtk) (Rust Token Killer, a shell-level Bash-output
hook). All 50 tasks scored under all **four** arms.

| dimension | baseline | **context-guru** | headroom | rtk |
|---|--:|--:|--:|--:|
| tasks solved | 86% | **88%** | 80% | 86% |
| **total billed cost** vs baseline | — | **−13.2%** | −5.3% | −9.0% |
| cache-read tokens vs baseline | — | **−17.8%** | −6.3% | −10.8% |
| cache-write tokens vs baseline | — | −0.4% | −0.9% | −1.1% |
| mean steps / task vs baseline | — | **−13.9%** | −2.8% | −8.0% |
| added latency / req | — | 117 ms | 63 ms | **0 ms** |
| tool's own LLM cost | — | $0.31 | $0 | $0 |

**context-guru is the cheapest arm and solves the most tasks** — it cuts billed cost **13.2%** vs no
compaction (headroom cuts only **5.3%**), driven by an **17.8%** cache-read reduction, while keeping
cache-write within **1%** of baseline (it never busts the cache). It does this by *freezing each compaction
and replaying it byte-identically every turn*, so the saving compounds across the whole session. headroom
keeps an edge on added latency (it is fully deterministic — no model on the hot path). Full three-way study,
per-task/per-component breakdowns, real before→after examples, and how to reproduce:
**[docs/RESULTS.md](docs/RESULTS.md)**.
compaction, driven by an **17.8%** cache-read reduction, while keeping cache-write within **1%** of baseline
(it never busts the cache). It does this by *freezing each compaction and replaying it byte-identically every
turn*, so the saving compounds across the whole session. The surprise is **rtk**: a simple deterministic
shell filter is the **2nd-cheapest** arm (**−9.0%**), **reward-neutral** (86% = baseline), at **zero
request-path latency and $0 tool cost** — it **beats the headroom proxy on both cost and reward**. rtk's
ceiling is that it only compresses **Bash-tool** output (Claude Code's built-in `Read`/`Grep`/`Glob` bypass
its hook), which is why the whole-request proxy goes deeper. Full four-way study, per-task/per-component
breakdowns, real before→after examples, and how to reproduce: **[docs/RESULTS.md](docs/RESULTS.md)**.

## Architecture

Expand Down Expand Up @@ -182,7 +186,7 @@ Details in [docs/integrations.md](docs/integrations.md).
- [docs/components.md](docs/components.md) — every registered component: how it works, live before→after, lossiness, config, best use.
- [docs/integrations.md](docs/integrations.md) — proxy gateway vs AuthBridge plugin, with request paths.
- [docs/setup.md](docs/setup.md) — setup + a concrete SWE-bench run through the eval-containers gateway.
- [docs/RESULTS.md](docs/RESULTS.md) — the live three-way SWE-bench Verified benchmark (Claude Code, `aws/claude-sonnet-5`): context-guru is the cheapest arm (−13.2% billed cost vs baseline) and solves the most tasks (88% vs baseline 86%, headroom 80%).
- [docs/RESULTS.md](docs/RESULTS.md) — the live four-way SWE-bench Verified benchmark (Claude Code, `aws/claude-sonnet-5`): context-guru is the cheapest arm (−13.2% billed cost vs baseline) and solves the most tasks (88%); headroom −5.3%/80%; rtk (shell-level Bash-output hook) −9.0%/86% at $0 tool cost.

## License

Expand Down
112 changes: 112 additions & 0 deletions deploy/harbor/claude_code_rtk_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
"""Claude Code + rtk (Rust Token Killer) agent.

Identical to the stock ``claude-code`` agent, except it makes the ``rtk`` CLI
proxy active inside the task container so that **bash tool output is compressed
at the shell** before it ever enters the model context.

rtk is architecturally different from request-stream compaction proxies
(context-guru, headroom): it is a Claude Code ``PreToolUse`` hook that rewrites
Bash commands (``pytest ...`` -> ``rtk pytest ...``, ``cat f`` -> ``rtk read f``,
``git status`` -> ``rtk git status``) *inside the container*. It is not on the
network path and needs no proxy. Model routing is therefore whatever the harness
already points ``ANTHROPIC_BASE_URL`` at — for an apples-to-apples comparison the
harness routes it exactly like the ``off`` baseline, so the ONLY difference from
baseline is the in-container bash compression.

This subclass:
1. ``install()`` — after the stock Claude Code install, uploads a host-built
static-musl ``rtk`` binary to ``/usr/local/bin/rtk`` (path from
``RTK_BIN_HOST``, default ``/tmp/rtk-runs/rtk``) and also symlinks it into
``~/.local/bin`` so it is on the same PATH Claude Code launches with.
2. ``run()`` — installs the rtk ``PreToolUse`` hook into the exact
``CLAUDE_CONFIG_DIR`` Claude Code will read (``rtk init -g --auto-patch``,
which honors ``CLAUDE_CONFIG_DIR``), runs the stock agent, then dumps
``rtk gain --all --format json`` to ``/logs/agent/rtk-gain.json`` so the
harness can report rtk's own bash-output savings per trial.

End-to-end trajectory metrics (reward, steps, cache-read/write, billed cost,
cache-hit, wall) are produced by the unchanged parent class, so they are
measured identically to the other benchmark arms.
"""

import os
from typing import override

from harbor.agents.installed.claude_code import ClaudeCode
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext
from harbor.models.agent.name import AgentName
from harbor.models.trial.paths import EnvironmentPaths

# Host path to the static-musl rtk binary uploaded into each container.
RTK_BIN_HOST = os.environ.get("RTK_BIN_HOST", "/tmp/rtk-runs/rtk")


class ClaudeCodeRTK(ClaudeCode):
@staticmethod
@override
def name() -> str:
return AgentName.CLAUDE_CODE_RTK.value

@override
async def install(self, environment: BaseEnvironment) -> None:
# Stock Claude Code install first (curl/npm bootstrap of the CLI).
await super().install(environment)

# Deliver the rtk binary (static-musl, runs on any glibc/musl image).
await environment.upload_file(RTK_BIN_HOST, "/usr/local/bin/rtk")
await self.exec_as_root(
environment,
command="chmod 755 /usr/local/bin/rtk && /usr/local/bin/rtk --version",
)
# Belt-and-suspenders: Claude Code launches with PATH prepending
# ~/.local/bin, and both the PreToolUse hook (`rtk hook claude`) and the
# rewritten commands (`rtk pytest`) must resolve `rtk`. Symlink it there
# too so it is found regardless of whether /usr/local/bin is on PATH.
await self.exec_as_agent(
environment,
command='mkdir -p "$HOME/.local/bin" && ln -sf /usr/local/bin/rtk "$HOME/.local/bin/rtk"',
)

@override
async def run(
self, instruction: str, environment: BaseEnvironment, context: AgentContext
) -> None:
# Install the rtk PreToolUse hook into the SAME config dir Claude Code
# will read (claude_code.py sets CLAUDE_CONFIG_DIR to this path). rtk
# init honors CLAUDE_CONFIG_DIR; --auto-patch is REQUIRED for headless
# (piped stdin otherwise defaults the settings-patch to "N").
config_dir = (EnvironmentPaths.agent_dir / "sessions").as_posix()
await self.exec_as_agent(
environment,
command=(
'export PATH="/usr/local/bin:$HOME/.local/bin:$PATH"; '
"mkdir -p /logs/agent "
f'"{config_dir}"; '
f'CLAUDE_CONFIG_DIR="{config_dir}" '
"rtk init -g --auto-patch "
"> /logs/agent/rtk-init.log 2>&1 || true; "
# Snapshot the installed hook so we can confirm it fired.
f'cp "{config_dir}/settings.json" /logs/agent/rtk-settings.json '
"2>/dev/null || true"
),
env={"RTK_TELEMETRY_DISABLED": "1"},
)

# Run the stock Claude Code agent (its own setup_command only mkdir's
# under CLAUDE_CONFIG_DIR, so it does not clobber settings.json).
await super().run(instruction, environment, context)

# Dump rtk's own savings ledger for this trial. The rewritten bash-tool
# commands ran as the agent user and tracked to the default per-user DB
# (~/.local/share/rtk/history.db); read it back the same way.
await self.exec_as_agent(
environment,
command=(
'export PATH="/usr/local/bin:$HOME/.local/bin:$PATH"; '
"rtk gain --all --format json > /logs/agent/rtk-gain.json "
"2>/dev/null || true; "
"rtk gain --history > /logs/agent/rtk-history.txt 2>/dev/null || true"
),
env={"RTK_TELEMETRY_DISABLED": "1"},
)
42 changes: 42 additions & 0 deletions deploy/harbor/deep_analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@
"baseline": ("/tmp/cg-runs/final50/rows-off.json", None),
"context-guru": ("/tmp/cg-runs/final50-v6/rows-codesmart.json", "/tmp/cg-runs/final50-v6/summary.json"),
"headroom": ("/tmp/hd-runs/swe50/rows-hd-cache.json", None),
# rtk (Rust Token Killer): in-container bash-output compression via a Claude
# Code PreToolUse hook. A 4th arm; its rows carry an extra `rtk` sub-dict with
# rtk's own bash-output savings ledger. Optional — only included if present, so
# the published 3-way matched set is unchanged unless the rtk run is supplied.
"rtk": ("/tmp/rtk-runs/swe50/rows-rtk.json", "/tmp/rtk-runs/swe50/summary.json"),
}
# Drop arms whose rows file is missing so the script still runs pre-rtk-run.
SRC = {k: v for k, v in SRC.items() if Path(v[0]).exists()}


def billed(r):
Expand Down Expand Up @@ -108,6 +115,41 @@ def agg(c, key):
except Exception:
pass

# rtk tool metrics: $0 own-LLM cost + deterministic; its native metric is
# bash-OUTPUT bytes saved (bytes/4 estimate), a DIFFERENT denominator than the
# proxies' whole-request content%, so it is reported separately. Reversibility
# is a tee-file on failure (no expand/retrieve bounce), so bounces=0.
if "rtk" in configs:
rrows = json.load(open(SRC["rtk"][0]))
bycmd = {}
for r in rrows:
for c in ((r.get("rtk") or {}).get("by_command") or []):
name = c.get("command") or c.get("name") or "?"
key = " ".join(str(name).split()[:2]) # e.g. "rtk pytest"
e = bycmd.setdefault(key, {"count": 0, "saved": 0})
e["count"] += c.get("count") or 0
e["saved"] += c.get("saved") or c.get("saved_tokens") or 0
try:
rs = next(x for x in json.load(open(SRC["rtk"][1]))["configs"])
except Exception:
rs = {}
# Prefer the per-command aggregate built from the rtk `gain --history` "By
# Command" tables (the --format json output has no per-command array); fall
# back to whatever the rows carried.
try:
bc = json.load(open("/tmp/rtk-runs/rtk_by_command.json"))
bycmd = {k: {"count": v["count"], "saved": v["saved"]} for k, v in bc.items()}
except Exception:
pass
tool["rtk"] = dict(
llm_cost=0.0, added_ms=0.0, bounces=0,
content_pct=rs.get("rtk_bash_savings_pct"), # NB: bash-output denom, not whole-request
bash_tokens_before=rs.get("rtk_bash_tokens_before"),
bash_tokens_after=rs.get("rtk_bash_tokens_after"),
bash_tokens_saved=rs.get("rtk_bash_tokens_saved"),
commands=rs.get("rtk_commands"),
per_command=dict(sorted(bycmd.items(), key=lambda x: -x[1]["saved"])))

out = dict(matched_tasks=len(common), configs=configs, aggregate=aggregate,
tool=tool, per_task=per_task)
Path(f"{a.out}/deep_analysis.json").write_text(json.dumps(out, indent=1))
Expand Down
31 changes: 23 additions & 8 deletions deploy/harbor/deep_plots.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
import matplotlib.pyplot as plt

SURFACE = "#fcfcfb"; INK = "#0b0b0b"; INK2 = "#52514e"; GRID = "#e6e5e2"
# baseline grey, context-guru blue, headroom orange
COL = {"baseline": "#8a8985", "context-guru": "#2a78d6", "headroom": "#eb6834"}
# baseline grey, context-guru blue, headroom orange, rtk teal-green
COL = {"baseline": "#8a8985", "context-guru": "#2a78d6", "headroom": "#eb6834", "rtk": "#1baf7a"}
COMP = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300"]
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE,
Expand Down Expand Up @@ -58,7 +58,7 @@ def fig_headline(A, out):
ax[1][2].set_xticks(range(len(cfgs))); ax[1][2].set_xticklabels(cfgs, fontsize=9)
ax[1][2].set_title("Compaction latency added / request"); ax[1][2].set_ylabel("ms")
ax[1][2].grid(axis="x", visible=False); style(ax[1][2])
fig.suptitle("baseline vs context-guru vs headroom — SWE-bench Verified (matched %d tasks)" % A["matched_tasks"],
fig.suptitle("%s — SWE-bench Verified (matched %d tasks)" % (" vs ".join(cfgs), A["matched_tasks"]),
fontsize=13, fontweight="bold")
fig.tight_layout(); fig.savefig(out); plt.close(fig)

Expand Down Expand Up @@ -87,11 +87,14 @@ def fig_per_task_cost(A, out):
rows = sorted(A["per_task"], key=lambda r: r["baseline"]["cost"])
n = len(rows)
fig, ax = plt.subplots(figsize=(8, max(6, n * 0.22)))
lo = lambda r: min(r[c]["cost"] for c in A["configs"] if c != "baseline")
hi = lambda r: max(r[c]["cost"] for c in A["configs"] if c != "baseline")
for i, r in enumerate(rows):
ax.plot([r["headroom"]["cost"], r["context-guru"]["cost"]], [i, i], color=GRID, lw=1.5, zorder=1)
for c, mk in [("baseline", "|"), ("headroom", "o"), ("context-guru", "o")]:
ax.scatter([r[c]["cost"] for r in rows], range(n), s=30, color=COL[c], zorder=3, label=c,
marker=mk if c != "baseline" else "D")
ax.plot([lo(r), hi(r)], [i, i], color=GRID, lw=1.5, zorder=1)
markers = {"baseline": "D", "headroom": "o", "context-guru": "o", "rtk": "s"}
for c in A["configs"]:
ax.scatter([r[c]["cost"] for r in rows], range(n), s=26, color=COL[c], zorder=3, label=c,
marker=markers.get(c, "o"))
ax.set_yticks(range(n)); ax.set_yticklabels([r["task"] for r in rows], fontsize=6)
ax.set_xlabel("billed input cost per task ($)"); ax.set_title("Per-task cost")
ax.legend(loc="lower right", frameon=False, fontsize=8); ax.grid(axis="y", visible=False); style(ax)
Expand All @@ -118,7 +121,9 @@ def fig_components(A, out):
cg = A["tool"]["context-guru"]
uq = (cg.get("unique") or {}).get("components", {})
hr = A["tool"]["headroom"].get("per_strategy", {})
fig, axes = plt.subplots(1, 2, figsize=(11, 3.6))
rtk = (A["tool"].get("rtk") or {}).get("per_command", {})
npanel = 3 if rtk else 2
fig, axes = plt.subplots(1, npanel, figsize=(5.5 * npanel, 3.6))
# context-guru: cumulative vs unique
items = [(k, v.get("cum", 0), (uq.get(k, {}) or {}).get("uniq_saved", 0)) for k, v in cg["per_component"].items() if v.get("cum")]
# map dump categories to component names best-effort
Expand All @@ -138,6 +143,16 @@ def fig_components(A, out):
ax.set_yticks(range(len(hs))); ax.set_yticklabels([k for k, _ in hs], fontsize=8)
ax.set_xlabel("tokens removed"); ax.set_title("headroom — per compressor")
ax.grid(axis="y", visible=False); style(ax)
# rtk per-command (bash-output tokens saved)
if rtk:
rs = sorted([(k, v.get("saved", 0)) for k, v in rtk.items() if v.get("saved")], key=lambda x: x[1])
ax = axes[2]
ax.barh(range(len(rs)), [v for _, v in rs], color="#1baf7a", height=0.6, zorder=2)
for i, (k, v) in enumerate(rs):
ax.text(v, i, f" {int(v):,}", va="center", fontsize=8, color=INK)
ax.set_yticks(range(len(rs))); ax.set_yticklabels([k for k, _ in rs], fontsize=8)
ax.set_xlabel("bash-output tokens removed"); ax.set_title("rtk — per command")
ax.grid(axis="y", visible=False); style(ax)
fig.tight_layout(); fig.savefig(out); plt.close(fig)


Expand Down
Loading
Loading