Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
version: 2

updates:
# Python runtime dependencies (requirements.txt).
- package-ecosystem: pip
directory: "/"
schedule:
interval: weekly
open-pull-requests-limit: 5
commit-message:
prefix: "[bench] deps:"

# GitHub Actions used by .github/workflows. Keeps the CI supply chain from
# silently aging into unmaintained action versions.
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: weekly
open-pull-requests-limit: 5
commit-message:
prefix: "[bench] ci:"
9 changes: 7 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,20 @@ on:
pull_request:
branches: [main]

# Least privilege: this workflow only reads the repo. Without this block the
# job would inherit the repository default token scope.
permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ venv/
*.egg-info/
.coverage
htmlcov/
.pytest_cache/
.ruff_cache/
.mypy_cache/

# Environment
.env
Expand Down
6 changes: 6 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ PreToolUse Hook -> Challenger (Sonnet) -> Defender (Sonnet) -> Oracle (Opus) ->
- Constitution snapshot loaded once per pipeline run (frozen within, hot-reload between)
- Oracle verdict is PASS or VETO. VETO is binding.
- Every verdict hashed and chained into bench-ledger.json
- Ledger destination is project-scoped via `ledger.chain.resolve_ledger_path()`:
Bench's own repo uses `ledger/bench-ledger.json`, any other governed project
uses `<project>/.bench/bench-ledger.json`, and `BENCH_LEDGER_PATH` overrides
both. Readers (`load_ledger`, `verify_chain`, the viewer) resolve through the
same function, so the auditor never inspects a different file than the writer
appends to.
- On VETO: JSON permissionDecision "deny" with remediation feedback
- On PASS: JSON permissionDecision "allow"
- Exit code is ALWAYS 0. Flow control is via JSON, not exit codes.
Expand Down
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,20 @@ Not all tool inputs are simple text edits. Bench handles three edge cases:
- **Large diffs** exceeding 300 lines are truncated while preserving governance-critical lines: imports, function/class signatures, and exception handlers.
- **New file creation** is typed as `change_type: "create"` so the pipeline knows it is reviewing a creation, not a modification.

### Project-Scoped Ledger

Bench's hook can be registered globally in `~/.claude/settings.json`, which governs every project on the machine. Each project's verdicts land in that project's own ledger, not in Bench's:

| Working directory | Ledger |
|---|---|
| Inside the Bench repo | `ledger/bench-ledger.json` (Bench governing itself) |
| Any other project | `<project>/.bench/bench-ledger.json` |
| `BENCH_LEDGER_PATH` set | That path, overriding both |

`ledger-meta.json` is written alongside whichever ledger is selected, so every chain carries its own anchor and verifies independently. `python -m cli verify` prints which ledger it read, and validates that chain only: per-project chains under `.bench/` are verified by running the command from that project.

This matters because a ledger records the full diff of every change it governs. Routing all projects into one chain mixes unrelated codebases together, and if that chain is committed to a public repository, it publishes them. Set `BENCH_LEDGER_PATH` if you deliberately want one central ledger across projects.

## Models

| Role | Constant (in `utils/api.py`) | Current model | Purpose |
Expand Down
126 changes: 126 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Security Policy

Bench is a constitutional governance layer for Claude Code. It sits between a
model's proposed file change and your filesystem, and it keeps a hash-chained
record of every verdict. A vulnerability in Bench is therefore usually one of
two things: a way to get a change past governance without a verdict, or a way
to make the ledger lie about what happened.

## Supported Versions

| Version | Supported |
|---------|--------------------------------------------|
| 1.1.x | Yes. Security fixes land here. |
| 1.0.x | No. Upgrade to 1.1.x. |
| `main` | Yes, best effort. Fixes land here first. |

## Reporting a Vulnerability

Do not open a public issue for a suspected vulnerability.

1. **Preferred:** open a private report at
https://github.com/Nuralyn/Bench/security/advisories/new
2. **Alternative:** email dburks@nuralyn.com with `[bench-security]` in the
subject line.

Please include:

- The version or commit SHA you tested.
- Which provider was configured (`BENCH_PROVIDER`: `anthropic`, `openrouter`,
or `claude_code`).
- Reproduction steps, ideally a minimal diff or tool input that triggers the
behavior.
- What you believe the impact is (governance bypass, ledger forgery, secret
exposure, and so on).
- A redacted ledger entry or stderr excerpt if one is relevant. Scrub API keys
before sending.

What to expect:

- Acknowledgement within 3 business days.
- An initial assessment (accepted, needs more information, or out of scope)
within 10 business days.
- A fix and a public advisory for accepted reports, with credit unless you ask
otherwise.

Please allow 90 days before public disclosure, or less by mutual agreement if a
fix ships sooner. Bench is maintained by a single author, so timelines are best
effort rather than contractual.

## In Scope

These are the classes that matter most for this project:

- **Governance bypass.** Any tool input, path, or encoding that causes a
Write/Edit/MultiEdit to reach disk without an adjudicated verdict, including
abuse of the `BENCH_SUBPROCESS` reentrancy guard to mark arbitrary edits as
pipeline-internal.
- **Fail-open regressions.** Any path where an API timeout, malformed judge
response, unimportable pipeline module, or unreadable constitution yields
`permissionDecision: "allow"` instead of `"deny"`. Bench is fail-closed by
design; a fail-open path is a bug, not a convenience.
- **Ledger integrity.** Forging, reordering, rewriting, or deleting entries in
a way that still passes `python -m cli verify`. Also: entries that record a
constitution hash they were not actually judged under.
- **Prompt injection through governed content.** Text inside a diff or a file
under review that steers the Challenger, Defender, or Oracle into a PASS, that
suppresses findings, or that exfiltrates the constitution or surrounding
context into a model response or ledger entry.
- **Constitution tampering.** Anything that lets the three stages of one
pipeline run see different constitution versions, breaking the per-run
snapshot guarantee.
- **Secret exposure.** `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, or session
credentials leaking into ledger entries, stderr output, subprocess argument
vectors, or the generated HTML viewer.
- **Local code execution** triggered by a crafted `bench.json`, a crafted
ledger file, or a crafted hook payload.

## Out of Scope

- **Model judgment quality.** An Oracle that passes a change you consider bad,
or vetoes one you consider fine, is not a vulnerability. Bench is adversarial
probabilistic review with a binding verdict, not a proof system. Report it as
a normal issue so the constitution or prompts can improve.
- **Attacks that require existing write access to your machine.** Someone who
can already run a shell in your working tree can edit files outside the
governed tool path. That is documented and intentional: it is the recovery
route when the pipeline itself is broken.
- **Upstream vulnerabilities** in the Anthropic API, OpenRouter, the Claude Code
CLI, or Python itself. Report those to their maintainers. Do report cases
where Bench's specific use of them creates exposure that would not otherwise
exist.
- **Dependency CVEs with no reachable call path** in Bench, and missing
hardening with no demonstrated exploit. Both are welcome as normal issues.
- **Resource exhaustion against your own pipeline** by feeding it very large
diffs. Bench truncates at 300 lines and is a local single-user tool.

## Security Model and Assumptions

Understanding these will save you time deciding whether a finding is in scope.

- **Local tool, local trust boundary.** Bench runs on a developer machine as a
Claude Code hook. There is no server, no multi-tenant surface, and no network
listener. The trust boundary is your machine and your API credentials.
- **Fail-closed, always exit 0.** The hook always exits with code 0 and controls
flow through the JSON `permissionDecision` field. When the pipeline cannot
render a verdict, the change is denied and a `pipeline_error` VETO is written
to the ledger. The sole exception is the reentrancy guard for a
Bench-spawned governance subprocess.
- **The ledger is tamper-evident, not tamper-proof.** Entries are not signed.
Anyone with write access to `ledger/bench-ledger.json` can rewrite it; the
SHA-256 chain makes that detectable through `python -m cli verify`, and
committing the ledger to git gives you a second copy to compare against.
There is no cryptographic proof of authorship today.
- **Governed content is sent to a model provider.** Every diff Bench adjudicates
is transmitted to whichever backend `BENCH_PROVIDER` selects. Treat the
contents of governed files accordingly.
- **Only the PreToolUse path is governed.** Direct shell commands, editors, and
anything that does not route through Claude Code's Write/Edit/MultiEdit tools
is outside Bench's reach by design.

## Credentials

Bench reads API keys from the environment only. No key is ever committed to this
repository, and constitutional constraint C-006 vetoes hardcoded secrets in
governed changes. If you find a credential in this repository or its history,
report it through the private channels above rather than opening an issue.
6 changes: 3 additions & 3 deletions bench.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"constitution": "bench-v1",
"version": 1,
"version": 2,
"author": "Nuralyn LLC",
"created": "2026-04-21T00:00:00Z",
"description": "Constitutional constraints for the Bench governance pipeline",
Expand Down Expand Up @@ -65,9 +65,9 @@
"id": "C-008",
"name": "Ledger Immutability",
"scope": "evidence",
"rule": "No code may modify, delete, or overwrite existing ledger entries. New entries may only be appended. The hash chain must remain intact.",
"rule": "No code may modify, delete, or overwrite existing ledger entries. New entries may only be appended. The hash chain must remain intact. Editing, reordering, or removing an individual entry is a violation under all circumstances, with no exception. A whole chain may be retired, and retirement is not deletion, but the sole permitted trigger is that the chain contains content which must not be published. Retirement requires ALL of the following: (a) an explicit human decision, never an automated or agent-initiated one, with the decision recorded in the anchor entry; (b) an archived copy that passes ledger.verify.verify_chain with valid=true BEFORE the original is moved; (c) an anchor entry opening the successor chain that records the retired chain's tip hash, genesis hash, entry count, first and last timestamps, the verbatim archive path, and the reason for retirement; (d) indefinite retention of the archive at that recorded path. An auditor confirms any retirement by running verify_chain against the archive and checking that its tip hash and entry count match the values recorded in the anchor entry. A retirement missing any of these elements is a violation of this constraint.",
"severity": "veto",
"rationale": "The ledger is the evidence chain. Tampering with evidence is the highest governance violation."
"rationale": "The ledger is the evidence chain. Tampering with evidence is the highest governance violation. Chain retirement is bounded rather than forbidden because the alternative is worse: a chain that has accumulated unpublishable third-party content would otherwise force entry-level edits, which are exactly what this constraint exists to prevent. Retirement preserves every entry, is triggered only by a human, and leaves a hash reference an auditor can independently check, so custody is documented rather than broken. The enumerated requirements make the exception verifiable instead of discretionary."
}
]
}
45 changes: 42 additions & 3 deletions cli/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,37 @@
_RULE_PREVIEW_LEN: int = 100


def _display_path(raw: str) -> str:
"""Render a ledger path for terminal output, preferring a relative form.

Now that the ledger is project-scoped, operators need to see which chain
a command actually read. Printing the absolute path would put the user's
home directory (and username) into any output they paste into a bug
report, so collapse it to a CWD-relative path when the ledger sits under
the working directory, and fall back to the absolute path otherwise.

Mirrors the normalization idiom in ``hooks/pre-tool-use.py``.
"""
if not raw or raw == "-":
return "-"
try:
rel: str = os.path.relpath(os.path.realpath(raw), os.getcwd())
except ValueError:
# Windows: path on a different drive from CWD, so no relative form
# exists. Expected control flow, not a fault.
return raw
except OSError as exc:
# An unresolvable path is a genuine fault; do not hide it (C-001).
print(
f"[bench cli] cannot resolve ledger path {raw!r}: {exc}",
file=sys.stderr,
)
return raw
if rel == os.pardir or rel.startswith(os.pardir + os.sep):
return raw
return rel


def cmd_verify() -> int:
"""Validate the ledger hash chain and print a pass/fail summary."""
result: dict[str, Any] = verify_chain()
Expand All @@ -46,6 +77,7 @@ def cmd_verify() -> int:
print("Ledger: EMPTY (nothing to verify)")
return 0
print("Ledger: VALID")
print(f" ledger : {_display_path(result.get('ledger_path', '-'))}")
print(f" entries : {entries}")
print(f" first entry : {result.get('first_entry', '-')}")
print(f" last entry : {result.get('last_entry', '-')}")
Expand Down Expand Up @@ -131,12 +163,19 @@ def cmd_stats() -> int:
else:
integrity = f"INVALID ({verify.get('failure_type', 'unknown')})"

anchors: int = stats["anchors"]
adjudicated: int = stats["adjudicated"]

print("Bench Governance Statistics")
print("=" * 40)
print(f"Total governed changes : {total}")
print(f"Passed : {passed} ({pct(passed, total)})")
print(f"Vetoed : {vetoed} ({pct(vetoed, total)})")
print(f"Total governed changes : {adjudicated}")
print(f"Passed : {passed} ({pct(passed, adjudicated)})")
print(f"Vetoed : {vetoed} ({pct(vetoed, adjudicated)})")
print(f"Pipeline errors : {pipeline_errors}")
if anchors:
# Chain-retirement markers (C-008). Not adjudicated changes, so they
# sit outside the pass/veto rates above.
print(f"Chain anchors : {anchors}")
if most_cited is not None:
print(
f"Most cited constraint : {most_cited[0]} "
Expand Down
Loading