th-4c3e2d: Smooth Operator engine-parity benchmark (engine × model axis)#246
Open
brentrager wants to merge 6 commits into
Open
th-4c3e2d: Smooth Operator engine-parity benchmark (engine × model axis)#246brentrager wants to merge 6 commits into
brentrager wants to merge 6 commits into
Conversation
Restore the deleted `smooth-bench` crate — the focused Aider-Polyglot slice only (single-task runner, curated sweep, WS chat driver, scoring, auto-approve) — against current main. Deps now resolve to the published `smooai-smooth-operator-core` engine plus in-tree cast/code/pearls; the slice compiled with only two mechanical fixups (`run_sweep` relaxed to `?Sized`, `CommandExt` import). The SWE-bench/replay/research/cleanup/ TUI-driver scorers were dropped. Add the engine-parity benchmark: `smooth-bench score` runs the curated aider-polyglot suite through each of the five smooth-operator LocalServer implementations (rust/go/ts/python/dotnet), scoring per engine and per model. - `--engine` (repeatable, default all) + `--model` (repeatable, default deepseek-v4-flash). - Each engine booted the way `scripts/operator-serve.sh` does (uniform env contract + per-engine bind var), sweep driven over the canonical WS protocol, torn down before the next cell. - Per-cell results carry engine+model and emit as JSON-lines + summary. - Matrix runner is parameterised on an `EngineBooter` trait; the engine×model aggregation and engine→boot-command mapping are unit- tested without a live LLM. A real scoring run needs SMOOAI_GATEWAY_KEY. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
🦋 Changeset detectedLatest commit: 5d052b9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
brentrager
enabled auto-merge (squash)
July 22, 2026 23:21
…in ensure_server
BigSmoothClient::ensure_server probed GET {url}/health and, on failure,
spawned 'th up' + waited 10s. The polyglot smooth-operator LocalServers
serve /ws but no /health, so the bench never reached the WS connect
(~65s = 5 retries). Short-circuit on a live TCP socket at the target
addr; the /health+autostart path still runs for th code when nothing
is listening. Layer 2 (canonical-protocol driver rewrite) tracked in
th-7232fb.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…ocol The bench's default drive path assumed the deleted microVM "create a pearl + dispatch a teammate" model, and the legacy direct path spoke the retired bespoke /ws handshake — so no engine ever completed a scored turn. Replace both with a new `canonical_driver` speaking the schema-driven protocol every engine now uses (create_conversation_session -> immediate_response .sessionId -> send_message -> drain to eventual_response), modeled on the daemon's OperatorTurnDriver::drive_once. It parses stream_chunk tool-result events into the BenchResult tool-calls and best-effort-scans the terminal event for cost (polyglot servers don't surface it -> $0, noted). Boot the engine PER TASK with its workspace pointed at the task's scratch work_dir: rust via SMOOTH_WORKSPACE; go/ts/python/dotnet (no workspace env) with cwd = work_dir. Go runs a prebuilt binary (go run can't launch from a foreign cwd); the rest use an absolute path / --project. Retire chat_driver + the smooth-code headless dependency. Verified live (deepseek-v4-flash / claude-haiku on the daemon): the go engine boots per task, the canonical turn runs, tests execute, and a real scored table is produced. The rust daemon engine additionally edits solution files in the task workspace (read_file/edit_file/bash tool results parsed) — a real 5/6 scored table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…indows CI windows-latest failed: `cannot find unix in os` + `no method named process_group`. The new-process-group call (so kill-on-drop reaps `go run`/`dotnet run` children) is unix-only. Gate the import and the call; the bench only ever RUNS on unix, but the crate must COMPILE on windows for CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
The compile fix landed (tests now RUN on windows), exposing the next failure: boot_command_mapping_matches_operator_serve asserts exact unix-style path strings, but repo.join(..) yields backslashes on windows. The bench only ever RUNS on unix (engines need unix toolchains); the crate just has to compile+test green on windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
No way to measure whether the 5 polyglot smooth-operator engines (rust/go/ts/python/dotnet) behave at parity. The mature
smooth-benchharness was deleted in the microVM/daemon rewrite.Solution
Resurrect the Aider-Polyglot slice of
smooth-benchonto current main (the rest — SWE-bench/replay/research/tmux/tui scorers — intentionally omitted; resurrect later if needed), and add an engine × model axis:crates/smooth-bench/src/engine.rs—Engineenum with a pureboot_command(repo, port)mirroringoperator-serve.sh's per-engine cases + env contract;ProcessBooterspawns each LocalServer, waits for the port, kills-on-drop.run_engine_matrix(parameterised on anEngineBootertrait) loops engine×model, points the sweep at each booted engine, tags eachScorewith engine+model, emits JSON-lines + summary.smooth-bench score --engine <...> (repeatable, default all) --model <id> (repeatable, default deepseek-v4-flash).The slice compiled clean on first build against current main (only fixups: a
?Sizedrelaxation + aCommandExtimport).smooth-operatornow resolves to publishedsmooai-smooth-operator-corev1.7.0.Verification
cargo test -p smooai-smooth-bench→ 110 passed. New tests: engine→boot-command mapping for all 5, distinct default ports, engine×model matrix aggregation via aFakeBooter+cannedTaskRunner(no live LLM).SMOOAI_GATEWAY_KEY+ the smooth-operator repo + per-engine toolchains. Harness builds + unit-tests green without it.Pearl th-4c3e2d (depends on th-3f46fd / PR #245).
🤖 Generated with Claude Code