SWE-bench gold-test runtime measurement + weight buckets (Plan B Task 5)#129
Merged
Conversation
…tives Derive the canonical gold-test command + directives for every deck instance straight from the installed swebench package, matching what swebench itself runs at eval time (swebench.harness.constants.MAP_REPO_VERSION_TO_SPECS + swebench.harness.test_spec.python.get_test_directives) rather than a driver-side per-repo heuristic. The upcoming measurement driver (Task 5a) reads test_cmd/test_directives straight off each deck instance. deckHash is unchanged (ff962cb83fe5c624): it hashes only instance_id+env_key, and neither changed. bake-list.json is untouched. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Adds deploy/knative/measure-swebench-runtimes.sh (Plan B Task 5a), a gated (MEASURE_LIVE=1) driver that measures each SWE-bench deck instance's gold-test wall-clock inside a live swebench sandbox pool pod and writes test_runtime_ms + weight_bucket (tercile: light/medium/heavy) back into experiments/swebench/deck.json. Per instance: git clone --local of the baked /repos/<repo>.git mirror + checkout base_commit (not a worktree -- worktree-from-baked-mirror fails as uid 65532), a per-instance venv --system-site-packages over the baked conda env, HOME=/workspace pip install -e with --no-build-isolation --no-cache-dir, then time <test_cmd> <directives...> under a 20-minute timeout with set -f + per-token single-quoting. Exit-code handling: at base_commit the FAIL_TO_PASS tests fail by definition, so a non-zero test exit is expected and the runtime is still recorded (status=ran). Only timeout (status=timeout, ms capped at the timeout) and setup failures (status=setupfail, ms=null, excluded from tercile) are special-cased. Terciles are computed deterministically (sort by ms ascending, split by rank into 3 as-equal-as- possible groups) in embedded python3 on the host, preserving the deck generator's JSON formatting. This is authoring-only: MEASURE_LIVE is unset by default (usage + exit 0), so this change does not run against a cluster and deck.json is unmodified. Validated via bash -n and shellcheck (clean); tercile/bucketing logic demonstrated offline on synthetic runtimes. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
git clone --local hardlinks objects, but the baked mirror at /repos (image overlayfs) and the scratch checkout under /workspace (EBS PVC) are different filesystems, so the clone fails with 'Invalid cross-device link' and every instance lands in setupfail on the live OCP gate. Switch to --no-hardlinks, which keeps the local-path fast-path but copies objects instead of hardlinking them, so it works across devices. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Two fixes to measure-swebench-runtimes.sh proven live on the OCP sandbox pool: 1. Splice test_cmd VERBATIM into the pod-side shell instead of tokenizing it on whitespace and re-quoting each token. Tokenizing broke inline env-var-assignment prefixes such as sympy's PYTHONWARNINGS='...' bin/test -C --verbose: quoting the assignment as argv[0] made the pod shell try to exec a literal command named 'PYTHONWARNINGS=...', exiting 127 in a few ms and masquerading as a completed run. test_cmd is now passed unmodified into a nested `bash -c 'set -f; <test_cmd> <directives...>'` invocation under timeout, so shell prefixes and flags are parsed correctly; each directive is still individually single-quoted via the existing qq() helper so '[param]'-style brackets are never glob-expanded. 2. Two-attempt editable install for the per-instance venv: first try the existing --no-build-isolation --no-cache-dir (fast, already proven for 13 repos), and on failure retry the same install WITH build isolation (still --no-cache-dir). This fixes repos whose baked env has a setuptools/setuptools_scm skew (pytest, pylint) without touching the repos that already worked. Only a failure of BOTH attempts is treated as a setup failure. Exit-code semantics are unchanged: timeout (124) records the cap and buckets heavy; any other completion records the measured wall-clock; only a setup step failing (clone/checkout/venv/both pip attempts) records test_runtime_ms = null. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Add build-essential, python3-dev, pkg-config, libfreetype6-dev, and libpng-dev to the base-tools apt-get install list so scikit-learn and matplotlib (and other C/Cython repos) can be editable-installed in the sandbox pool. Proven live on the OCP pod. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
All 24 deck instances measured live on the OCP swebench pool (gcc-enabled image); test_runtime_ms populated and light/medium/heavy terciles assigned (8/8/8). Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Plan B Task 5: measure each SWE-bench Verified deck instance's gold-test wall-clock in a live OCP sandbox pool and assign
light/medium/heavyweight buckets. This closes the spec §4 bucketing item and gives Plan C'sWORKLOAD=swebenchE6 driver real, populated buckets. Off-by-default and additive.What's in it (6 commits)
scripts/gen_swebench_deck.pynow emits per-instancetest_cmd+test_directivesfrom swebench's canonical API (MAP_REPO_VERSION_TO_SPECS[...]["test_cmd"]+get_test_directives);deck.jsonregenerated.deckHashunchanged (ff962cb83fe5c624— hashed over instance_id+env_key only);bake-list.jsonunchanged.deploy/knative/measure-swebench-runtimes.sh(gatedMEASURE_LIVE=1): per instance,git clone --no-hardlinksthe baked mirror + checkoutbase_commit→ per-instancevenv --system-site-packagesover the baked conda env → editable install → time the realtest_cmd+directives under a 20-min cap → median → terciles → writetest_runtime_ms/weight_bucketback intodeck.json.build-swebench-sandbox.shbase-tools:build-essential python3-dev pkg-config libfreetype6-dev libpng-dev) so C/Cython repos (scikit-learn, matplotlib) can be editable-installed. The…-15of15image was rebuilt via the emitter.Design notes (forced by live spikes)
git clone --no-hardlinks(notgit worktreeoff the baked mirror —/reposis read-only root-owned image layers; hardlink clone fails cross-device onto the EBS PVC)./testbedis intentionally absent), and its site-packages/HOMEare read-only to uid 65532 → per-instance venv + editable install (--no-build-isolation, falling back to build isolation for setuptools-skew repos).test_cmdis run verbatim (preserves inline env-prefixes, e.g. sympy'sPYTHONWARNINGS=... bin/test); directives are single-quoted underset -f.base_committheFAIL_TO_PASStests fail by design, so a non-zero test exit is expected and still records the wall-clock; only genuine setup failure → null; timeout → cap → heavy.Caveat
matplotlib__matplotlib-24177's recorded time (~2.3s) reflects import+collection: its pytest collection errors on a newer-setuptoolsDeprecationWarning-as-error atbase_commit(0 tests collected). Faithful to the gold command; flagged for awareness.Validation
24/24 measured live on OCP 4.20 (gcc-enabled pool image), 0 setupfail / 0 timeout; structural test
swebench-deck.test.tsgreen with populated buckets; deckHash intact.Assisted-By: Claude Code