feat(evals): unified evals lite profile, harness fixes, and luna scorecard#4723
Merged
Conversation
Install one PEP 508 spec per line so an unreleased Harbor build can override both harbor and harbor-langsmith at once. Newline is the only safe delimiter since a spec contains spaces (name @ git+url); specs are collected into an array and passed as separate args, preserving the existing env-passed, no-eval injection safety.
Add an `agent_impl` choice input (dcode|bare) to the unified workflow, threaded through unified_prep.py. It swaps the harness for the code categories (autonomous, context) only; conversation stays on tau3. Lets a run use the bare create_deep_agent graph, which avoids deepagents-code's reasoning_effort injection (that breaks function tools on models gated to the Responses API). Defaults to dcode; invalid values fail closed.
The harbor langgraph agent builds models with init_chat_model directly, bypassing the deep-agents OpenAI provider profile. OpenAI's model profile defaults reasoning_effort for gpt-5.x, and OpenAI gates reasoning_effort + function tools to /v1/responses, so every trial 400'd on /v1/chat/completions (all harnesses: dcode, bare, tau3). Set use_responses_api=True for openai: models in a shared _build_model helper; a caller-supplied model_kwargs wins.
harbor-index verifiers are OpenAI LLM judges but only received OPENAI_BASE_URL, so they exited without writing a reward file (RewardFileNotFoundError) on both docker and langsmith sandboxes. Forward OPENAI_API_KEY to the verifier env (mirroring the always-on base URL), and grant the key to the job for harbor-index runs so non-openai agent models can still be judged.
harbor-index verifiers run native_judge, which requires JUDGE_PROVIDER, JUDGE_MODELS, JUDGE_REPEATS, JUDGE_CONCURRENCY set 'by the job yaml' or it raises JudgeConfigurationError and writes no reward file. Wire an OpenAI judge into verifier_env_args (provider=openai, repeats/concurrency=1) with a configurable judge_models input (default gpt-5.6-luna). Prefer an independent grader, not the model under test, for real scorecards.
harbor-index tasks with large base images + capsule downloads (e.g. bix*) exceed their environment.build_timeout_sec on the first cold snapshot build and hit EnvironmentStartTimeoutError. Plumb --environment-build-timeout-multiplier as an input (default 1.0). Raising it for a one-time cold build lets it finish; the content-addressed (force_build=false) snapshot then persists so subsequent runs cache-hit at 1.0.
Heavy harbor-index images exhaust the LangSmith snapshot filesystem (harbor's 32 GiB floor) during the layer-copy build and fail with a masked 'Snapshot creation failed. Please retry.'. Plumb harbor's --override-storage-mb as an input (0 = default) so heavy tasks can build with e.g. 64 GiB. (Distinct from env_build_timeout_multiplier, which only extends harbor's client-side wait, not the server build.)
… sandbox The LangSmith sandbox + Client should authenticate with the standard LANGSMITH_API_KEY, not a separate LANGSMITH_SANDBOX_API_KEY (which routed to a different tenant). Remove it from deepagents_harbor.resolve_langsmith_api_key precedence and from the workflow env + credential check; LANGSMITH_API_KEY (always set) covers both. Update tests accordingly.
unified_prep minted a per-model LangSmith dataset (harbor-index__<model>-<hash>) per run, duplicating examples and preventing cross-model comparison — and the per-model override name broke the context (local-dataset) experiment attach (dataset synced, 0 sessions). Pass empty langsmith_dataset so the leaf derives the canonical shared dataset per category (harbor-index/harbor-index-1.0, tau3-subset, local/datasets-context-retrieval-evals); each model is then an experiment on the one dataset. Removes now-dead ls_dataset/slugify/_short_hash + updates tests.
profile=lite runs a frozen high-signal task subset per category (lite_tasks.py: autonomous 15, conversation 11, context 8) at FULL rollouts — biased to the difficulty frontier (partial-pass / hard-but-solvable on a weaker model), excluding saturated + verifier-unstable tasks. unified_prep sets per-category include_tasks + right-sizes n_shards; unified_evals threads include_tasks to each eval job; the tau3-subset step now respects a caller include_tasks (lite wins).
The two bix* tasks pull the ~6 GB chenzizhao/bixbench image, which exhausts the Docker-sandbox runner disk when co-scheduled in one shard (and fails the LangSmith builder). Replace with light python-slim tasks (build-word2vec-pipeline, arcagi2-grid-transform-8b7b) to keep the frontier size at 15 without the disk blowout.
The lite profile was right-sizing shards to ~5 tasks each (min(default, ceil(n/5))), so a 15-task autonomous slice ran only 3 shards => 12 concurrent trials of a 40/model budget (~30% utilization). Fan out to ~2 tasks/shard capped at shard_parallel so every shard runs at once: autonomous 3->8 shards (32 concurrent), conversation 3->6, context 2->4. Also scatters heavy task images one-per-runner instead of concentrating them in a fat shard.
Cap-at-shard_parallel meant n_shards never exceeded max-parallel, so there was no queue: all shards started at once with a static task partition, and a runner that drew fast tasks went idle while the long-pole shard ran. Set n_shards = n_tasks so shard_parallel is purely the concurrent-slot count and GitHub's matrix scheduler pulls the next single-task shard the instant one finishes. The long pole becomes the slowest single task, and each runner builds exactly one task image (no heavy-image concentration).
One-task-per-shard left --concurrency idle: a shard ran only its `rollouts` trials (3 of 4). Size each shard to ceil(concurrency/rollouts) tasks so its tasks*rollouts trial count fills concurrency and every active runner runs a full 4-wide. Rollout-aware: rollouts=1 -> 4 tasks/shard instead of wasting slots. Threads concurrency+rollouts through prep (new UNIFIED_ROLLOUTS env). Full-profile sharding is unchanged.
…t-aware)" This reverts commit 7f6abae.
Nick Hollon (nick-hollon-lc)
requested a review
from Mason Daugherty (mdrxy)
as a code owner
July 14, 2026 13:25
github-actions
Bot
added
evals
ruff-format two evals files (hand-wrapped lines the repo's ruff collapses), and mirror the multi-spec harbor_package_override description into harbor.yml so the both-surfaces consistency test passes.
_build_model had no return annotation, so ty inferred object and rejected it at the create_cli_agent / create_deep_agent call sites (which want str | BaseChatModel[ | None]). Annotate the return as BaseChatModel (what init_chat_model returns); TYPE_CHECKING-only import, no runtime change.
![open-swe[bot]](https://avatars.githubusercontent.com/in/1395372?s=60&v=4){kind=small}
Released harbor-langsmith==0.1.1 suffixes every shard's LangSmith experiment
with -{job.id[:8]}, so a model fragments into one experiment per shard on the
shared dataset. harbor main honors the explicit experiment_name the leaf sets
and reuses the session across shards; document that the harbor_package_override
(commit af2e862) is required until that ships in a release.
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 14, 2026 14:11 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 14, 2026 14:11 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
had a problem deploying
to
evals
July 14, 2026 14:11 — with
GitHub Actions
Error
GitHub Actions
Error
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:39 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
had a problem deploying
to
evals
July 14, 2026 16:40 — with
GitHub Actions
Failure
GitHub Actions
Failure
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 21:37 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 21:37 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 21:37 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 21:37 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 22:20 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 22:20 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 22:20 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Shrikar Seshadri (davibinboi)
temporarily deployed
to
evals
July 14, 2026 22:20 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…evals # Conflicts: # .github/scripts/test_unified_prep.py # .github/scripts/unified_prep.py # .github/workflows/unified_evals.yml
Member
|
I don’t think we should remove The underlying bug appears to be that
If the current sandbox integration cannot accept a dedicated credential, we should document that limitation explicitly. Otherwise, using one key for both purposes removes the isolation the two-secret setup was designed to provide. |
…nt-kwarg
Pass --environment-kwarg langsmith_api_key='${LANGSMITH_SANDBOX_API_KEY}' for the langsmith sandbox and expose LANGSMITH_SANDBOX_API_KEY to the run env, so the sandbox authenticates with a dedicated, independently-revocable key while tracing/datasets/results stay on LANGSMITH_API_KEY (resolve_langsmith_api_key still prefers LANGSMITH_API_KEY).
Requires a harbor build with --environment-kwarg ${VAR} support (harbor#2344); langsmith-sandbox runs must therefore set harbor_package_override until it releases. Noted on the harbor_package_override input description. Validated end-to-end (1 trial, 0 exceptions) on nh/smoke-env-kwarg-test.
…e descriptions identical The override requirement for langsmith-sandbox runs is already documented in the _harbor_run.yml env_kwarg comment; the input-description note diverged from harbor.yml and broke test_override_inputs_warn_against_mutable_or_credentialed_sources.
The credential-verify gate still read 'Uses LANGSMITH_API_KEY' and didn't check the sandbox key, even though the sandbox now authenticates with LANGSMITH_SANDBOX_API_KEY via --environment-kwarg. Restore the gate so a langsmith run fails fast when the dedicated key is missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the unified multi-model Harbor evals workflow (
unified_evals.yml) and adds a model scorecard. Validated end-to-end onopenai:gpt-5.6-luna(docker sandbox, bare harness).Workflow / harness
profile=lite: a frozen high-signal task subset (lite_tasks.py, difficulty-frontier tasks; full rollouts, fewer tasks) selectable via theprofileinput. Sharded one task per shard so GitHub's matrix drains a dynamic queue (long pole is the slowest single task, and heavy task images never concentrate on one runner). Roughly 8x faster and 6x cheaper than the full run.agent_implinput (dcode|bare) to pick the harness for the code categories; conversation stays on tau3.harbor_package_override(one pip spec per line) soharborandharbor-langsmithcan be pinned to the same commit.LANGSMITH_SANDBOX_API_KEY, passed via harbor's--environment-kwarg(resolved from the host env at construction, so it never lands on argv or in the serialized job/trial config), while tracing/datasets/results stay onLANGSMITH_API_KEY. Requires a harbor build with--environment-kwarg ${VAR}support (feat(cli): resolve ${VAR} templates in --environment-kwarg / --plugin-kwarg harbor-framework/harbor#2344); langsmith-sandbox runs must therefore setharbor_package_overrideuntil that releases.Fixes
reasoning_effort+ tools 400 on Responses-only models.JUDGE_*config +OPENAI_API_KEYto the harbor-index verifier env for LLM-judge tasks, fixingRewardFileNotFoundError.resolve_langsmith_api_keypreferringLANGSMITH_SANDBOX_API_KEYfor tracing/dataset/results calls (which routed them to the wrong tenant); those now useLANGSMITH_API_KEY.Docs
libs/evals/UNIFIED_SCORECARD.md: GH aggregate pass@k / avg@k for luna, with Full (default) and Lite tables plus per-run conditions.