[STEPPING] grade against the live anchor, retire the baseline, and make a zero accuracy explain itself

[ZhengGong-amd]

Branch: fix/zgong/explore-opt-12, 14 commits on top of f52bd2e
(refactor(state): resolve the grading anchor in one place).

That base commit added resolve_grading_anchor_tput() and routed the dispatch
sites through it. Working from there through two real sessions
(MiniMax-M3-MXFP8, and a DSV4 run whose PRELUDE lasted six hours) surfaced a
connected set of defects in what the loop measures against, what it bothers
re-measuring, and what a measurement is allowed to mean when it comes back
wrong. This branch closes all three, plus the diagnostics needed to see it
happen next time.

---

1. The grading anchor

f52bd2e put every dispatch site on one anchor resolver. Three defects survived
it, all of the same shape: a number captured at one moment being used to judge a
measurement taken at another.

A winner could lower current_best — b8e6308b5. _lift_to_current_best
overwrote unconditionally, so a variant its own executor called a KEEP (against a
stale task-level base_tput) could regress the recipe. In MiniMax-M3-MXFP8 a
warm replay had reached 2358.8 and an explore variant measuring 2355.5 was still
lifted — the run ended up reporting less gain than it had already achieved. The
lift is now refused when the winner does not beat the anchor it was composed on,
and it returns whether it landed so callers stop stamping a promotion, advancing
cumulative_gain_validated, or firing a watermark roofline for work that was
refused. The audit row records no_promote rather than discarded, matching
_promote_baseline, so orchestration reads it as measured-but-flat instead of
retrying it as a failure. No allow_regression escape: all four call sites were
audited and none needs one.

A queued task graded against a stale snapshot — 959c4200e. base_tput is
written into task params at queue time and the backstop only consulted live state
when it was missing. A task that waits behind others — PRELUDE ran six hours in
the session that surfaced this — grades against an anchor current_best has
since moved past, and the regression reads as a win. Explore and integrate_patch
now take the live anchor whenever it exceeds the snapshot and warn on the drift,
so the next occurrence is one grep away. This also feeds the stack rebench a
correct stability floor, which on its own would have rejected the variant that
started this.

...except when revalidating the whole stack — d13d51df7. The base commit
had carved out resume/geak revalidation on purpose: those tasks reproduce the
entire stack, so their gain is cumulative rather than a delta over
current_best. The live-anchor rule above would have broken that. Explore now
keeps its queued base_tput when params.source == "resume_stack_revalidate".
The same commit extends the live-anchor rule to framework_agent, which had
only been consulting live state when the snapshot was absent.

2. Baseline lifecycle

baseline is LLM-proposable, PRELUDE admits it so the run can reach
baseline_tput > 0, and nothing retired it afterwards.
_sequence_denial_for_action exempts baseline by design — it exists to hold
everything else back until the anchor lands — and PolicyGate carried a
singleton rule only for sweep. A repeat baseline was therefore denied by
nothing and reviewed against an empty evidence checklist. MiniMax-M3-MXFP8
executed ten of them, eight after the anchor already existed, at roughly nineteen
GPU-minutes each.

• 254fd8bf8 adds baseline_phase_singleton to PolicyGate on the same shape
  as sweep_phase_singleton: denied on both the propose_action and delegate
  channels once the anchor is positive.
• 05dd00dc1 cancels the queued backlog when baseline_tput turns positive
  and reports it as a baseline_drain observation. cancel_family grows a
  reason and an exclusion set — it previously hardcoded prune_branch into
  every cancellation's history evidence, which would have misattributed these.
  The revalidation params reason becomes a shared constant instead of a literal
  repeated across four call sites.
• a0d708e15 stops the new singleton from catching the Coordinator's own
  work. The dispatched-task re-validation path had no task id, so the tracked
  enablement revalidation baseline — which re-anchors a stack the specialist
  changed, rather than re-measuring one the session already has — was denied on
  its way to the executor. validate_dispatched_task now takes task_id and
  skips the singleton for the tracked id.
• d13d51df7 removes the params.bypass_baseline_singleton escape hatch
  entirely. It was introduced as an operator override but sits on a channel the
  agent controls, so it was an agent-controlled baseline bypass in practice. The
  test flipped from "bypass works" to "bypass is rejected".

08e813670 is the related capability gap: orchestration could already emit
prune_branch, but the handler always added the family to the persistent pruned
set. That is right for an action that has to stop and wrong for a backlog that
merely outlived its purpose — draining stale baselines should not cost the run
the ability to re-baseline later. prune_branch gains a scope field following
the convention kill_task established; the default family keeps today's
behaviour, and queued cancels the queued tasks and leaves the pruned set alone,
routing through the same helper the Coordinator uses so the enablement
revalidation is spared there too. Documented for the model in
prompts/orchestration.md alongside kill_task / send_message /
extend_lease.

3. What a zero accuracy means

Three separate ways a real accuracy signal was being lost or misread.

A measured zero was reported as "no measurement" — 7dae704f0. The
sibling-accuracy salvage filtered on accuracy > 0.0, so a baseline that
genuinely scored gsm8k=0.0 was indistinguishable from one whose eval never ran.
The cold-start guard evaluates only in warmup_round, so the deciding
measure_round always salvages: a real zero was dropped, acc stayed None, and
classify_accuracy_failure stamped accuracy_unavailable with evidence
accuracy=None ... source=None. The enablement specialist was handed "the eval
produced no number" while a full GSM8K run of degenerate output sat unreferenced,
and the "salvaged but below the floor" branch was unreachable for exactly zero.
Salvage now returns any finite score and the callers decide usability through the
existing accuracy_meets_floor. _finite_score replaces a hand-rolled float()
guard that would have admitted NaN/inf, and the write-back block duplicated
across both call sites moves into _apply_salvaged_accuracy.

A non-terminating model could burn the whole session — 336803652,
f24811b59, 2a354dca7. InferenceX runs lm-eval with
--gen_kwargs max_tokens=min(16384, EVAL_MAX_MODEL_LEN - 4096), and Hyperloom
does not set MAX_MODEL_LEN, so for any model with a 32k+ window that resolves to
16,384 tokens per request — overriding lm-eval's own max_gen_toks: 256
default, which is why reading the harness alone gives the wrong answer by 64x.
A model that never emits EOS then generates 1319 x 16,384 ≈ 21.6M tokens
against ~0.26M for a healthy one, roughly 100x.

Nothing bounded it. The soft deadline is retired at eval start by design (it is
anchored on throughput), the detokenizer-stall watchdog needs 1800s of total
log silence and a runaway decode keeps the log busy, and --max-hours is only
checked between coordinator ticks. The 7800s/9000s baseline subprocess timeout
was the only backstop — and blowing it also discards the throughput benchmark
that had already finished, skips the RUN_EVAL=false salvage retry (explicitly
disabled for genuine baselines), and ends the run on baseline_accuracy_failed.

The fix is a short-circuit, not a new failure mode: such a model was always
going to score ~0, so the probe reaches that verdict in minutes instead of hours
and lets lm-eval finish normally with a real results*.json. Every downstream
consumer is untouched — baseline stops on the existing baseline_accuracy_failed
(no new stop_reason, STOP_REASON_VOCAB unmodified), a variant REVERTs through
the existing accuracy_keep_block.

ensure_benchmark_lib_eval_probe_patched() appends a Python body to the
sitecustomize.py InferenceX already writes in _patch_lm_eval, anchored on the
unique export PYTHONPATH line that closes it — appending to the same file is
what guarantees the probe installs after, and on top of, InferenceX's own
monkeypatches. It watches finish_reason, and once observed >= 16 with >= 75%
at the max_tokens cap it writes $RESULT_DIR/hyperloom_eval_probe.json and
answers every remaining generate request with an empty string. Loglikelihood
requests are never short-circuited — they emit no tokens, so the pathology cannot
apply.

The short-circuit hooks amodel_call, not _create_payload:
get_batched_requests creates one asyncio task per request up front and
amodel_call builds its payload before awaiting the inner semaphore, so all
1319 payloads already carry the large max_tokens within milliseconds — lowering
it after the fact is a no-op. An equally sized outer gate parks the tasks
instead, loop-keyed because asyncio.run() builds a fresh loop per batch.
Tunable via HYPERLOOM_EVAL_PROBE{,_MIN_SAMPLES,_LENGTH_RATIO}; the three knobs
join _EVAL_CONTRACT_ENV_KEYS so runs compared under different settings no
longer fingerprint as comparable.

The reason is traceable end to end. Two paths, because the two action kinds
record differently — integrate_patch is not in _AUDIT_ACTIONS, so the journal
is its equivalent of the extras channel:

	baseline	integrate_patch
carrier	result["eval_probe"]	result["reason"]
channel	audit_extras → state.baseline_attempts[].extras	JournalEntry.reason → optimization_journal.json
in session_breakdown.json	baseline.attempts_history[].extras.eval_probe, action_timeline[].extras	action_timeline[].reason

f24811b59 is what makes the left column work: record_action_attempt already
persisted an arbitrary extras dict per attempt, but collect_baseline dropped
it, so anything the writeback audit recorded was invisible in the breakdown.
Additive only — schema.py states schema_version bumps on breaking changes, so
SCHEMA_VERSION is unchanged.

4. Orchestration diagnostics

d8b1b208c + 23cffe6b4. When a control-plane call failed there was nothing on
disk to diagnose it after the session ended. A new
orchestrator/trace/orchestration_trace.py persists a redacted row per
orchestration turn — SDK/CLI version, gateway endpoint, request id, resume and
session-id hashes, prompt hashes and lengths, allowed tools, MCP servers,
messages, result, parse errors, usage — plus the MCP setup, through two new
session_paths entries (orchestration_turns_path, agent_mcp_setup_path).
Secrets are scrubbed with the existing redact_secrets.

d13d51df7 then gates it behind an opt-in capture_turn_diagnostics flag that
only the orchestration backend sets in cli/backends.py, so the kernel / critic /
robustness backends do not pay for capture they never read. The latency_ms
argument drops out of the coordinator's recorder along with it.

5. Reference-script env defaults

b8bcfabbf. export FOO=${FOO:-1} is the idiomatic overridable default in every
InferenceX recipe, but _extract_envs skipped any value containing $, so those
settings never reached the lifted recipe even though the script applies them
whenever the caller does not override. The self-referential default is now
resolved to its literal, and quote stripping runs before the check so
export FOO="${FOO-1}" is covered too. Only the self-referential form counts:
export FOO=${BAR:-1} depends on an unrelated variable, so its default is not
FOO's effective value here and that line is still skipped, as is any default that
itself contains a $.

---

Commits

sha	subject
b8e6308b5	fix(writeback): never lower current_best when lifting a winner
959c4200e	fix(explore,integrate): re-resolve the grading anchor at execution time
05dd00dc1	fix(baseline): drain the queued backlog once the anchor is established
254fd8bf8	feat(policy): retire baseline once the session has an anchor
08e813670	feat(prune): let orchestration drain a queue without retiring the family
7dae704f0	fix(baseline): report a measured zero accuracy as below-floor, not unavailable
b8bcfabbf	feat(reference-script): lift the literal default out of ${FOO:-value} exports
a0d708e15	fix(policy): admit tracked enablement revalidation baselines
d8b1b208c	feat(trace): preserve orchestration turn diagnostics
23cffe6b4	test(trace): cover orchestration diagnostics
d13d51df7	fix(orchestrator): preserve valid revalidation anchors
336803652	feat(eval): add generation-pathology kind and probe sidecar readers
f24811b59	feat(breakdown): carry writeback audit extras into baseline attempt history
2a354dca7	feat(eval): cut short an accuracy eval whose model never emits EOS

d13d51df7 deliberately spans sections 1, 2 and 4: it is the follow-up pass that
reconciles the three earlier changes with each other.

Testing

New coverage lands with the commit that needs it:

• anchor rules — test_promote_shared_state_lock.py (lift refusal, drain),
  test_explore_executor.py (stale-snapshot supersede, parametrised over the
  resume_stack_revalidate carve-out), test_framework_agent_executor.py
  (reverts when the live anchor exceeds the queued baseline);
• baseline lifecycle — test_sweep_phase_auto.py (singleton on both channels,
  inert before the anchor, bypass now rejected), test_dispatched_task_policy.py
  (tracked revalidation admitted, bypass denied),
  test_coordinator_runtime.py + test_orchestration_prune_branch_permission.py
  (prune_branch scope);
• diagnostics — test_coordinator_async_batch2_unit.py,
  test_claude_backend_branches_unit.py (capture opt-in),
  test_session_paths_unit.py;
• reference script — test_reference_script.py::test_parse_env_self_defaults;
• the eval probe — test_eval_probe.py (new, 19 cases) execs the probe body
  against stub lm-eval modules. Hermetic, and it pins the one thing nothing else
  can: the body ships as a string constant, so no linter or import ever checks
  it — one case compiles the embedded source for exactly that reason. Covers the
  trip threshold, no short-circuit below min_samples, a terminating model left
  alone, loglikelihood left alone, harness cache stays in sync, the gate survives
  a fresh event loop, sidecar contents and one-shot write, disable switch,
  lm-eval absent, malformed responses, and the breakdown passthrough.
  test_inferencex_patcher.py gains 6 cases in the existing fixture style: append
  ordering relative to InferenceX's own patch, heredoc terminator at column 0,
  embedded body compiles, idempotency, fail-soft on a missing anchor, 8-thread
  concurrency.

The probe was additionally verified end to end against the real
benchmark_lib.sh: patched, bash -n clean, _patch_lm_eval executed in bash,
the resulting 215-line sitecustomize.py carries both patches and compiles, and a
real interpreter started with that PYTHONPATH trips the probe and
short-circuits.

[github-actions]

CI E2E report — ✅ Succeeded

item	value
result	✅ Succeeded
model	Qwen/Qwen3-0.6B (dense)
resources	1× GPU, TP=1
PR branch	fix/zgong/explore-opt-12
commit	8e15515473a59f89247865d0b7ffe80def9f7c2c
session_id	c123fb63-64a6-4ee2-bf7c-02ea7e2cab66
queue → dispatch	0s
run time	152m 22s
total	152m 22s

details