Skip to content

refactor(do): remove fable from the model selection policy - #896

Merged
notque merged 1 commit into
mainfrom
refactor/do-remove-fable
Jul 25, 2026
Merged

refactor(do): remove fable from the model selection policy#896
notque merged 1 commit into
mainfrom
refactor/do-remove-fable

Conversation

@notque

@notque notque commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Directive

Owner, verbatim: "our document for what agent to use should remove fable entirely". PR #891 demoted Fable-5 to a labelled "prior measurement" row and a manual-only pick. This removes it.

What was removed

skills/meta/do/SKILL.md — the Fable-5 variant-table row, the manual-pick set and its ordering clause, the openai max-power miss → fable/xhigh escalation target, and the felt-quality axis clause fable > sol. Sentences that depended on fable were rewritten, not left dangling (the manual-only rationale moved from "felt-quality escalation" to the constraints the remaining rows actually serve).

Code and other live routing instructions: scripts/build-dispatch.py (VALID_MODELS, ANTHROPIC_MODELS, the ("fable","sonnet") manual gate → sonnet), hooks/routing-decision-recorder.py valid-model set, scripts/validate_component_models.py ALLOWED_MODELS, docs/PHILOSOPHY.md Model Policy by Task Class, skills/meta/agent-creator/references/agent-frontmatter-template.md, agents/system-upgrade-engineer/references/component-audit-checklists.md. Tests updated to match, including the SUPPLIED_CLAUDE_POINTS fixtures and the fable-only cost-monotonicity test (re-pointed at sonnet, which still exercises the same property).

No agent or skill frontmatter declared model: fable, so nothing needed migration.

What was kept, and why — Opus-4.8 and Sonnet-5

The directive named fable only, so this is a judgment call, stated explicitly rather than made silently: both remaining prior-measurement rows stay, as manual-only picks.

  1. They are the Anthropic lane's only DeepSWE data. Opus 5 is unbenchmarked (n/a in every cell), so deleting them leaves the lane with no numbers at all and no basis for the start-low-escalate-on-miss cost argument.
  2. They answer constraints the benchmark does not measure and Opus 5 does not cover: cost step-down, latency, context window, fan-out breadth. Fable's rationale was felt quality — that rationale left with fable; theirs did not.

If the owner wants the Anthropic lane reduced to Opus 5 alone, that is a one-line follow-up, but it is a different decision from this one.

Kept as historical record

docs/what-didnt-work.md (dated A/B entry naming fable arms and judges) and scripts/routing-ab-results/ (model-ab-v1 PROTOCOL/REPORT, self-route-v1 VERDICT). Editing these would falsify dated experiment records, and neither is a live routing instruction — they describe what was run, not what to run. skills/review/security-review/references/coverage.md matches the grep only on the word "Spoofable"; out of scope.

Also out of scope: PR #891's component list named skills/workflow/references/workflow-patterns.md and skills/meta/codex/SKILL.md as fable sites. Neither contains a fable reference on main — noted as a correction in the ADR rather than acted on.

Traps checked

  • No dollar-digit (commit 758b862): grep -n '\$[0-9]' skills/meta/do/SKILL.md → empty. All costs remain plain numbers.
  • Off-policy invariant: manual picks still require manual_model_override=true and model_effort. The gate set narrows from ("fable","sonnet") to sonnet; the requirement is unchanged and still tested.

Gates

ruff check + ruff format --check clean; pytest scripts/tests/ 4789 passed, 1 skipped, 150 xfailed; hooks/tests/test_routing_decision_recorder.py green; scripts/validate-do-references.py OK (55 mentions resolve).

ADR adr/model-selection-policy.md amended locally (gitignored) with the directive, the retention reasoning, and the correction above.

Owner directive: "our document for what agent to use should remove fable
entirely" — removed, not demoted.

- skills/meta/do/SKILL.md: drop the Fable-5 variant row, the manual-pick
  set and ordering, the cross-provider escalation target, and the
  felt-quality clause comparing fable to sol; reword the sentences that
  depended on them.
- scripts/build-dispatch.py: drop fable from VALID_MODELS,
  ANTHROPIC_MODELS, and the manual-override gate (now sonnet alone).
- hooks/routing-decision-recorder.py: drop fable from the valid-model set.
- scripts/validate_component_models.py: ALLOWED_MODELS is {sonnet, opus}.
- docs/PHILOSOPHY.md, agent-frontmatter-template.md,
  component-audit-checklists.md: same removal in live instructions.
- Tests updated to match, including SUPPLIED_CLAUDE_POINTS.

Opus-4.8 and Sonnet-5 prior-measurement rows are kept as manual-only:
they are the Anthropic lane's only DeepSWE data (Opus 5 is unbenchmarked)
and cover cost, latency, context-window, and fan-out constraints.

Dated historical records keep their fable references
(docs/what-didnt-work.md, scripts/routing-ab-results/) — editing them
would falsify experiment records; neither is a live routing instruction.
@github-actions

Copy link
Copy Markdown

Skill-eval coverage

Changed skills in this PR: 1 (426ef925b7f3..a7dc37157144).

Report only. CI cannot run evals (the runner needs the claude CLI). Run the ablation locally to get the base->head delta.

Mapped

  • do -> eval evals/new-skills-ab-test. Run locally: make skill-eval-ablation BASE=426ef925b7f3 HEAD=a7dc37157144 SKILL=do

@notque
notque merged commit eecca51 into main Jul 25, 2026
11 checks passed
@notque
notque deleted the refactor/do-remove-fable branch July 25, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant