refactor(do): remove fable from the model selection policy - #896
Merged
Conversation
Owner directive: "our document for what agent to use should remove fable
entirely" — removed, not demoted.
- skills/meta/do/SKILL.md: drop the Fable-5 variant row, the manual-pick
set and ordering, the cross-provider escalation target, and the
felt-quality clause comparing fable to sol; reword the sentences that
depended on them.
- scripts/build-dispatch.py: drop fable from VALID_MODELS,
ANTHROPIC_MODELS, and the manual-override gate (now sonnet alone).
- hooks/routing-decision-recorder.py: drop fable from the valid-model set.
- scripts/validate_component_models.py: ALLOWED_MODELS is {sonnet, opus}.
- docs/PHILOSOPHY.md, agent-frontmatter-template.md,
component-audit-checklists.md: same removal in live instructions.
- Tests updated to match, including SUPPLIED_CLAUDE_POINTS.
Opus-4.8 and Sonnet-5 prior-measurement rows are kept as manual-only:
they are the Anthropic lane's only DeepSWE data (Opus 5 is unbenchmarked)
and cover cost, latency, context-window, and fan-out constraints.
Dated historical records keep their fable references
(docs/what-didnt-work.md, scripts/routing-ab-results/) — editing them
would falsify experiment records; neither is a live routing instruction.
Skill-eval coverageChanged skills in this PR: 1 ( Report only. CI cannot run evals (the runner needs the Mapped
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Directive
Owner, verbatim: "our document for what agent to use should remove fable entirely". PR #891 demoted Fable-5 to a labelled "prior measurement" row and a manual-only pick. This removes it.
What was removed
skills/meta/do/SKILL.md— the Fable-5 variant-table row, the manual-pick set and its ordering clause, theopenai max-power miss → fable/xhighescalation target, and the felt-quality axis clausefable > sol. Sentences that depended on fable were rewritten, not left dangling (the manual-only rationale moved from "felt-quality escalation" to the constraints the remaining rows actually serve).Code and other live routing instructions:
scripts/build-dispatch.py(VALID_MODELS,ANTHROPIC_MODELS, the("fable","sonnet")manual gate →sonnet),hooks/routing-decision-recorder.pyvalid-model set,scripts/validate_component_models.pyALLOWED_MODELS,docs/PHILOSOPHY.mdModel Policy by Task Class,skills/meta/agent-creator/references/agent-frontmatter-template.md,agents/system-upgrade-engineer/references/component-audit-checklists.md. Tests updated to match, including theSUPPLIED_CLAUDE_POINTSfixtures and the fable-only cost-monotonicity test (re-pointed at sonnet, which still exercises the same property).No agent or skill frontmatter declared
model: fable, so nothing needed migration.What was kept, and why — Opus-4.8 and Sonnet-5
The directive named fable only, so this is a judgment call, stated explicitly rather than made silently: both remaining prior-measurement rows stay, as manual-only picks.
n/ain every cell), so deleting them leaves the lane with no numbers at all and no basis for the start-low-escalate-on-miss cost argument.If the owner wants the Anthropic lane reduced to Opus 5 alone, that is a one-line follow-up, but it is a different decision from this one.
Kept as historical record
docs/what-didnt-work.md(dated A/B entry naming fable arms and judges) andscripts/routing-ab-results/(model-ab-v1PROTOCOL/REPORT,self-route-v1VERDICT). Editing these would falsify dated experiment records, and neither is a live routing instruction — they describe what was run, not what to run.skills/review/security-review/references/coverage.mdmatches the grep only on the word "Spoofable"; out of scope.Also out of scope: PR #891's component list named
skills/workflow/references/workflow-patterns.mdandskills/meta/codex/SKILL.mdas fable sites. Neither contains a fable reference on main — noted as a correction in the ADR rather than acted on.Traps checked
grep -n '\$[0-9]' skills/meta/do/SKILL.md→ empty. All costs remain plain numbers.manual_model_override=trueandmodel_effort. The gate set narrows from("fable","sonnet")tosonnet; the requirement is unchanged and still tested.Gates
ruff check+ruff format --checkclean;pytest scripts/tests/4789 passed, 1 skipped, 150 xfailed;hooks/tests/test_routing_decision_recorder.pygreen;scripts/validate-do-references.pyOK (55 mentions resolve).ADR
adr/model-selection-policy.mdamended locally (gitignored) with the directive, the retention reasoning, and the correction above.