RFC: Classification-First Routing & Heuristic Task Classification

[YasserYG8]

Discussion Title

RFC: Classification-First Routing & Heuristic Task Classification

---

Discussion Body

Hello David,

I want to propose the design for our next major feature: Classification-First Routing.

Our pre-pivot experiments showed that a naive "start cheap, escalate on failure" routing strategy actually loses money (e.g., GPT-4o-mini failing on a hard task and then escalating to GPT-4o is 31% more expensive than just using GPT-4o from the beginning).

To solve this, we are designing a Heuristic Task Classifier that determines the difficulty of a task before sending it to a provider, routing it directly to the right model tier.

---

The Core Proposal: Heuristic Task Classifier

To keep Modlane local, privacy-friendly, and blisteringly fast, we will use deterministic heuristics (no ML model/API latency) to classify requests in < 1ms.

We will classify incoming requests into three tiers:

1. FAST (e.g., GPT-4o-mini / Gemini Flash): For simple read-only operations and small context sizes.
2. BALANCED (e.g., Claude 3.5 Sonnet / GPT-4o): For standard coding, writing files, and medium context.
3. POWERFUL (e.g., O1-Pro / reasoning models): For hard debugging, active test failures, repeated edits, or massive context.

---

­The Three Classification Heuristics

1. Context Size (Character Count)

• FAST Threshold: Requests under 15,000 characters.
• POWERFUL Threshold: Requests exceeding 40,000 characters (where smaller models start losing retrieval accuracy).

2. Intent / Tool Use (Read vs. Edit)

• Read-Only: If the agent is only calling read/search tools (e.g., read_file, grep_search, list_dir), it gets routed to FAST.
• Modification: If the agent is using code-modification tools (e.g., write_to_file, replace_file_content), it goes to BALANCED.

3. Execution History (Loops & Failures)

• We inspect the incoming message history for active error signals:
  • Test Failures: Presence of exit status, command failed, or test assertion failures in tool_result messages.
  • Consecutive Failures: Repeated tool execution errors.
  • Repeated Edits: Editing the same file multiple times in a row (indicating the model is stuck in a loop).
  • Action: Instantly escalate the task to POWERFUL to break the failure loop.

---

­ Decision Flow

graph TD
    A[Incoming Request] --> B{Any errors, loops, or huge context?}
    B -- Yes --> C[Route directly to POWERFUL]
    B -- No --> D{Small context AND read-only tools?}
    D -- Yes --> E[Route directly to FAST]
    D -- No --> F[Default to BALANCED]

Loading

---

­Key Benefits

• Zero Latency: Classification runs in memory using simple checks.
• 80%+ Cost Savings: Diverting read/search operations (which make up the majority of agent runs) to cheaper models.
• Robust Loop Breaking: Escales to reasoning models the moment the agent struggles.

­Questions for the Community

1. Do you think 15,000 chars (FAST) and 40,000 chars (POWERFUL) are good default limits, or should they be adjustable in modlane.yaml?
2. Should we also scan prompt text for keywords like refactor, migrate, or architect to force-route to a higher tier?
3. Are there any specific coding agent tool names we should add to our EDIT_TOOLS list?

Looking forward to your thoughts and feedback!