Skip to content

feat(router): add openai-compatible llm judge router#179

Open
panpan0000 wants to merge 3 commits into
ulab-uiuc:mainfrom
panpan0000:feat/llm-judge-router-openai
Open

feat(router): add openai-compatible llm judge router#179
panpan0000 wants to merge 3 commits into
ulab-uiuc:mainfrom
panpan0000:feat/llm-judge-router-openai

Conversation

@panpan0000

Copy link
Copy Markdown

Description

This PR adds a small custom router that uses an OpenAI-compatible judge model to choose between two backend models. It also surfaces judge latency and optional routing errors in serve logs.

Related Issues

  • Fixes #
  • Related to #

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • New router implementation
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Performance improvement
  • Code refactoring
  • Test addition or update

Changes Made

  • add llmjudgerouter as a custom router for OpenAI-compatible judge endpoints
  • keep the judge output short and structured with model, confidence, reason, and signals
  • add fail_on_routing_error and judge_ms logging in serve

Router Affected (if applicable)

  • Other / Core framework

Checklist

  • I have tested my changes locally
  • I have reviewed my own code
  • I have added/updated documentation where necessary
  • I have added tests for my changes (if applicable)
  • My changes do not introduce new warnings or errors
  • Any dependent changes have been merged and published

Test Plan

python3 -m pytest tests/test_llmjudgerouter.py tests/test_llmmultiroundrouter_decomposition.py
python3 -m pytest tests/test_plugin_system.py -q

Screenshots (if applicable)

Additional Notes

  • The judge router uses only the OpenAI-compatible /chat/completions path.
  • The prompt is intentionally short so users can tune it for their own provider.

@panpan0000 panpan0000 force-pushed the feat/llm-judge-router-openai branch from faf0af6 to ae9b9fe Compare June 11, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant