Typical Model Routing flow: (1) Query classification - the router (trained classifier or rules) extracts features: prompt length, detected domain (code, math, creative writing), complexity (heuristic or score from a small judge LLM), historical success rate on similar tasks; (2) Model selection - the router compares features against each candidate's profile (cost/token, p50/p99 latency, quality on domain benchmarks, current availability) and picks the best under the active optimisation policy; (3) Invocation and verification - request hits the chosen model, optionally with a verifier (tests, execution check, LLM-as-judge); (4) Escalation/fallback - if the verifier rejects the response or the model fails, the router iteratively escalates to a stronger candidate. Variants: offline routing (before inference), online (during streaming), agentic (per step in a tool chain), speculative (parallel calls with quorum voting).
Frontier LLMs are expensive (GPT-5 in the USD 10-30 per 1M tokens range), while quality gaps versus cheaper models (USD 0.10-1.00 per 1M) are marginal for most real tasks. Without routing, the user pays for the flagship model even on trivial queries. Model Routing solves this 'one model fits all' choice by dynamically matching compute power to actual task difficulty and business criteria (cost, latency, privacy, availability).
Decision component - can be rule-based (if-then), ML classifier (LoRA on a small judge LLM like Qwen 0.8B), MoA (mixture-of-agents), or an LLM agent invoking a routing function. Input: query features. Output: model identifier from the candidate pool.
Official
Registry of available backend models with their profiles: input/output/cache cost, p50/p99 latency, domain benchmark results, supported modalities, context limits, current availability (health check). Updated dynamically from production metrics.
Official
Optional response-quality validator for the selected model. Variants: unit tests (for code), execution check (running code in a sandbox), LLM-as-judge (a stronger model rating a weaker one), schema validation (JSON), format checking. Its feedback drives escalation.
Official
Rule defining the retry sequence: cheap_chain (order of cheap candidates), escalate_to (stronger model on verifier reject), max attempts k, per-query budget. Critical for agentic routers like ACRouter.
Official
Collecting production data: per-request decision, chosen_model, cost, latency, verifier result, user feedback. Powers router training or tuning (offline retraining or online RL). Without it, routing degenerates to static rules.
When the router is a strong LLM, its decision cost can exceed the savings from picking a cheap model - the 'router-as-a-tax' antipattern.
A bad verifier accepts weak responses (false positive) or rejects good ones (false negative). Either way, routing degenerates - always escalates or always sticks with cheap.
A router trained on Chatbot Arena data performs worse on real coding requests. Without a feedback loop, routing degenerates over time.
Try-then-escalate serially accumulates latency: cheap model (2s) + verify (0.5s) + expensive model (5s) = 7.5s instead of 5s without routing. Critical for real-time UX.
Cascading business logic into the router (prompts, verifiers, policies) creates a new silo - hard to move to another routing provider without redesign.
Chen, Zaharia, Zou (arXiv 2305.05176) show that an LLM cascade (cheap → expensive with a condition) can lower cost by 98% while retaining GPT-4 quality on selected tasks.
Anyscale describes production OSS + closed-source routing to lower cost - the first public case study of a router in a company.
The service offers a single API to hundreds of models with automatic selection and fallback - popularises the pattern for developers.
Open-source framework for training routers (matrix factorization, BERT classifier) using Chatbot Arena data - the first published ML router training methodology.
Startups positioning routing as the main product - end of the 'router in a library' phase, start of 'router as SaaS'.
Zhou et al. (arXiv 2606.22902) - router as an LLM agent with escalation and a verifier, OOD176 and CodeRouterBench benchmarks. Introduces 'agentic routing' and demonstrates integration with Claude Code / Codex / Opencode.
Number of models available to the router. More = finer selection granularity and better Pareto optimisation, but also higher catalogue, monitoring, and decision cost.
Variant of the decision component. Impacts complexity, decision cost, quality, and tunability.
Metric that the router minimises/maximises - fundamentally shapes the decision policy.
Maximum number of router attempts per query - controls worst-case cost.
Key distinction from MoE: MoE routes between expert subnetworks within a single end-to-end trained network. Model Routing routes between independently trained, often heterogeneous models from different vendors.
The router is external to backend models - it is not weights inside a single network (as in MoE) but an infrastructure layer that picks an entire model unit to invoke.
Routing is sequential in the try-then-escalate variant (the verifier must see the result before deciding on escalation). The speculative variant runs N candidates in parallel and picks a quorum - higher cost, lower latency.
Model Routing is an infrastructure pattern - runs on any hardware where candidates are hosted. The router itself (LoRA classifier or rules) is lightweight and has no hardware preferences.
Classic rule-based routers or small classifiers (BERT, DistilBERT, Qwen 0.5B) work great on CPU - decision latency <50ms without GPU dispatch.
