Route-then-Probe: A Two-Stage Cascade for Cost-Effective LLM Routing

Routing algorithms reduce the cost of LLM service by selecting a suitable model for each request from a pool of models with varying sizes, capabilities, and prices. We observe that a well-matched small model can answer many requests at a fraction of the cost of a flagship model. However, even a well-matched small model can fail, so realizing these savings requires two decisions: which small model to select and whether to escalate to a stronger model. We propose Route-then-Probe (RTP), a two-stage cascade that first selects exclusively among small models and then uses the inference-time signals of the selected model to decide whether to escalate. In a single forward pass, a lightweight router jointly encodes each request and the features of all candidate small models to select the model most likely to answer correctly. A calibrated probe gate then examines the hidden states of the selected model and determines, before decoding, whether the estimated accuracy gain from escalation justifies its cost. Existing routers typically select directly from a mixed pool of small and flagship models using request-side evidence alone, which may be insufficient to predict per-request success. Hidden-state probes provide model-specific correctness signals, but probing every candidate incurs substantial inference overhead and requires access to the internals of each model. RTP exploits complementarity between small models while probing only the selected model, avoiding the overhead of probing the entire pool. Experiments with seven self-deployed small models and an API-based flagship model on a mixed dataset spanning more than ten domains show that RTP achieves better cost–accuracy trade-offs than existing routing and cascading methods across the evaluated cost range. Its modular architecture also accommodates existing routers as first-stage selectors, extending the benefits of selective probing to other routing algorithms.