Evaluating Escalation Signals for Optimal Language Model Routing

Cascading a small language model into a large one requires a signal that predicts, before the large model is called, whether the call is needed. We study semantic entropy over $k$ sampled answers as that signal, and evaluate it on three tasks of increasing openness: templated arithmetic, GSM8K reasoning, and text-to-SQL. On GSM8K with a Gemma-3 1B/12B pair, semantic entropy predicts the small model's own failures at AUROC $0.871$, beats a difficulty-calibrated baseline by $+0.313$, and converts into routing gains of up to $+0.093$ accuracy over random escalation at matched cost. On synthetically generated chained arithmetic it adds nothing over a regular expression on the question text: a single-family benchmark makes difficulty a free surface feature of the question, so sampling reveals no more than the question already gives away. On text-to-SQL (Gemma-3 4B/12B, $n=1200$), semantic entropy scores only $0.545$ AUROC pooled, below a free difficulty baseline ($0.850$) – but it is undefined, not weak, on the $77\%$ of items where the $k$ samples agree; on the remaining $23\%$ it scores $0.676$. Splitting on this free-to-compute distinction and calibrating each half before recombining reaches a held-out AUROC of $0.907$, our strongest result. A retrieval alternative – caching large-model outcomes and querying by embedding similarity – trades signal for cost: it appears to predict large-model success on arithmetic ($0.908$), but a probe shows this is again difficulty lookup ($\rho =0.98$), and it collapses to chance on GSM8K ($0.518$), where the same probe returns $\rho =0.23$. It costs $0.57\times$ always-large against $0.82\times$ for live sampling at its tuned budget. Framed as prediction rather than diagnosis, semantic entropy earns its keep only where the difficulty of a query cannot be read off the question itself. We report the diagnostics that produced these outcomes – a difficulty baseline, an oracle-headroom check, per-generation cost accounting, a retrieval-difficulty probe, and a regime split – against a measured AUROC noise floor of $\approx 0.04$ at these sample sizes.