acceptodds
Under review as a conference paper at ICLR 2027

Predictable performance is suboptimal, and what it means for model routing

Abstract

Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models’ performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model’s self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor’s advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor’s additional parameters, we bound its advantage by the model’s suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Performance prediction, Cascading, Routing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uqrQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when a separate performance predictor can beat a model’s self-confidence at predicting whether the model answers correctly. It shows that any gain in correctness log loss over self-confidence transfers exactly to the model’s answer log loss through recalibration (Proposition 1). Assuming a lowest achievable loss at each parameter budget, this gain is bounded by the model’s suboptimality plus the loss reduction available from the predictor’s parameters (Theorem 1), and a related bound covers the extra weak-model errors a learned router can capture in a two-model cascade (Theorem 2). Experiments with 16 models on four benchmarks, before and after task-specific finetuning, find that models with higher answer loss show larger predictor and routing gains, and that these gains shrink after finetuning.

Critical strengths

A clean and correct identity with a useful reading. Proposition 1 shows that a predictor’s advantage over self-confidence is exactly loss the model itself could recover. I checked the proofs of Propositions 1–3, Lemma 1, Theorems 1–2 and both corollaries and found no errors. This reframes a common practice in routing, where separate predictors are trained without self-confidence as a baseline.

Careful and transparent experiments. Predictors are retrained for every finetuned model, and the appendix reports the controls a reader would ask for: Platt calibration, three predictor sizes, within-stage correlations, and the split into weak- and strong-model errors, including results that go against the main claim. This is why I do not recommend clear rejection.

Critical weaknesses

The theory is close to tautological as stated, numerically uninformative, and not clearly separated from prior work. Assumption 1 defines the floor as the lowest loss over all models with at most P parameters on the fixed task distribution. That is a statement about which weights exist, not about trained models, and it is unobservable. Taken literally it can sit at the Bayes loss: over a finite benchmark, a lookup table attains it with few parameters. The capacity term is then zero and Theorem 1 reduces to “the gain is at most the answer loss above the Bayes loss”, which already follows from Proposition 1. If the floor is instead the empirical scaling-law frontier, the assumption need not hold for the combined model, because the predictor is trained on labelled task data. The bounds are never compared with the data. Plugging the paper’s empirical suboptimality into Theorem 2 and dropping the non-negative capacity term gives, averaged over escalation rates, roughly 10–15 percentage points at 0.05–0.1 nats and over 30 at 0.5 nats. The measured weak-error terms are 0.17–1.80 points (Table 9). Theorems 1 and 2 follow the template of Błasiok et al. (2024): a violation found by a size-k auditor that reads the model’s output yields a network of size n+k+2 with lower loss (their Lemmas 3.1–3.2), so violations are bounded by the loss drop between sizes, with the same square-root form (their Theorem 1.2), and their Appendix B covers cross-entropy. That paper is cited only in general terms.

The experiments do not test the bound, and the headline associations are largely explained by calibration and by the finetuning setup. Empirical suboptimality is answer loss minus a constant, and answer loss contains the self-confidence loss, which is also the first term of excess predictability. Miscalibration therefore raises both axes of Figures 1 and 2. A few released models on WinoGrande and CommonsenseQA have excess predictability above log 2, meaning raw self-confidence is worse than a constant base-rate predictor. On WinoGrande these points carry the correlation: Pearson is 0.98 but Spearman 0.73 (Table 6). The appendix control confirms this. Beyond two-parameter Platt calibration, the mean predictor gain on released models is only 0.005–0.072 nats (Table 8). The direction of the finetuning effect survives, but the paired-change correlations fall to 0.24, 0.40, 0.55 and 0.25; by my calculation only GSM8K is significant at 0.05. The only intervention is supervised finetuning on the task. The predictor is trained on 1,904–6,868 labelled in-distribution questions, the released model sees five demonstrations, and the finetuned model sees 5,088–32,313. The predictor’s edge over the released model may reflect who has seen task labels, which limits the routing takeaway to cases where the weak model can be finetuned on the query distribution. On GSM8K and MMLU-Pro the weak models are finetuned on reasoning traces from the model that also serves as the strong model. The fall in routing advantage may then reflect shared errors: on MMLU-Pro after finetuning, the 0.94 points of extra weak errors captured are offset by 0.87 points of extra strong errors (Table 9).

Questions for authors

Clarifications: Is the floor a minimum over all functions with at most P parameters, or over models reachable by training? For these tasks, what do you expect the capacity term to be at P around 8B and Q = 4B? How do Theorems 1 and 2 differ from the argument of Błasiok et al. (2024)? What are the excess predictability and routing advantage for the five excluded model pairs? Their accuracy fell after finetuning, so they test the relation in the other direction.

Additional evidence that would change my assessment (within the scope of the current experiments): Figures 1 and 2 with Platt-calibrated confidence as the main-text baseline, with confidence intervals. An equal-data control, for example a predictor trained on the finetuning set, or a weak model finetuned only on the predictor’s fitting split. The numerical value of the Theorem 2 bound next to the measured weak-error terms. Routing advantage on the generation tasks with a strong model that is not the finetuning teacher, and with confidence and predictor inputs from one or a few samples instead of 128.

Reviewer_ufRYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when a learned correctness predictor can outperform an LLM’s own confidence in LLM routing. It shows that improvements in correctness log loss transfer exactly to answer log loss through recalibration, and bounds this improvement using model suboptimality and added predictor capacity. It extends to a two-model cascade experiment. Experiments with 16 models on four benchmarks find that predictor advantages over confidence generally decrease after task-specific fine-tuning.

Critical strengths
  1. The connection between correctness prediction and answer loss is clear and useful.
  2. The experiments retrain predictors after fine-tuning and include predictor-size and Platt-calibration controls. These strengthen the observation that improving the answering model often reduces the additional value of a separate predictor.
Critical weaknesses
  1. I find the theoretical contribution limited. Theorem 1 follows directly from the loss decomposition and the definition of the optimum over the augmented model class. It is unclear when the bound provides meaningful quantitative information. Correctness of the derivation alone does not establish a strong explanation of routing limits.
  2. The empirical evidence does not fully establish that explanation. Empirical suboptimality is simply answer loss minus a fixed reference, and it shares a loss component with excess predictability. The strong correlations therefore provide less independent support than they initially suggest.
Questions for authors
  1. Can you demonstrate a setting where the proposed bound is quantitatively informative, and clarify what the experiments establish about its suboptimality and capacity terms?
  2. How do the paired results change when all five excluded model–task pairs are included, and uncertainty accounts for dependence between related models?
  3. How does assigning aggregate unseen-answer probability to an unsampled gold answer correspond to the common answer-space log loss assumed in the theory?
Reviewer_LqC5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates when and why an external performance predictor or learned router can outperform a language model’s own answer confidence. Theoretically, the authors prove that the advantage of an external predictor over a model's self-confidence is fundamentally upper-bounded by the candidate model’s suboptimality and the additional capacity introduced by the predictor's parameters. They also extend these theoretical bounds to two-model cascade routing. Empirically, they validate these findings across various open-weight LLMs and reasoning benchmarks. The results demonstrate that task-specific finetuning improves model optimality, which shrinks the performance gains offered by router/predictor.

Critical strengths
  1. Solid theoretical grounding. The paper mathematically links a model's predictability to its suboptimality (Proposition 1 and Theorems 1-2). The authors provide a theoretical explanation for a common phenomenon that post-generation routers mostly just compensate for uncalibrated or under-adapted base models. It does not provide "free" external oversight.

  2. Comprehensive experimental design. The author fine-tuned 16 models to reduce suboptimality, and showed that the routing advantages shrink accordingly. The experiments of simple calibration fixes (like Platt scaling) also contribute the core claim.

Critical weaknesses
  1. Claim may not be applicable to practical settings. The authors finetuned weak models directly on the exact in-distribution training splits of the evaluated benchmarks. Real-world LLMs handle open-domain, unpredictable traffic. LM will always have long-tail weaknesses (suboptimal regions) in a general-purpose setting. This is hard to be mitigated.

  2. Confidence measurement is not realistic and potentially weaken the router's position. The paper evaluates the base model's confidence using 128 sampled reasoning traces. If the authors compared routers against a base model restricted to a realistic single-pass budget, the base model's self-confidence would likely drop significantly, making lightweight routers more valuable than the paper implies.

Questions for authors

Evaluating the base model's confidence is too expensive in the paper for practical routing. How does the cascade gain of the 4B predictor change if the base model just uses log-probabilities or verbalized confidence? Does the predictor still retain a substantial advantage after task finetuning in this setup?

If the weak model undergoes broad, multi-task finetuning instead, how much of the cascade routing advantage persists?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.