acceptodds
Under review as a conference paper at ICLR 2027

Predictable but Not Actionable: A Decision-Gap Analysis of Static LLM Routing

22 views

Abstract

Large language model (LLM) routing routes queries to balance quality and cost, yet static routers plateau below achievable quality. Why this plateau persists, and how routing should be evaluated, remains open. Existing evaluations are flawed because they score routers as predictors, not policies whose switches determine value. A corrected standard separates genuine gains from measurement artifacts before deployment. We formalize the decision gap D(q), the oracle improvement over a stated prior, and decompose value into switch rate and per-switch gain. On RouterBench (36,497 queries), we evaluate query-only readers, prefill probes, and a two-stage cascade. Our protocol separates oracle-aware from selection-valid evaluation via deployable inputs. Shuffle controls and nested cross-validation govern comparisons. Ten-seed sweeps, calibration audits, and leakage-free replay validate the findings. We find headroom on 18.1% of queries, readers flipping a third of decisions for near-zero gain, and one cascade gaining under oracle-aware but losing under leakage-free replay.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, model selection, decision gap, evaluation methodology, LLM cascade

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_29i6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why static LLM routers can predict model performance without improving on a strong default model. It defines the oracle improvement over the default and decomposes the routing gains into switch frequency and the average gain per switch. Experiments on RouterBench compare query-only readers, prefill probes, and two cascade evaluation protocols. The main finding is that predictive improvements often fail to produce better routing decisions.

Critical strengths

The focus on actual gains over a stated deployment baseline is useful. Reporting switch frequency and gain per switch makes router behavior easier to interpret, and the authors explicitly report negative results and limitations of their oracle-aware experiments.

Critical weaknesses
  1. Several central conclusions do not follow from the experiments. In Table 2, with the stated shared Ridge setup, predicting sm−sm0s_m-s_{m_0} gives s^m−s^m0\hat s_m-\hat s_{m_0}, so the selected model is unchanged by construction. This does not establish that changing the training target fails to improve decisions. Likewise, Section 6.2 uses a break-even precision derived for random candidate selection to explain a probe-selected cascade. The actual switch-conditioned gains and losses are needed, and selecting the trigger threshold based on balanced accuracy does not optimize the routing value.

  2. The main cascade comparison and supporting results need substantial clarification. Table 3 attributes the difference to trigger information, but compares a nonrepresentative 5,000-query subset with the full pool. Their prior qualities and detection difficulty differ substantially. There are also unresolved numerical inconsistencies: Table 12 reports an aligned-subset prior near 0.828, whereas elsewhere it is approximately 0.717; and Table 19's displayed rows average approximately 0.711/0.912 for its prior/oracle proxies, rather than the stated 0.500/0.747.

Questions for authors
  1. What aspect of Table 2's implementation breaks Ridge's linearity in the targets? If none does, what evidence supports the claim that decision-oriented supervision was tested meaningfully?
  2. On identical held-out queries, does the oracle-aware versus leakage-free difference persist? What happens when thresholds optimize routed quality within the training folds, and what break-even precision follows from the actual probe-selected switches?
  3. Can you reconcile the prior values in Tables 10 and 12 and the aggregates in Table 19? Please also specify the tie convention required by Eq. (3): a prior can be tied for best while differing from the chosen argmax, and switching to a tied model causes no quality loss.
  4. What new insight does this work establish beyond the cited The Routing Plateau, which already discusses limited query-specific prediction and routing changes whose gains and losses cancel? Please clarify what the decision-gap framework adds and which result demonstrates that contribution.
Reviewer_feoUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies why static LLM routers can predict model performance without improving routing decisions. It defines the improvement available over a default model, analyzes model-switching behavior, and compares evaluations with and without access to test-time outcome information.

Critical strengths

The distinction between prediction quality and actual routing gains is useful. The paper also highlights legitimate concerns about tied labels and unavailable test-time information, and reports multi-seed results and negative findings.

Critical weaknesses
  1. The main findings offer limited new insight. The switch decomposition is a straightforward identity, and several conclusions are expected: a strong default model leaves fewer opportunities for improvement, better prediction metrics do not guarantee better decisions, and using test-time scores can inflate results. These observations can motivate an analysis, but the experiments do not yet provide a sufficiently deeper explanation of the routing plateau.

  2. The measurement explanation is not fully tested. The rejudging study shows that finer-grained scores reduce ties, but the paper does not rerun the routing experiments under those scores to establish whether the plateau changes. Similarly, the candidate-prefill comparison changes the target and evaluation subset, while the oracle-aware and leakage-free cascade results use different query pools. These comparisons do not cleanly isolate the factors used to explain the results.

  3. The definition of routing value is too narrow. The main analysis focuses on improving quality over the strongest default model. However, maintaining similar quality at much lower cost is also a useful routing outcome. Appendix J reports precisely this behavior for some readers. The broader claim that these predictions are “not actionable” therefore needs qualification.

  4. The presentation makes the argument unnecessarily difficult to follow. Several simple ideas are introduced through overlapping terminology, and the figures are not sufficiently self-contained. Figure 1 also contains a concrete inconsistency: its right panel shows values of 0.703 and 0.739, corresponding to per-model score discrimination, while the caption describes gap-detection values of 0.539 and 0.567. These are different prediction tasks and should be clearly separated.

Questions for authors

See weaknesses.

Reviewer_dxqaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A router may predict model scores reasonably well but still choose the wrong queries. This paper studies the difference between prediction accuracy and useful routing decisions. The results show that several routers switch often but gain little quality.

Critical strengths

The distinction between prediction accuracy and routing value is useful. Equation 4 gives a simple way to explain routing results. Reporting switch frequency and gain per switch can reveal problems that prediction metrics miss.

The experiments examine several possible explanations for the results. The paper includes multiple splits, alternative baseline models, different representations, shuffled-feature controls, and a judge-resolution audit.

Reporting a deployable cascade that fails to improve quality is also useful.

Critical weaknesses

Some central interpretations look incorrect or unsupported. When the baseline ties with another model, switching to that model causes no quality loss. The paper repeatedly treats switches on baseline-optimal queries as guaranteed losses, despite reporting a 93% tied-best rate. Equation 3 also needs a tie-breaking rule that favors the baseline. In addition, the 0.676 break-even precision comes from random candidate selection. It does not establish the threshold required by the actual probe-based cascade.

The cascade comparison changes more than the available information. Table 3 compares the oracle-aware cascade on 5,000 selected queries with the deployable cascade on the full pool. The deployable threshold also maximizes balanced accuracy rather than routing utility. The comparison therefore does not isolate the effect of removing observed test-query scores.

Questions for authors

Can both cascade variants be evaluated on the same queries and splits, with thresholds chosen using held-out routing utility? Can you report beneficial, neutral, and harmful switches separately and derive break-even conditions for the actual selection policy? How does the Ridge cost of $0.00020 per query relate to its reported 34% switch rate from GPT-4? Please provide matching selection counts and cost calculations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.