acceptodds
Under review as a conference paper at ICLR 2027

The Routing Plateau: Understanding the Accuracy Limits of LLM Routers

Abstract

LLM routing has become a popular approach to improve the cost–quality trade-off of LLM services by adaptively selecting a model for each query. Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches. Our extensive study of 21 routing methods across five benchmarks reveals a consistent phenomenon that we call the routing plateau (Fig. 1): many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router. Our analysis supports a correctness-prediction bottleneck hypothesis: current routers primarily learn global-average model performance trends rather than fine-grained, query-specific routing signals. As a result, they collectively fail on queries that require instance-specific routing decisions. Moreover, to understand whether the plateau can be alleviated with a better training setup, we construct a 300K-query benchmark (9x300k). More data, larger encoders, and end-to-end fine-tuning improve eight routers by 1.24 percentage points on average, but leave the plateau largely intact. These findings suggest that further progress may require inputs beyond the query itself, such as partial output trajectories that reveal how models attempt the task.

Primary area:
datasets and benchmarks
Keywords:
LLM routing, model selection, correctness prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_x4YHOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper conprehensively studies most the existing routing methods including the 21 routing methods across five common benchmarks: routerbench, nineby30, embedllm, r2bench and carrot. It finds that the simple KKN can achieve comparable performance compared with some SOTA methods. It further studies if finetuning on a larger dataset can resolve the issue that the classifier struggles on domain-specific queries.

Critical strengths
  1. The writing of this work is clear and readers can easily understand all the findings.
  2. This paper offers a systematic empirical study. It considers most SOTA routers and uses five common benchmarks.
  3. The authors also conduct boostrap comparisons and fine tuning experiments which make the work complete.
Critical weaknesses
  1. I feel the scope of the final conclusion might be limited. In the new 5k testing dataset, most of them are reading and QA queries, so I feel the plateau finding may not be easily generalized to wider domains.
  2. I may not understand the oracle definition very well, but I feel there is some gap between oracle and the best single model that we cant close. For example, if both model A and B can solve a question with probability 0.7, then the P(correct)=0.7 whatever the router model is. However, if the oracle knows the result in the hindsight, then its accuracy is 1-(0.3)^2=0.91, which means there are 0.21 gap we can never resolve. I think Appendix B.5 partially addresses this with the repeated sampling, but Table 22 computes the majority-vote oracle while still using the original router accuracy (0.684) as the reference. I feel the router should also be evaluated under the same majority-vote labels. Therefore, I feel the claim that around 93% of the oracle gap remains does not fully rule out generation randomness. This is my major concern. Can you provide a matched evaluation?
Questions for authors

Please see the above weakness section.

2 comments
AuthorsOpenReview ↗

Remindar to participate in discussion

Thank you again for your review. As the discussion period comes to an end, we wonder whether our responses have addressed your concerns. If any remain, we would be glad to clarify or provide any further results :)

AuthorsOpenReview ↗

A quick summary for your final assessment

Dear Reviewer x4YH,

For your final assessment, we summarize your two concerns and our responses.

  1. Matched evaluation of router and oracle (your major concern): Your reasoning about oracle inflation is correct, and we tested it on real data, scoring the router and the oracle on the same labels (each model's expected score over 10 samples). The top routers stay at 68.2–68.4%, while the oracle drops from 82.7% to 80.5%. However, the oracle inflation is only 2.1 pp of the 14.3 pp gap, so 85% of the gap remains. Generation randomness is therefore not the main cause of the gap below the oracle; the correctness-prediction bottleneck is, as routers cannot predict from the query whether a given model will succeed.

  2. Generalization beyond reading and QA: QA-type questions are very common in routing benchmarks because their answers are easy to verify. To test a wider domain, we ran routers on Chatbot Arena, using real open-ended requests ranging from creative writing to coding, with human preference as the label. Routers trained on 2.6k–82.1k votes plateau again: at every training size, the top five lie within 0.41 pp, and doubling the data from 41.0k to 82.1k votes changes each router by at most 0.6 pp. The plateau, therefore, extends to a broader domain and is not limited to QA.

We hope this summary helps your final assessment.

Reviewer_37vVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates 21 LLM routing methods across five benchmarks. It finds that many methods, including kNN, achieve similar accuracy below a per-query oracle. The authors propose a correctness-prediction bottleneck and introduce Nine-by-300k to study data scaling, larger encoders, and fine-tuning. These interventions improve the eight evaluated routers by 1.24 percentage points on average.

Critical strengths
  1. The comparison covers 21 methods and five benchmarks under a shared evaluation protocol. It provides a useful check on whether complex routers improve over simple baselines such as kNN.

  2. Nine-by-300k expands the training data for the same nine models while keeping the validation and test splits fixed. This helps isolate the effect of data scale.

  3. The paper examines model-selection patterns, hard queries, and pairwise error exchange. The repeated-sampling experiment also addresses whether oracle accuracy depends strongly on single-response labels.

Critical weaknesses
  1. I feel the paper could say more about how much of the oracle gap a better router could actually close. The oracle can pick a correct response after seeing all models' outcomes, so part of its advantage may come from random successes that a router cannot predict before generation. Table 22 provides useful evidence that oracle accuracy remains high with majority labels from ten samples. However, its gap of 13.3 percentage points compares the new oracle accuracy of 81.7% with the original router accuracy of 68.4%. Without the router's accuracy under the new labels, it is still unclear whether about 93% of the gap remains when both are scored in the same way. The contribution of generation randomness to the gap also remains unquantified.

  2. The hard query definition and reported counts appear inconsistent. Section 5.1 defines hard queries in Nine-by-30k as those answered correctly by at most three models. Table 21 reports that 3,663 of 5,000 queries are solved by at least three models. This leaves 1,337 queries with fewer than three correct models, which should all count as hard under the same threshold. Yet Figure 4 reports only 702. There may be another filtering step, such as removing queries on which every model fails. Without knowing which queries were included, it is difficult to check Table 3's claim that a small hard subset accounts for most of the oracle gap.

  3. Figure 1 does not show whether accuracy would keep improving with more computation. The paper already tests λ = 0, so the evaluated routers can choose a model without a cost penalty. With a larger budget, a system could also collect responses from several models before choosing an answer. The current experiments do not cover this setting, leaving open whether these strategies would also reach a plateau and how their gains would change as the budget grows.

Questions for authors
  1. It would help to score the existing router choices using the majority labels and report the new gap with a confidence interval. The trained routers can be reused for this check. Since repeated responses have already been collected, the authors could also examine how often each model answers each query correctly. This could help estimate how much of the gap comes from variation in the responses.

  2. Could the authors explain how the 702 hard queries were selected, especially whether queries on which every model fails were removed? Reporting the counts before and after filtering would help. It would also be useful to state the correctness threshold and explain how the gap share in Table 3 is calculated, then check that these details agree across Section 5.1, Figure 4, and Table 21.

  3. It would be useful to add more points to the curves showing cost and accuracy, and mark the λ = 0 results. As an additional experiment, the authors could query all candidate models and use a practical rule to choose an answer. Reporting the full cost, including answer selection, would help show whether the extra computation brings further gains and how those gains change as the budget grows. The curve's shape could then be assessed from the results.

1 comment
AuthorsOpenReview ↗

Remindar to participate in discussion

Thank you again for your review. As the discussion period comes to an end, we wonder whether our responses have addressed your concerns. If any remain, we would be glad to clarify or provide any further results :)

Reviewer_jNguOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study of 21 routing methods across five benchmarks and identifies a “routing plateau,” where leading routers achieve similar accuracy while remaining substantially below an oracle. It attributes this phenomenon to a correctness-prediction bottleneck and investigates whether additional training data, larger encoders, and end-to-end fine-tuning alleviate it. The authors also construct Nine-by-300k to support these experiments.

Critical strengths
  1. The experimental effort is substantial. In particular, expanding the training data from approximately 23K to 302K queries while retaining shared validation and test splits provides a useful setting for studying data scaling.
  2. The paper is clearly written and well organized. The progression from observing the plateau to analyzing routing behavior and testing training interventions is easy to follow.
  3. The unified evaluation and comparison with simple baselines such as kNN are useful empirical contributions.
Critical weaknesses
  1. Limited coverage of routing paradigms. Despite the number of methods, the evaluation largely follows a query-embedding-based formulation. It is unclear whether the plateau extends to LLM-based routers or agentic routers that explicitly analyze task requirements and model capabilities. Using an LLM as an encoder does not test this alternative.

  2. A representation bottleneck is not ruled out. The encoder comparisons and fine-tuning experiments are useful, but the evaluated encoders may share similar limitations in capturing model-specific routing signals. Why use Qwen2.5-0.5B rather than a Qwen embedding model or stronger LLM-based representations? The results support a plateau under the tested representations, but do not sufficiently distinguish this from a broader correctness-prediction bottleneck.

  3. The input setting limits the conclusions. The paper suggests that progress may require partial output trajectories, but does not adequately investigate richer pre-execution information, such as model-capability profiles or retrieved success/failure experiences. While kNN already uses historical performance data, this does not exhaust how such information could support routing.

  4. Cost prediction deserves more attention. The paper includes cost-utility evaluations, but selects architectures for accuracy and applies cost scalarization afterward. This does not directly evaluate jointly predicting correctness probabilities and query-dependent costs, such as expected output-token counts. It should also clarify whether routing-time costs are predicted or obtained from recorded outputs.

Questions for authors

See weaknesses. Overall, I find the empirical observations more convincing than the proposed explanation. The analysis does not sufficiently distinguish a correctness-prediction bottleneck from limitations of the evaluated representations and training setups. I would consider raising my score if the rebuttal convincingly addresses this concern.

3 comments
AuthorsOpenReview ↗

Remindar to participate in discussion

Dear Reviewer jNgu,

You noted that you would consider raising your score if the rebuttal convincingly distinguishes a correctness-prediction bottleneck from the limits of the evaluated representations and training setups. To address this, we added LLM-based routers, encoders with up to 35B parameters, and hyperparameter and loss ablations, and the plateau holds across all of them. We also show that even a slightly better correctness prediction would raise routing accuracy steeply, yet none of these changes improves it, pointing to a correctness-prediction bottleneck. We would appreciate knowing whether this addresses your concern, and would be glad to clarify anything that remains :)

Best, Authors

Thank you for the additional experiments. The reasoning routers, retrieval ablations, and stronger encoders address my main empirical concerns and make the routing plateau more convincing in the tested setting. The predicted-cost results also clarify the cost evaluation. Given the stronger empirical evidence, I am raising my rating from 2 to 3.

AuthorsOpenReview ↗

Appreciation!

Thank you for engaging with our responses and for raising your rating. We will add the reasoning routers, retrieval ablations, stronger encoders, and predicted-cost results to the revision. Your comments are very constructive to the paper, and we really appreciate your carefulness and effort!

We wish you all the best, and the best of luck with your own work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for AC

Dear AC,

Below, we present a summary of our contributions along with our responses to the reviewers' concerns.


Our contribution

We identify and explain the routing plateau of LLM routers, with:

  1. A unified benchmark of 21 routers on five benchmarks, where the leading routers, including kNN, plateau about 14 pp below the oracle,
  2. Nine-by-300k, a 302K-query training set that isolates the effect of data scale,
  3. An analysis that traces the plateau to a correctness-prediction bottleneck, and
  4. Interventions (more data, larger encoders, fine-tuning) that gain only 1.24 pp, as the bottleneck predicts.

Reviewers' recognition

All reviewers recognized the scale of our evaluation and the value of its findings.

  • The evaluation is substantial and systematic:
    • Reviewer jNgu (2→3): "The experimental effort is substantial."
    • Reviewer 37vV (3): "The comparison covers 21 methods and five benchmarks under a shared evaluation protocol."
    • Reviewer x4YH (2): "This paper offers a systematic empirical study. It considers most SOTA routers and uses five common benchmarks."
  • The comparison with simple baselines is a useful contribution:
    • Reviewer jNgu (2→3): "The unified evaluation and comparison with simple baselines such as kNN are useful empirical contributions."
    • Reviewer 37vV (3): "It provides a useful check on whether complex routers improve over simple baselines such as kNN."
  • Nine-by-300k is a useful setting for data scaling:
    • Reviewer jNgu (2→3): "[...] provides a useful setting for studying data scaling."
    • Reviewer 37vV (3): "This helps isolate the effect of data scale."
  • The analysis is thorough, and the paper is clear:
    • Reviewer jNgu (2→3): "The progression from observing the plateau to analyzing routing behavior and testing training interventions is easy to follow."
    • Reviewer 37vV (3): "The paper examines model-selection patterns, hard queries, and pairwise error exchange."
    • Reviewer x4YH (2): "The writing of this work is clear, and readers can easily understand all the findings."

Responses to reviewers' concerns

We present a summary of all concerns and our responses.

  1. Matched evaluation of router and oracle (New results): Scoring both with each model's expected score over 10 samples, the top routers stay at 68.2–68.4% and the oracle at 80.5%. A single sample inflates the oracle by 2.1 pp, but 85% of the gap remains, and our claim holds.
    • Reviewer x4YH (2): W2 (stated as the major concern), Reviewer 37vV (3): W1/Q1
  2. Generalization beyond reading and QA (New results): On Chatbot Arena, open-ended requests labeled by human preference, routers trained on 2.6k–82.1k votes plateau again, with the top five within 0.41 pp. The plateau and its prediction bottleneck therefore extend beyond QA, to predicting the human-preferred response.
    • Reviewer x4YH (2): W1
  3. Representation limit vs. correctness-prediction bottleneck (New results): Encoders from 22M to 35B parameters, 1,196 hyperparameter configurations, and four loss families all stay on the plateau (at most 69.38%). Yet removing just 1% of kNN's correctness-prediction error would reach 70.99%, so even a slightly better prediction would show. The plateau is therefore a correctness-prediction bottleneck, not a limit of the representations or training setups we tested.
    • Reviewer jNgu (2→3): W2/Q1 (stated as the main concern; confirmed addressed)
  4. LLM-based and agentic routers (New results): A prompted reasoning LLM (68.12%) and Router-R1, trained with RL to reason before routing (68.18%), both stay below the best benchmarked router (68.38%).
    • Reviewer jNgu (2→3): W1 (confirmed addressed)
  5. Capability profiles and retrieved success/failure cases (New results): Given both, an LLM router reaches 68.1–68.2%, on par with kNN (68.36%), which already routes by retrieved successes and failures.
    • Reviewer jNgu (2→3): W3 (confirmed addressed)
  6. Cost prediction (Clarification, new results): Routing-time costs are the recorded costs. Predicting them lowers utility by only 0.07–0.32 pp, and 97% of the router–oracle gap remains.
    • Reviewer jNgu (2→3): W4 (confirmed addressed)
  7. Hard-query selection (Clarification): Tab. 21 used a different hard-query threshold from §5.1; aligning the two leaves its conclusion unchanged.
    • Reviewer 37vV (3): W2/Q2
  8. Calling several models per query (New results): Majority voting over the kNN router's top-k models adds about 2 pp at 4.5× the cost, all on easy queries; accuracy drops on the hard queries that hold most of the gap.
    • Reviewer 37vV (3): W3/Q3

Reviewer jNgu (2→3), who had four concerns, found them addressed and raised the rating. Reviewer x4YH has not yet responded; we answered both of their concerns with the matched evaluation (point 1) and generalization beyond QA (point 2) above. We hope this summary helps the AC assess them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.