- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies several model routing problems from a unified quality–cost perspective, including model selection, portfolio expansion, cascading, retries, verification, and memory.
I find the discussion on portfolio expansion particularly interesting. When a new model is added to an existing model pool, it changes the downstream fallback task distribution. As a result, it is not sufficient to evaluate the new model only by its marginal success rate or residual coverage. The full policy needs to be reevaluated. I think this is the most valuable part of the paper.
However, beyond this point, I have substantial concerns about the current version. The paper has significant issues with novelty, experimental validation, baseline coverage, reproducibility, and overall organization. In my view, several of these issues require major redesigns of the experiments and substantial rewriting rather than minor revisions. Therefore, I would give the current version a Clear Reject.
Critical strengths
The paper points out that adding a new model changes the fallback population, so directly comparing the complete policy is more appropriate than looking only at the success rate or residual coverage of the new model. This is a practically relevant problem, and the results on CodeRouterBench provide some supporting evidence.
Critical weaknesses
-
There is an important reproducibility concern.
As of the time I am submitting this review, the official CodeRouterBench repository does not provide the actual query/prompt field for the ID queries. This makes me unclear about how the prompt-based experiments in this paper were conducted. -
For a paper submitted to ICLR 2027 with the title “Route Anything,” I find the experimental results underwhelming.
-
Showing that a small performance drop can lead to a large cost reduction is already one of the basic goals of model routing. In particular, the large efficiency multiplier reported on CodeRouterBench appears to depend heavily on using the expensive Always-Opus policy as the reference. Once a stronger yardstick is used, the gain becomes much smaller. Therefore, the current results do not convincingly show that the proposed approach produces a new or better Pareto frontier.
-
The main paper provides very limited comparison with either classic or recent routing methods. The appendix includes several additional baselines, but most of them are relatively early methods, and several are only the authors’ own “-style” implementations. More recent routing methods, such as UniRoute [1], CSCR [2], and R2-Router [3], are not systematically compared. Moreover, some of the baselines in the appendix already come very close to the frontier achieved by the proposed approach. This further weakens the contribution of the paper as a new routing method.
-
The experiments mix replay, oracle verifiers, online calls, and semi-synthetic memory settings. In practice, the paper does not demonstrate a truly unified end-to-end router that jointly decides model selection, portfolio composition, retries, verification, and memory. Instead, the same accounting framework is applied to several separate case studies. I believe the current framing is substantially broader than what the experiments actually support.
-
The new-model addition analysis is one of the stronger parts of the paper, but the advantage on RouterBench is limited. The paired selection method performs well on CodeRouterBench, but on RouterBench the gap between paired selection and the cost-aware saving baseline is very small, for example 63.9% versus 63.5%. The paper’s own bootstrap analysis also shows that the confidence intervals for these best-candidate frequency differences all include zero. The conclusions in this section should therefore be stated more cautiously.
-
RouterBench is a classic benchmark, but its strongest model, GPT-4, is now outdated. The paper would be more convincing if the authors also validated their conclusions on newer routing benchmarks such as MMR-Bench [4] and LLMRouterBench [5].
-
-
The organization of the paper needs substantial improvement.
I appreciate the large amount of work that went into this paper. The manuscript is over 70 pages long, and the authors clearly made an effort to provide a comprehensive treatment of the problem. However, the current organization makes the paper unnecessarily difficult to follow. As a reader, I often had to spend significant effort locating the results or conclusions I actually wanted to see.-
In my view, some of the most interesting findings are the routing-specific observations about fallback-population shift, failure-conditioned routing, and the cost of information. However, these findings are scattered across different sections and appendices and are not given enough emphasis.
-
In contrast, a large portion of the main paper is devoted to the unified formulation, which is relatively conventional. This makes it difficult for the reader to quickly understand what the genuinely new findings of the paper are.
-
The paper also contains a large amount of theory in both the main text and the appendix. While this makes the presentation look comprehensive, the actual novelty of the theoretical results is not sufficiently clear. The reference-relative quality–cost formulation, quality floor, and fractional programming treatment can all be derived fairly naturally from existing quality–cost routing or cascade formulations. The authors also acknowledge that the main optimization tools are classical. At the moment, the paper does not clearly distinguish which theoretical results are genuinely new and which are mainly reformulations or reorganizations of existing ideas.
-
Overall, I would encourage the authors to substantially reorganize the paper around the routing phenomena and conclusions that are genuinely new, and to provide stronger evidence for the practical value of the proposed framework through more recent baselines and more unified experimental settings. This would significantly improve the clarity, persuasiveness, and potential impact of the work.
[1] Universal Model Routing for Efficient LLM Inference. ICLR, 2026.
[2] Cost-Aware Contrastive Routing for LLMs.
[3] R2-Router: A New Paradigm for LLM Routing with Reasoning. ICML, 2026.
[4] MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing. ECCV, 2026.
[5] LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. Findings of ACL 2026.
Questions for authors
Here I list the questions I am most concerned about. Please see the Weaknesses section for more details.
-
CodeRouterBench does not publicly provide the original queries for the ID tasks. How were the prompt-based results in this paper obtained?
-
Which results in the paper provide the strongest evidence for the broad “Route Anything” claim?
-
How does the proposed approach compare with more recent routing methods? This is not limited to the three methods mentioned above. There are now many routing methods, including several that can reduce cost while maintaining or even improving performance.
-
How does the proposed approach perform on newer routing benchmarks such as MMR-Bench and LLMRouterBench?
-
The paper contains a large amount of theoretical material. What do the authors consider the most central theoretical result, and what is the most novel theoretical contribution?