acceptodds
Withdrawn from ICLR 2027

Route Anything: Quality–Cost Tradeoffs in Model Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An inference router chooses which model, tool, or combination should handle a task using the current state and records from earlier outcomes. We study three coupled decisions: what capabilities to combine, when to spend more computation, and what information to retain. We use a reference-relative accounting that compares delivered quality with total resource cost, including routing, verification, and information costs, and apply fractional programming to finite-menu routing with a quality floor. On CodeRouterBench and RouterBench, the accounting shows that selection changes the tasks left to a fallback: comparing both portfolios on the same tasks, instead of freezing the current portfolio's conditional outcomes, raises best-candidate selection on CodeRouterBench from 86.3–94.3% to 96.4–97.1%. With a reference policy and an allowed quality drop, maximizing quality per unit cost amounts to choosing a particular balance between quality and cost. The resulting routers remain effective when routing states are inferred from prompts and perform comparably to learned quality–cost routers. With feedback and scoring tests kept disjoint, feedback retries recover 13.9% more failures than fresh retries (0.304 vs. 0.165) across seven models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, LLM, Router, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9RHpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies several model routing problems from a unified quality–cost perspective, including model selection, portfolio expansion, cascading, retries, verification, and memory.

I find the discussion on portfolio expansion particularly interesting. When a new model is added to an existing model pool, it changes the downstream fallback task distribution. As a result, it is not sufficient to evaluate the new model only by its marginal success rate or residual coverage. The full policy needs to be reevaluated. I think this is the most valuable part of the paper.

However, beyond this point, I have substantial concerns about the current version. The paper has significant issues with novelty, experimental validation, baseline coverage, reproducibility, and overall organization. In my view, several of these issues require major redesigns of the experiments and substantial rewriting rather than minor revisions. Therefore, I would give the current version a Clear Reject.

Critical strengths

The paper points out that adding a new model changes the fallback population, so directly comparing the complete policy is more appropriate than looking only at the success rate or residual coverage of the new model. This is a practically relevant problem, and the results on CodeRouterBench provide some supporting evidence.

Critical weaknesses
  1. There is an important reproducibility concern.
    As of the time I am submitting this review, the official CodeRouterBench repository does not provide the actual query/prompt field for the ID queries. This makes me unclear about how the prompt-based experiments in this paper were conducted.

  2. For a paper submitted to ICLR 2027 with the title “Route Anything,” I find the experimental results underwhelming.

    1. Showing that a small performance drop can lead to a large cost reduction is already one of the basic goals of model routing. In particular, the large efficiency multiplier reported on CodeRouterBench appears to depend heavily on using the expensive Always-Opus policy as the reference. Once a stronger yardstick is used, the gain becomes much smaller. Therefore, the current results do not convincingly show that the proposed approach produces a new or better Pareto frontier.

    2. The main paper provides very limited comparison with either classic or recent routing methods. The appendix includes several additional baselines, but most of them are relatively early methods, and several are only the authors’ own “-style” implementations. More recent routing methods, such as UniRoute [1], CSCR [2], and R2-Router [3], are not systematically compared. Moreover, some of the baselines in the appendix already come very close to the frontier achieved by the proposed approach. This further weakens the contribution of the paper as a new routing method.

    3. The experiments mix replay, oracle verifiers, online calls, and semi-synthetic memory settings. In practice, the paper does not demonstrate a truly unified end-to-end router that jointly decides model selection, portfolio composition, retries, verification, and memory. Instead, the same accounting framework is applied to several separate case studies. I believe the current framing is substantially broader than what the experiments actually support.

    4. The new-model addition analysis is one of the stronger parts of the paper, but the advantage on RouterBench is limited. The paired selection method performs well on CodeRouterBench, but on RouterBench the gap between paired selection and the cost-aware saving baseline is very small, for example 63.9% versus 63.5%. The paper’s own bootstrap analysis also shows that the confidence intervals for these best-candidate frequency differences all include zero. The conclusions in this section should therefore be stated more cautiously.

    5. RouterBench is a classic benchmark, but its strongest model, GPT-4, is now outdated. The paper would be more convincing if the authors also validated their conclusions on newer routing benchmarks such as MMR-Bench [4] and LLMRouterBench [5].

  3. The organization of the paper needs substantial improvement.
    I appreciate the large amount of work that went into this paper. The manuscript is over 70 pages long, and the authors clearly made an effort to provide a comprehensive treatment of the problem. However, the current organization makes the paper unnecessarily difficult to follow. As a reader, I often had to spend significant effort locating the results or conclusions I actually wanted to see.

    1. In my view, some of the most interesting findings are the routing-specific observations about fallback-population shift, failure-conditioned routing, and the cost of information. However, these findings are scattered across different sections and appendices and are not given enough emphasis.

    2. In contrast, a large portion of the main paper is devoted to the unified formulation, which is relatively conventional. This makes it difficult for the reader to quickly understand what the genuinely new findings of the paper are.

    3. The paper also contains a large amount of theory in both the main text and the appendix. While this makes the presentation look comprehensive, the actual novelty of the theoretical results is not sufficiently clear. The reference-relative quality–cost formulation, quality floor, and fractional programming treatment can all be derived fairly naturally from existing quality–cost routing or cascade formulations. The authors also acknowledge that the main optimization tools are classical. At the moment, the paper does not clearly distinguish which theoretical results are genuinely new and which are mainly reformulations or reorganizations of existing ideas.

Overall, I would encourage the authors to substantially reorganize the paper around the routing phenomena and conclusions that are genuinely new, and to provide stronger evidence for the practical value of the proposed framework through more recent baselines and more unified experimental settings. This would significantly improve the clarity, persuasiveness, and potential impact of the work.

[1] Universal Model Routing for Efficient LLM Inference. ICLR, 2026.

[2] Cost-Aware Contrastive Routing for LLMs.

[3] R2-Router: A New Paradigm for LLM Routing with Reasoning. ICML, 2026.

[4] MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing. ECCV, 2026.

[5] LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. Findings of ACL 2026.

Questions for authors

Here I list the questions I am most concerned about. Please see the Weaknesses section for more details.

  1. CodeRouterBench does not publicly provide the original queries for the ID tasks. How were the prompt-based results in this paper obtained?

  2. Which results in the paper provide the strongest evidence for the broad “Route Anything” claim?

  3. How does the proposed approach compare with more recent routing methods? This is not limited to the three methods mentioned above. There are now many routing methods, including several that can reduce cost while maintaining or even improving performance.

  4. How does the proposed approach perform on newer routing benchmarks such as MMR-Bench and LLMRouterBench?

  5. The paper contains a large amount of theoretical material. What do the authors consider the most central theoretical result, and what is the most novel theoretical contribution?

Reviewer_tgwHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes one accounting scheme for inference routing that covers dispatch, cascades, portfolios, verifiers, routing signals, and memory. Every policy is compared to a named reference model on the same tasks: Q is the quality change and S the cost saving, and efficiency improves exactly when Q + η_B·S > 0. Quality preservation is a separate constraint, Q ≥ −ε. The theory has four parts. A Dinkelbach-style result finds the ratio-optimal router under a quality floor, and that router randomizes in at most one state. A finite-sample certificate for the floor comes from fixed-sequence testing along the price path. A proposition characterizes when the reference model is worth calling at all. An exact correction separates paired portfolio comparisons from residual coverage. Most experiments replay recorded outcomes on CodeRouterBench and RouterBench. These are supplemented by online retry and verifier studies and a semi-synthetic memory study.

Critical strengths
  1. Certification turns an unreliable practice into a guarantee that holds up on held-out data. On CodeRouterBench, a quality floor fitted on half of the probing tasks keeps test ΔV ≥ 0 in only 12.5% of 200 splits. With Proposition 3 certification it holds in all 200, at a median efficiency of 19.4×. RouterBench shows the same pattern: the uncertified floor holds in 0 of 200 halvings and the certified one in 200 of 200. This matters for my assessment because quality-floor routing is common, and the paper shows that the usual fitted version fails out of sample. It also supplies a fix with a finite-sample guarantee, and that fix is the part of the paper most likely to change how practitioners work.
  2. The comparisons are on matched tasks, and the paper reports what they find even when it weakens the paper's own claims. Three examples: • The paper concedes that learned quality–cost routers reach within 0.006 score of its frontier, so the floor selects an operating point rather than producing a better frontier. • It shows that scoring retries on the same tests used for feedback overstates repair. • It finds that adding a model lowers the miss-conditional fallback value in 97% of CodeRouterBench comparisons, which is why frozen conditional means misrank candidates. Evaluating on matched tasks is what exposes the fallback-population shift. This methodology is likely to outlast the specific numbers.
Critical weaknesses
  1. The theoretical contribution is largely a reparameterization, and the empirical gain over simple baselines is small. By its own Theorem 2(iii), the ratio optimum under a floor is a J_λ optimum at λ = η*_ε/(1+κ), and the paper states that "the ratio adds no new frontier point." What it adds is a way to choose the operating point (the reference model plus a tolerance) and a certificate for that point. The certificate is a real contribution. The operating-point choice is modest next to Dekoninck et al. (2025). The headline efficiencies also hide how small the increment is. Fixed Qwen3-Max dispatch already reaches 20.1× Opus efficiency. The 21-cell router's 25.5× is about +27% on top of that, while recovering 0.042 score that Qwen3-Max alone loses. So most of the "25×" comes from Opus being expensive and a single cheaper model being nearly as good, not from routing.

  2. The results depend heavily on the choice of reference model and the price table, and the paper's own numbers show this cuts against its conclusions. On CodeRouterBench the selected policies never escalate to Opus. Meanwhile ACRouter and the best fixed cascade score +0.063 and +0.056 above Opus (Table 7). The ε = 0 policy, by contrast, sits at −0.0044 with an interval that includes zero. The paper's "zero-loss" policies are therefore below simpler policies that improve quality outright. Their ranking comes from using Opus as the reference and dollars as the resource. When the reference changes to each cell's best probing model, the multiplier drops from about 25× to 3.18×. The paper also notes that a 31% cut in Opus's price would change the selected policy. With two benchmarks and one price table, it is unclear whether any specific finding carries over. The paper's unification claim depends on that.

Questions for authors
  1. Abstract claims. Will you revise the best-candidate-selection headline to report that the paired-versus-residual differences are not statistically decided, and include the RouterBench results? Will you state the retry result as percentage points? If the abstract stays as written, that lowers my Presentation score.
  2. Choice of reference. Given that the best per-cell probing model scores 0.083 above Opus and changes the multiplier from about 25× to 3.18×, what principle should a practitioner use to pick the reference? Without one, can the framework's main output (the operating point) be defended?
  3. Quality-improving baselines. ACRouter and the best fixed cascade raise quality by +0.06 over Opus. How does your framework rank these against your ε = 0 policies at matched cost? Do the certified policies dominate any quality-improving policy, or do they only operate at a different point on the same frontier?
  4. Oracle verifier in replay. How much of the result in Sections 3.1–3.2 survives with deployable verifiers? The rebuild suggests that without hidden tests only Compiler cascades enter, at 2.33× with ΔV = −0.015. Is that the realistic number for a deployment?
  5. Certification cost. Certification needs the reference model's outcome on every certification task. Roughly what fraction of the claimed savings does that consume at the sample sizes you used?
Reviewer_9MTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inference routing under a reference-relative quality constraint, with the objective of maximizing expected delivered quality per unit cost. It develops a common accounting framework for model portfolios, verification, retries, and retained information, together with finite-menu optimization and quality-floor certification. Experiments on CodeRouterBench and RouterBench examine portfolio selection, routing policies, and the effects of verifier quality and execution costs.

Critical strengths
  1. The portfolio analysis identifies a consequential evaluation pitfall. Adding a model changes the population of tasks reaching fallback, so the fallback’s previous conditional performance cannot simply be reused. Comparing complete portfolios on the same tasks provides a useful correction for evaluating model-routing systems.

  2. The evaluation examines important assumptions behind the efficiency claims. The learned-router comparisons, imperfect-verifier experiments, measured overheads, and distinction between fitted and certified quality floors make the analysis more informative than reporting cost savings alone. In particular, explicitly checking whether a fitted quality constraint holds on held-out tasks is a valuable part of the paper.

Critical weaknesses
  1. The incremental practical benefit depends on the metric and setting. On CodeRouterBench at theta=1, best-candidate selection is 97.1% for the paired criterion, compared with 86.3% for residual coverage and 95.7% for the saving criterion. These are candidate-selection frequencies, rather than task success rates. Appendix A.2 reports a paired-minus-saving difference of 1.4 percentage points, with a clustered interval of [-4.1, 5.4]. Thus, saving recovers most of the hit-rate improvement over residual coverage in this setting. However, Table 6 provides important positive evidence for the paired criterion: relative and task-level regret are lower than saving in all ten CodeRouterBench theta=1 resplits. Mean relative regret is 0.04% versus 0.19%, with standard deviations across splits of 0.03% and 0.08%, respectively. The paired correction has demonstrated value; my reservation concerns the practical size of the additional gain over an already strong cost-aware control, rather than whether any gain exists.

  2. The practical verifier experiment and the certification experiment are not yet joined empirically. In the pooled without-hidden-tests row of Table 12, the policy achieves 2.33-fold quality-per-dollar efficiency on the 334-task verifier sample. Its cost is 2.277 m$ per task versus 5.413 m$ for always-Opus, while its quality change is -0.0153 with an interval of [-0.0447, 0.0146]. This is a lower-cost operating point with a negative quality point estimate; the interval does not establish noninferiority at a zero-loss margin. Appendix K establishes a general certification procedure and reports certified results for other menus. I could not locate its empirical application to this particular charged-verifier menu and task subset. The remaining question is how much efficiency survives certification in this operational setting, rather than whether verifier charges can be included in the theory.

Questions for authors

Has independent quality-floor certification been evaluated for the pooled without-hidden-tests menu used in Table 12? If so, please identify the result and the selected quality-cost operating point, distinguishing the fitting, certification, and final evaluation samples. This would clarify how the certified results in Appendix K relate to the charged-verifier experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.