acceptodds
Under review as a conference paper at ICLR 2027

CoRoute: Learning Routing Frontiers as Ordered Policy Hulls

Abstract

A routing service exposes several operating points at once, and callers expect them to agree: a query accepted at a lower tier should stay accepted when coverage or compute allowances increase. Independent per-allowance optimization, however, offers no such guarantee: a more generous tier can withdraw a query that a stricter tier accepted. We treat the routing frontier as the object to learn, rather than one operating point, and require the learned family to preserve acceptance across tiers. For a shared score and a finite library of threshold-action policies, CoRoute learns tier policies within an ordered policy hull, using first-order stochastic dominance to couple their threshold distributions. One shared draw nests realized acceptance for every input while leaving model selection flexible. The order condition is linear, enabling globally optimal joint fitting within the library. Its fitted-risk cost is zero exactly when independently optimal policies admit a coherent selection. As a reusable policy layer, CoRoute keeps router predictors and LLMs fixed and adds only tens of microseconds once scores are available. Independent service data can additionally certify coverage and bounded resource budgets. Experiments on published LLM cascades and multiple routing architectures establish withdrawal and evaluate the component. In an independent external evaluation, CoRoute reduces the mean fraction of requests losing service on at least one upgrade from about 8% to zero at a mean accuracy cost of 0.04 percentage points, while reducing the largest tier losses relative to a strong single-cut alternative.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, selective prediction, learning to defer, model cascades, stochastic dominance, linear programming, robust selective risk

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9vPAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies service withdrawal in multi-tier LLM routing: a request accepted at a lower tier can be rejected after an upgrade, even when aggregate coverage and budget increase. CoRoute represents each tier as a mixture of threshold policies and routing rules. Ordered threshold distributions and shared randomness ensure persistent acceptance, while joint optimization minimizes the fitted performance cost within a fixed policy library. Across 96 external cases, the method reduces withdrawal from 7.99% to zero, with a mean accuracy cost of 0.039 percentage points.

Critical strengths
  1. A clear problem with a theoretically grounded solution. The paper clearly distinguishes aggregate coverage from request-level persistence. The ordered-policy construction provides a way to prevent service withdrawal without retraining the underlying router or LLMs, supported by the theoretical analysis.

  2. A small accuracy cost for preventing withdrawal. The external results in Table 3 show that withdrawal can be eliminated at a small average accuracy cost. This makes the approach promising for maintaining service continuity across tiers.

Critical weaknesses
  1. The empirical gain from joint fitting over tierwise fitting remains uncertain. Appendix A.7 and the known-risk diagnostics in Appendix D demonstrate that joint fitting can help. However, the external results in Table 3 give both approaches the same rounded mean maximum-tier loss of 0.38 percentage points, and the matched comparisons in Appendix G have intervals that include zero. The authors explicitly acknowledge this limitation. Thus, the evidence for the persistent-policy construction is stronger than the evidence for a reliable additional benefit from joint fitting with estimated risks. This is a limitation of the demonstrated empirical contribution, not a flaw in the persistence guarantee.

  2. Repeated qualifications make the presentation harder to follow. The distinctions between fitted-risk guarantees and test accuracy, and between population diagnostics and held-out improvements, are important. However, these qualifications are scattered and revisited throughout the paper, which interrupts the main argument. A short overview of what each result establishes, with cross-references where appropriate, would make the exposition easier to follow.

Questions for authors
  1. Given the estimation sensitivity documented in Appendix D, is there a practical diagnostic based on fitting or validation data that would help users decide whether joint fitting is likely to improve on tierwise fitting? A discussion based on the existing results would be sufficient.
Reviewer_gTH3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRoute for LLM routing across different service tiers. It studies a problem where a request answered at a lower tier may be declined at a higher tier, even when the higher tier has a larger budget. CoRoute fits the policies for all tiers together to keep previously accepted requests accepted. It also tries to limit the loss in answer quality while meeting coverage and budget requirements. Although the authors have made good efforts, some issues still need to be addressed.

Critical strengths
  1. The paper studies a useful problem. A larger budget does not always mean that previously answered requests will still receive an answer. The paper explains this problem clearly and provides a method with an acceptance guarantee.

  2. The experiments cover several router designs and tasks. The authors report answer quality, withdrawal, cost, and coverage. These results help readers understand both the benefits and the costs of the method.

Critical weaknesses
  1. The benefit of joint fitting is not clear enough. In Tables 2–4, CoRoute performs very similarly to tierwise fitting. Both methods remove withdrawal, and their answer quality and maximum tier loss are close. It is still unclear when joint fitting is needed.

  2. The comparison with existing routing methods needs to be strengthened. Table 1 mainly tests CoRoute with different predictors, and some predictors use simplified designs. These tests are useful, but they do not fully show the advantage over complete existing methods.

Questions for authors
  1. In the Introduction and Experiments sections, the authors should explain the main benefit of CoRoute more clearly. Since simpler methods also remove withdrawal, the authors should show what extra benefit CoRoute provides and when this benefit matters.

  2. In Tables 2–4, the authors should further explain why joint fitting and tierwise fitting give similar results. It would be helpful to show whether joint fitting helps more when there are more tiers or stronger conflicts between tiers.

  3. In Table 1 and Appendix B.3, the authors should clarify which methods follow the original implementations and which use simplified designs. For example, the RouterDC-style model does not use the full original training pipeline. The authors should consider adding a complete existing routing method under the same coverage and budget requirements.

  4. In Table 3, CoRoute reduces the mean maximum tier loss from 0.95 to 0.38 points compared with single-cut mixtures, but their mean accuracy is close. The authors should show which tiers benefit most and explain the value of this improvement in practice.

Reviewer_kaZKOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoRoute learns a family of abstaining LLM routing policies with persistent acceptance across ordered service tiers. It mixes threshold–routing-rule policies, imposes stochastic dominance on threshold distributions, and uses shared randomness to obtain nested acceptance. Joint fitting minimizes maximum excess selective risk within a finite library; independent data can certify coverage and bounded expected resource use. Benchmark replays demonstrate zero withdrawal with a small average accuracy cost relative to independent fitting.

Critical strengths
  1. A precise and tractable consistency contract. The withdrawal lower bound, stochastic-order coupling, and finite-library optimization give a clean account of when persistent acceptance is possible and what fitted-risk cost it incurs. The construction in Appendix A.7 demonstrates a genuine coordination gap, rather than assuming joint fitting must help.
  2. Substantial and transparent evaluation. The paper includes optimized single-cut controls, tierwise ablations, original-cascade replays, and reserved external data. Table 3 demonstrates a useful trade-off: withdrawal falls from 7.99% to zero at a mean accuracy cost of 0.039 percentage points. Negative ablations and service failures are reported rather than omitted.
Critical weaknesses
  1. Joint fitting has limited demonstrated incremental value. In Table 3, joint and tierwise fitting both have 0.38-point mean maximum tier loss; tierwise fitting has slightly higher mean accuracy. Appendices D, G, and H do not establish a stable advantage from the optimizer alone under matched settings. Table 14's selected-pipeline gains do not isolate joint optimization. The evidence therefore supports the ordered policy class more strongly than the necessity of joint learning; the contribution should be framed accordingly.

  2. Strong withdrawal repair and certified resource control are demonstrated in different regimes. CoRoute passes only 778/864 service points in Table 3; Appendix F reports 77 budget misses, with maximum relative excess of 9.74%. Full certification instead uses a bounded resource proxy on another workload where Independent's withdrawal is only 0.084% (Section H). These results do not jointly establish certified operation on the substantial-withdrawal, variable-call-cost setting. The practical resource contract needs sharper delineation.

  3. Main quality results are not directly deployment-level results. The headline external accuracy comparisons integrate mixture weights, whereas deployment uses input-dependent fixed hashes; Section J explicitly notes that their quality can differ. Section H checks fixed-hash quality on a different workload, not the main 96-case evaluation. Moreover, the cascade guarantee assumes shared prefix realizations, scores, identifiers, and policy versions (Sections A.8 and J). The primary claims need fixed-policy replay contrasts and a clearer account of how this cross-tier state is maintained.

Questions for authors
  1. Given the matched ablations, should the main contribution be understood as the ordered policy hull rather than an empirically superior joint optimizer? Which existing result best supports choosing joint over tierwise fitting in practice?
  2. Can the external cost setting satisfy the certification assumptions using a known call-cost cap? If not, please state explicitly which deployment resources are covered by the formal guarantee and which remain empirical constraints.
  3. For the frozen deployment hash used in the main 96-case evaluation, what are the realized accuracy difference and maximum-tier-loss contrast against Independent and single-cut mixtures? These can be computed from the existing cached outcomes without new model calls.
  4. Does the deployment interface persist the first-model response and score across separately issued tier requests, or must the caller provide this shared state? Please specify the upgrade workflow covered by the guarantee.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.