acceptodds
Under review as a conference paper at ICLR 2027

Cost–Accuracy and Budget Adherence for Out-of-Distribution Reasoning Routing

Abstract

Reasoning routers decide when a language model should spend additional computation on a reasoning response. A useful router must both allocate reasoning to requests where it improves accuracy and continue meeting a compute budget when the request distribution changes. Cost–accuracy curves measure the first requirement, but not the second: deployment uses a threshold selected before the shift. We evaluate both requirements across two prediction models, diverse reasoning tasks, and unseen-dataset shifts by selecting thresholds on source validation data and applying them unchanged at test time. Scaling supervised routers from 0.8B to 27B yields only marginal in-distribution cost–accuracy gains, no consistent out-of-distribution gains, and greater variability in budget transfer. Moreover, routers with nearly identical cost–accuracy can differ substantially in realized cost and in how much their routing rates change, showing that ranking quality alone is insufficient. Existing supervised routers use only the question as input, whereas heuristic routers use signals from the direct prediction. We combine these approaches by also giving supervised routers the completed direct prediction and its confidence. This significantly improves in-distribution and out-of-distribution cost–accuracy at both 0.8B and 27B, and substantially reduces out-of-distribution budget-adherence error at 27B. Across our evaluated settings, the 0.8B router with these inputs remains the practical default. Reasoning routers should therefore be evaluated using cost–accuracy, cost adherence, and routing-rate adherence together under distribution shift.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning routing, adaptive inference, test-time compute, cost-aware inference, budget adherence, distribution shift, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YnMaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This is an empirical paper evaluating routing in LLMs across a range of benchmarks and studies the question in particular of whether the size of a router matters, whether feeding the router the answer + confidence helps. The paper argues that cost-accuracy curves only measure how well a router ranks requests and therefore propose also measuring budget adherence (i.e. whether a threshold chosen to meet a (cost-)budget on validation data, also meets that budget on in/out of distribution data. The evaluation is based on two prediction models and reports that scaling the size of the router yields marginal benefits, routers with similar cost-accuracy can differ in realized costs when thresholds are fixed, and giving routers access to the complete response and confidence (in addition to the question) improves cost-accuracy and OOD budget adherence).

Critical strengths
  • The paper reports honest statistics even if they go against the main claim of the paper. I appreciate the honesty and this strengthens the credibility of the findings.
  • The question is timely, but I believe it would be more timely and practical if it concerned reasoning efforts.
Critical weaknesses
  • The paper misses a comparison to RADAR as a baseline. I think this is particularly important because RADAR afaik is exactly the type of setup the paper argues against (routing based on a question-only embedding) and because RADAR beats the baseline the paper choses (RouterBench kNN). This is such a fundamental and silent omission that it drives my motivation for a Clear Reject rating.
  • The paper reads like an LLM written technical writeup. Paragraphs filled with empirical results and abbreviations that make it very difficult for a human to understand. These presentation concerns are substantial.
  • Recently the questions has become about how much reasoning effort to chose rather than whether to reason or not, and the paper doesn't concern/evaluate this question at all. I believe this makes the practical significance minimally.
  • The results read like a mostly negative result but don't include the analysis to make this result informative enough to merit a publication at ICLR. Other reviewers may disagree with this, and I'm happy to change my opinion on this.
  • While there is empirical breadth in terms of benchmarks, I think there are not many models evaluated that would support more generally meaningful findings. One family and two sizes are minor evidence for the broad scaling claim.
  • The main positive result of the paper (giving routers the cheap answer + its confidence) has limited use for proprietary models where access to logits may be limited. This limits the practical usefulness of this result to open weights self-hosted models.
Questions for authors
  • Why don't you compare to RADAR and completely omit it in the paper?
  • Do you have any indications about whether your findings are informative about reasoning effort selections?
1 comment

Thank you for taking the time to review our paper, and for pointing out the missing RADAR comparison and the presentation problems. We agree that both weakened the submission, and appreciate your recognition of the honest reporting of the results.

We significantly changed our paper's narrative, and believe it is much clearer now. We first establish a strong baseline for reasoning routing that beats all existing approaches in ID and OOD cost--accuracy. We do this by making two novel contributions. The first is to correct the CSR label from existing work, which groups harmful and neutral outcomes together. Our proposed CUR fixes this by subtracting the probability of harm from the probability of help. This is exactly the expected accuracy gain, giving the router a principled objective (lines 58–62 and Section 3.2). The second is to combine the two types of inputs used by existing routers: the question, and the direct prediction and its confidence (lines 53–56 and 342–345). These are simple changes, but the resulting improvements are robust, and span both in-distribution (ID) and out-of-distribution (OOD) settings.

Building from this strong foundation, we investigate whether scaling offers any benefit. We reach the sobering conclusion that increasing parameter size does not meaningfully improve cost--accuracy in both ID and OOD settings. Finally, we show that budget error gets worse as we scale the existing CSR router, while our proposed CUR router is unaffected. A scoring ablation shows that the probability of help and harm tend to rise and fall together, thus subtracting the latter helps to offset this effect (lines 411–430).

Weakness 1 and Question 1 — Missing RADAR comparison. We agree that RADAR was a relevant omission, so we added it to Table 1. CUR 0.8B achieves higher aggregate ID and OOD cost–accuracy (lines 318–339). Appendix A.5 explains our implementation of RADAR (lines 932–938).

Weakness 2 — Dense, difficult-to-follow writing. We have substantially rewritten the paper to improve the clarity. We aggregate results across datasets to form a simpler narrative, and moved the dense tables to the appendix (Tables 4–5, lines 1027–1029 and 1081–1083).

Weakness 3 and Question 2 — Relevance to reasoning effort selection. Our work studies routing whether to reason at all, an established and active area with confidence-based and supervised approaches. We distinguish it from routing between reasoning effort levels in Related Work (lines 99–107 and the following paragraph). Establishing a strong baseline and understanding its scaling behavior remain useful for our setting.

Weakness 4 — Mostly negative results without explanatory analysis. We now emphasize the strong positive result that the submission undersold. We establish a strong baseline for reasoning routing that improves cost--accuracy over all existing baselines that we evaluated. The improvements are robust, and span both ID and OOD settings on 17 datasets across eight task families (lines 225–239). We provide a mechanical explanation for why budget error gets worse for the flawed CSR router as we scale its parameter size, while our proposed CUR router does not suffer from this problem (lines 371–377 and 421–430).

Weakness 5 — Limited model coverage and broad scaling claims. We added the intermediate 9B size, longer 5,000-step 27B training, and five-seed evaluation (lines 240–242 and 352–359). All 30 CUR runs and 25 of 30 CSR runs selected their best validation checkpoint before the 5,000-step endpoint.

Weakness 6 — Confidence access limits proprietary-model applicability. We agree that requiring generated-token log probabilities limits the full-input router's applicability to models that expose them. This limitation is now explicit (lines 462–463).

Reviewer_txorOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies cost–accuracy and budget adherence in OOD reasoning routing. The authors argue that good cost–accuracy performance does not necessarily guarantee budget adherence under distribution shift. Experiments across 17 datasets and two prediction models suggest that scaling routers from 0.8B to 27B provides limited OOD benefits, while incorporating direct predictions and confidence improves routing performance.

Critical strengths

An interesting evaluation perspective. The paper highlights the distinction between cost–accuracy and budget adherence under distribution shift. The fixed-threshold evaluation protocol provides a useful perspective on the deployment reliability of reasoning routers.

Critical weaknesses
  1. Insufficient evidence for the scaling conclusions. The paper concludes that scaling routers from 0.8B to 27B provides limited OOD benefits, but only two model sizes are evaluated. The training dataset contains approximately 75K examples, and both models use the same training steps and LoRA configuration, raising concerns about whether the larger router is sufficiently trained. Additional experiments on training-data scaling and model-specific hyperparameter tuning are needed to support this conclusion.

  2. The routing objective is not explicitly cost-aware. CUR estimates expected accuracy improvement but ignores query-specific reasoning costs, even though reasoning costs can vary substantially across requests. This is particularly concerning because the paper identifies routed-cost drift as an important source of budget violations. Moreover, the paper mainly evaluates budget adherence rather than proposing a method to improve it, and comparisons with cost-aware routing objectives are missing.

  3. Limited experimental coverage and unclear presentation. The OOD evaluation only considers unseen datasets within known task families, leaving generalization to unseen task families and other distribution shifts unexplored. Several budget-adherence comparisons also have wide confidence intervals based on only three training seeds, weakening the conclusions. Additionally, Tables 1–2 are difficult to read.

Questions for authors
  1. How does router performance change with increasing training data, particularly for the 27B model?
  2. Have the authors explored longer training or model-specific hyperparameter tuning for the larger router?
  3. Why is query-specific reasoning cost not explicitly incorporated into CUR, and how would it compare with cost-aware alternatives?
  4. Do the budget-adherence findings generalize to unseen task families or other distribution shifts?
1 comment

Thank you for taking the time to review our paper, and for your valuable feedback. We agree that the submission was hard to follow and did not clearly explain the contribution or the limits of the scaling evidence.

We significantly changed our paper's narrative, and believe it is much clearer now. We first establish a strong baseline for reasoning routing that beats all existing approaches in ID and OOD cost--accuracy. We do this by making two novel contributions. The first is to correct the CSR label from existing work, which groups harmful and neutral outcomes together. Our proposed CUR fixes this by subtracting the probability of harm from the probability of help. This is exactly the expected accuracy gain, giving the router a principled objective (lines 58–62 and Section 3.2). The second is to combine the two types of inputs used by existing routers: the question, and the direct prediction and its confidence (lines 53–56 and 342–345). These are simple changes, but the resulting improvements are robust, and span both in-distribution (ID) and out-of-distribution (OOD) settings.

Building from this strong foundation, we investigate whether scaling offers any benefit. We reach the sobering conclusion that increasing parameter size does not meaningfully improve cost--accuracy in both ID and OOD settings. Finally, we show that budget error gets worse as we scale the existing CSR router, while our proposed CUR router is unaffected. A scoring ablation shows that the probability of help and harm tend to rise and fall together, thus subtracting the latter helps to offset this effect (lines 411–430).

Weakness 1 — Evidence from only two router sizes. We added an intermediate 9B router. Figure 2 now compares 0.8B, 9B, and 27B with error bars. The results support limited cost–accuracy gains and increasing CSR OOD budget error (lines 352–367).

Question 2 and Weakness 1 — Longer training and model-specific tuning. We trained the 27B CSR and CUR routers for 5,000 steps rather than 2,500. All 30 CUR runs and 25 of 30 CSR runs selected their best validation checkpoint before the 5,000-step endpoint. This reduces the concern about stopping too early. We have not performed model-specific hyperparameter or LoRA tuning. We acknowledge this limitation at lines 356–359.

Weakness 2 and Question 3 — Why CUR does not explicitly adjust for query-specific reasoning cost. CUR maximizes the expected accuracy gain of reasoning, which is a principled objective for determining whether to reason at all. This is an active research area and is separate from the question of routing among reasoning effort levels. While the latter is certainly interesting, it is beyond the scope of our work.

Weakness 2 and Question 3 — Comparison with cost-aware alternatives. The submission already included RACER, which explicitly trains against a budget constraint, in Appendix B.5. CUR achieves higher accuracy at matched realized cost, while RACER transfers budgets more accurately (Appendix B.4, lines 1174–1182).

Weakness 3 and Question 4 — Unseen task families and other shifts. We have not tested unseen families or established generalization to arbitrary shifts. The OOD results concern held-out datasets within familiar task families, and we acknowledge this limitation (Section 6, lines 456-458).

Weakness 3 — Wide intervals and only three training seeds. We expanded the learned-router evaluation from three to five seeds and report variation in the main table and figures (lines 240–242 and 325–329).

Weakness 3 — Difficult tables and unclear presentation. We aggregated the important results across datasets in order to simplify the narrative. We moved the detailed tables with individual dataset results to the appendix (Tables 4–5, lines 1027–1029 and 1081–1083).

Reviewer_EnTnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates reasoning routing under distribution shift, focusing on the tradeoff between cost–accuracy and budget adherence. It compares supervised routing objectives (CSR and CUR) using 0.8B and 27B routers across multiple benchmarks and OOD settings. The experiments show that scaling router size provides limited OOD benefits, while incorporating direct predictions and confidence improves the cost–accuracy trade-off.

Critical strengths
  1. The empirical evaluation is comprehensive.
  2. The practical motivation is clear and important.
Critical weaknesses
  1. Limited novelty: the proposed methods that largely build on existing routing approaches, with relatively straightforward modifications.
  2. Modest empirical gains: the performance gains are modest, and the proposed routers do not consistently outperform other baselines.
  3. Limited generalization: the OOD evaluation focuses on known task families, limiting the generality of the conclusions.
  4. Writing could be improved: the paper is difficult to follow, and its main contributions and technical insights are not clearly presented.
Questions for authors
  1. Could the authors clarify the methodological novelty relative to existing supervised reasoning routers?
  2. Given that the proposed routers achieve worse OOD budget adherence than several simpler baselines, what supports CUR as the practical default?
1 comment

Thank you for taking the time to review our paper, and for recognizing the breadth and practical motivation of the evaluation. We agree that the submitted version was difficult to follow and undersold its positive contribution.

We significantly changed our paper's narrative, and believe it is much clearer now. We first establish a strong baseline for reasoning routing that beats all existing approaches in ID and OOD cost--accuracy. We do this by making two novel contributions. The first is to correct the CSR label from existing work, which groups harmful and neutral outcomes together. Our proposed CUR fixes this by subtracting the probability of harm from the probability of help. This is exactly the expected accuracy gain, giving the router a principled objective (lines 58–62 and Section 3.2). The second is to combine the two types of inputs used by existing routers: the question, and the direct prediction and its confidence (lines 53–56 and 342–345). These are simple changes, but the resulting improvements are robust, and span both in-distribution (ID) and out-of-distribution (OOD) settings.

Building from this strong foundation, we investigate whether scaling offers any benefit. We reach the sobering conclusion that increasing parameter size does not meaningfully improve cost--accuracy in both ID and OOD settings. Finally, we show that budget error gets worse as we scale the existing CSR router, while our proposed CUR router is unaffected. A scoring ablation shows that the probability of help and harm tend to rise and fall together, thus subtracting the latter helps to offset this effect (lines 411–430).

Weakness 1 and Question 1 — Limited novelty and the methodological contribution. As mentioned above, we make two novel contributions to improve upon all baselines in terms of cost--accuracy. These are simple changes, but the resulting improvements over all baselines are robust, and span both in-distribution (ID) and out-of-distribution (OOD) settings.

Weakness 2 — Modest gains and inconsistent baseline improvements. CUR 0.8B significantly outperforms all evaluated existing baselines in aggregate ID and OOD cost–accuracy, including the additional RADAR comparison (Table 1, lines 318–339). The input ablation shows that adding the completed direct prediction and confidence improves cost–accuracy over the question alone (lines 342–345).

Weakness 3 — Limited generalization. We agree that holding out datasets within familiar families does not establish transfer to unseen families. We retain this limitation from the submission and make the scope of the scaling conclusions clearer (lines 456–460). However, we believe that holding out datasets from known task families is a very realistic setting, and our results are demonstrated extensively using 17 datasets across eight task families (lines 225–239).

Weakness 4 — Writing and unclear contributions. We agree that the previous narrative was confusing, and have made significant changes to improve it. We reframed the narrative around establishing the strong baseline that beats all existing approaches, the scaling question, and the explanatory ablation. Sections 5.1–5.3 follow that same order. The main figures now state their takeaways in the captions, and the prose explains what the results mean instead of repeating dense tables of numbers.

Question 2 — Why recommend CUR despite worse OOD budget adherence than simpler baselines? We recommend CUR over the baselines because it achieves better cost--accuracy than all baselines in both ID and OOD settings. CUR also has similar budget error to methods that have similar cost--accuracy. The heuristic baselines have better budget error, but they have significantly worse cost--accuracy (lines 318–324; Table 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.