- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This is an empirical paper evaluating routing in LLMs across a range of benchmarks and studies the question in particular of whether the size of a router matters, whether feeding the router the answer + confidence helps. The paper argues that cost-accuracy curves only measure how well a router ranks requests and therefore propose also measuring budget adherence (i.e. whether a threshold chosen to meet a (cost-)budget on validation data, also meets that budget on in/out of distribution data. The evaluation is based on two prediction models and reports that scaling the size of the router yields marginal benefits, routers with similar cost-accuracy can differ in realized costs when thresholds are fixed, and giving routers access to the complete response and confidence (in addition to the question) improves cost-accuracy and OOD budget adherence).
Critical strengths
- The paper reports honest statistics even if they go against the main claim of the paper. I appreciate the honesty and this strengthens the credibility of the findings.
- The question is timely, but I believe it would be more timely and practical if it concerned reasoning efforts.
Critical weaknesses
- The paper misses a comparison to RADAR as a baseline. I think this is particularly important because RADAR afaik is exactly the type of setup the paper argues against (routing based on a question-only embedding) and because RADAR beats the baseline the paper choses (RouterBench kNN). This is such a fundamental and silent omission that it drives my motivation for a Clear Reject rating.
- The paper reads like an LLM written technical writeup. Paragraphs filled with empirical results and abbreviations that make it very difficult for a human to understand. These presentation concerns are substantial.
- Recently the questions has become about how much reasoning effort to chose rather than whether to reason or not, and the paper doesn't concern/evaluate this question at all. I believe this makes the practical significance minimally.
- The results read like a mostly negative result but don't include the analysis to make this result informative enough to merit a publication at ICLR. Other reviewers may disagree with this, and I'm happy to change my opinion on this.
- While there is empirical breadth in terms of benchmarks, I think there are not many models evaluated that would support more generally meaningful findings. One family and two sizes are minor evidence for the broad scaling claim.
- The main positive result of the paper (giving routers the cheap answer + its confidence) has limited use for proprietary models where access to logits may be limited. This limits the practical usefulness of this result to open weights self-hosted models.
Questions for authors
- Why don't you compare to RADAR and completely omit it in the paper?
- Do you have any indications about whether your findings are informative about reasoning effort selections?
Thank you for taking the time to review our paper, and for pointing out the missing RADAR comparison and the presentation problems. We agree that both weakened the submission, and appreciate your recognition of the honest reporting of the results.
We significantly changed our paper's narrative, and believe it is much clearer now. We first establish a strong baseline for reasoning routing that beats all existing approaches in ID and OOD cost--accuracy. We do this by making two novel contributions. The first is to correct the CSR label from existing work, which groups harmful and neutral outcomes together. Our proposed CUR fixes this by subtracting the probability of harm from the probability of help. This is exactly the expected accuracy gain, giving the router a principled objective (lines 58–62 and Section 3.2). The second is to combine the two types of inputs used by existing routers: the question, and the direct prediction and its confidence (lines 53–56 and 342–345). These are simple changes, but the resulting improvements are robust, and span both in-distribution (ID) and out-of-distribution (OOD) settings.
Building from this strong foundation, we investigate whether scaling offers any benefit. We reach the sobering conclusion that increasing parameter size does not meaningfully improve cost--accuracy in both ID and OOD settings. Finally, we show that budget error gets worse as we scale the existing CSR router, while our proposed CUR router is unaffected. A scoring ablation shows that the probability of help and harm tend to rise and fall together, thus subtracting the latter helps to offset this effect (lines 411–430).
Weakness 1 and Question 1 — Missing RADAR comparison. We agree that RADAR was a relevant omission, so we added it to Table 1. CUR 0.8B achieves higher aggregate ID and OOD cost–accuracy (lines 318–339). Appendix A.5 explains our implementation of RADAR (lines 932–938).
Weakness 2 — Dense, difficult-to-follow writing. We have substantially rewritten the paper to improve the clarity. We aggregate results across datasets to form a simpler narrative, and moved the dense tables to the appendix (Tables 4–5, lines 1027–1029 and 1081–1083).
Weakness 3 and Question 2 — Relevance to reasoning effort selection. Our work studies routing whether to reason at all, an established and active area with confidence-based and supervised approaches. We distinguish it from routing between reasoning effort levels in Related Work (lines 99–107 and the following paragraph). Establishing a strong baseline and understanding its scaling behavior remain useful for our setting.
Weakness 4 — Mostly negative results without explanatory analysis. We now emphasize the strong positive result that the submission undersold. We establish a strong baseline for reasoning routing that improves cost--accuracy over all existing baselines that we evaluated. The improvements are robust, and span both ID and OOD settings on 17 datasets across eight task families (lines 225–239). We provide a mechanical explanation for why budget error gets worse for the flawed CSR router as we scale its parameter size, while our proposed CUR router does not suffer from this problem (lines 371–377 and 421–430).
Weakness 5 — Limited model coverage and broad scaling claims. We added the intermediate 9B size, longer 5,000-step 27B training, and five-seed evaluation (lines 240–242 and 352–359). All 30 CUR runs and 25 of 30 CSR runs selected their best validation checkpoint before the 5,000-step endpoint.
Weakness 6 — Confidence access limits proprietary-model applicability. We agree that requiring generated-token log probabilities limits the full-input router's applicability to models that expose them. This limitation is now explicit (lines 462–463).