- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CALIBER studies budgeted routing between direct answering and extended reasoning at two model scales. A lightweight text encoder predicts the success probability of each route from the question and a cheap initial model pass. These predictions are calibrated and combined with costs to select a route, with optional deferral. Experiments on MMLU-Pro and SuperGPQA evaluate accuracy, calibration, cost, and distribution shift. CALIBER improves over the selected confidence cascade in distribution, but misses its capability-retention target and shows no significant advantage over the cascade after target adaptation.
Critical strengths
S1. Executing all four routes on every question provides a useful basis for comparing routing policies, examining differences between routes, and measuring the gap to an outcome oracle.
S2. The evaluation separates training, calibration, and testing, includes tuned confidence cascades, and reports multiple router seeds and a paired confidence interval for the main comparison.
S3. The paper clearly reports its negative results, including the unmet retention target, the limited effect of temperature scaling, and the improvement obtained by removing the small reasoning route. These findings provide useful evidence about when additional routing options help.
Critical weaknesses
W1. The binary baseline omits the large direct route, a key contributor to CALIBER’s performance. The comparison therefore mixes routing quality with differences in available options. A text-based baseline using the same routes and budget is needed, including a direct comparison with Route to Reason.
W2. In the shift experiment, recalibration is bundled with changes to cost estimates, routing settings, and the budget. Better budget control could come from policy retuning alone. At a fixed target budget, keeping success scores unchanged while updating costs and routing settings would help isolate the contribution of recalibration.
W3. Table 4 uses the much weaker small direct model as the only deferral baseline, so the accuracy gap also reflects differences in answering ability. Compare raw and calibrated confidence on the same routed answers, and include a cascade at the same budget. Calibration should also be measured for the answers actually selected by the router.
Questions for authors
Q1. With the same encoder and training data, how does a simple router choosing between the large model's direct and reasoning modes perform at matched realized cost?
Q2. At a fixed target-domain budget, what improvement comes from probability recalibration beyond updating the action-cost estimates and operating point?
Q3. Keeping the routed answers fixed, how do raw and calibrated confidence scores compare for deferral, and how well calibrated are the selected-action scores?
2 comments
Thank you for the additional experiments. The response address my main experimental concerns. As the contribution remains limited and practical efficiency gains are unclear, I will maintain my rating of 2.
Thank you for a constructive review; the revision now includes each of the three analyses you requested.
W1 / Q1 (same routes, same encoder). We agree that the binary baseline omitted 27B direct, which mixed routing quality with the choice of available options. Every baseline now chooses among 4B direct, 27B direct and 27B reasoning under the same budget rule. The binary win classifier reaches 83.6% (73.3% in the submission), and our router minus the classifier is +1.0 [0.4, 1.7].
For Q1, we restrict the comparison to the large model's two modes at the 50% budget. A binary classifier with the same encoder and training data reaches 84.4%, our router restricted to these two actions 84.7%, and a confidence cascade from 27B direct to 27B reasoning 79.7%. Our router minus that cascade is +5.01 [3.55, 6.49]. At a matched realized cost of 1.54 GPU-s, the descriptive frontier estimates are 85.4% and 85.7% for the two learned routers and 81.2% for the cascade. The submitted router's heads without 4B reasoning (the feature variant in Table 2) minus the binary classifier is +0.10 [-0.28, 0.51]. The gain over cascades thus comes from choosing between 27B direct and 27B reasoning from the question text, and a simple binary router does almost as well. Route to Reason (reimplemented) on the same three actions reaches 83.75% (+0.9 [-0.1, 1.9]), and we claim no advantage over it (Sec. 5.2, Table 2, Appendix E).
W2 / Q2 (what restores budget adherence). We agree that the submitted recalibration combined several updates. Table 3 now fixes the target budget at half the mean 27B-reasoning cost on the target calibration split, keeps the success scores unchanged and adds one update at a time. Updating only the planning costs and the operating point, using the router's own predicted success and no labels (A), brings cost to 0.99x and 1.05x of budget. Accuracy is then 75.5% on held-out subjects and 54.9% on SuperGPQA. Choosing the operating point with labels (A+L) gives 75.1% and 54.9%. For Q2, probability recalibration on top of these updates (A+L+T) adds no accuracy relative to A: -0.01 [-0.08, 0.06] and -1.94 [-2.49, -1.40]. It lowers mean head ECE from 0.052 to 0.043 and from 0.218 to 0.101. As you suspected, better budget control comes from retuning the policy. Setting A still requires executing every action on the target calibration questions, which costs 1.87 and 3.08 GPU-h (Sec. 5.3).
W3 / Q3 (deferral on fixed answers). We agree, and we replaced the submitted comparison with 4B answers (Sec. 5.4, Table 4). On the same routed answers at the 50% budget, calibrated and raw scores rank almost identically. Both give 89.8% selective accuracy at 80% coverage, with AURC of 0.061 and 0.062. Temperature scaling is monotone within each head, so it cannot change the ranking within a head. The answering route's own confidence ranks better (93.5%; AURC 0.055), but it is known only after the route runs. The cascade at the same budget reaches 87.1% (AURC 0.097). For the selected action, calibrated and raw ECE are 0.028 and 0.033 in distribution. After target refitting, they are 0.028 and 0.063 on held-out subjects and 0.083 and 0.240 on SuperGPQA. Calibration therefore matters for the probability values, not for the ranking. Appendix G gives the same analysis for the submitted router.