- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies reasoning-strategy routing under per-query token budgets. It separates budget violations into those occurring when a strategy is predicted feasible and those occurring after the router enters fallback. Based on this formulation, it evaluates a router combining token-cost prediction, conservative corrections, accuracy priors, and a minimum-predicted-token fallback. The experiments show that conservative prediction can sometimes increase overall violations by increasing exposure to fallback.
Critical strengths
- The paper raises an interesting question: conservative resource estimates may improve compliance among predicted-feasible choices while worsening overall behavior through fallback.
- The authors report negative results and limitations transparently, including settings where the conservative router increases violations or substantially reduces accuracy.
Critical weaknesses
-
The technical contribution is insufficiently developed. The theoretical analysis primarily consists of a total-probability decomposition and an algebraic comparison. These provide a useful organizing perspective, but do not establish a strong theoretical contribution. The implemented router also does not estimate or guarantee the conditional risk introduced in the conceptual formulation. Although this limitation is acknowledged, the paper needs to explain more clearly what its framework contributes beyond documenting the fallback failure mode.
-
The evaluation does not adequately isolate the central design choices. Fallback is central to the paper’s motivation, yet the experiments do not directly compare alternative fallback policies while holding the feasibility estimator fixed. The reported ablations also show settings where simpler variants improve accuracy, compliance, and token consumption simultaneously. In addition, low- and medium-budget routing selects only Direct or Few-Shot, limiting the evidence for the broader reasoning-strategy selection problem. Missing literal prompts and predictor coefficients further hinder independent assessment.
Questions for authors
- With the feasibility estimator held fixed, how does the proposed fallback compare with fixed Direct, fixed Few-Shot, and a policy selected to minimize calibration-set violations?
- What explains the cases where removing accuracy priors improves all three metrics? Under what conditions does the full router offer a meaningful advantage?
- Can the authors clarify which conclusions are intended as diagnostic observations and which establish the effectiveness of the proposed method?
2 comments
Thank you for the candid response and additional analysis.
While I appreciate the narrowed framing and transparency, the new results reinforce my concern that the technical contribution and empirical gains over simpler baselines remain too limited for acceptance.
I will therefore maintain my rating of 2 (Weak Rejection).
Author Response to Reviewer BBfe — Fallback Sensitivity and Ablation Evidence
Thank you for the thoughtful review and for recognizing the importance of understanding how conservative feasibility estimates interact with fallback behavior. We apologize for the delayed response; we were completing additional analyses to address the concerns raised during review. We have now uploaded a revised manuscript.
1. Comparing fallback policies while holding feasibility fixed
We agree that the original evaluation did not sufficiently isolate the fallback-policy choice. We therefore conducted a separately labeled post-review offline analysis that retains the frozen feasibility decisions and compares the original fallback with fixed Direct, fixed Few-Shot, and a policy selected to minimize violations on the risk-calibration data. The calibration-selected policy uses no sealed evaluation outcomes for selection.
At the low-budget tier, averaged across four benchmarks:
These results show that the original minimum-predicted-token fallback is not uniformly optimal. For example, on Phi, fixed Direct increases accuracy from 46.75% to 58.50%, but violations rise from 22.50% to 37.00%. Fixed Few-Shot lowers violations to 15.00%, while accuracy falls to 41.75%.
Importantly, the original D1 online records and the exhaustive D3 records were generated independently. Thus, these are counterfactual sensitivity analyses using saved outcomes, not exact replays of the original online requests.
2. Why removing accuracy priors can improve all three metrics
We agree that this behavior is important rather than incidental. The accuracy prior is estimated from a relatively small development sample and does not explicitly depend on the budget. Consequently, it can misrank strategies on held-out examples and steer selection away from shorter, more effective outputs.
In three Qwen benchmark–budget settings (ARC at medium and high budgets, and GSM8K at high budget), removing the prior weakly improves accuracy, violation rate, and mean output length simultaneously.
For example, on ARC at the medium budget, the full router achieves 72% accuracy, 8% violations, and 128.74 mean output tokens; without the prior, these become 80% accuracy, 0% violations, and 8.96 mean output tokens.
These findings expose a genuine limitation of static accuracy priors. We do not claim they are universally beneficial.
3. What the framework establishes—and what it does not
We agree that the fallback-risk decomposition follows the law of total probability and is not a novel theorem or a statistical guarantee. Our intended contribution is primarily diagnostic and empirical: conservative resource corrections can improve compliance within predicted-feasible decisions while increasing exposure to a fallback whose conditional risk may be substantially different.
The current implementation does not directly estimate conditional fallback violation probabilities, and its safety margin provides no formal compliance guarantee. Nor do the experiments establish superiority across all budgets, models, or reasoning strategies.
We have revised the manuscript to distinguish these diagnostic observations from effectiveness claims, incorporate the fallback sensitivity results, and clarify the limitations of the accuracy-prior design. We also acknowledge the restricted strategy-selection diversity and the importance of complete prompts, predictor specifications, and reproducibility details for independent assessment.
Thank you again for these constructive questions. They helped us sharpen both the empirical analysis and the scope of our conclusions.