- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Inhibitory Deliberation for Problem Reasoning (IDPR), a response-conditioned routing method that determines whether to return a fast-generated answer or invoke additional slow reasoning. The method employs a controller trained on paired fast–slow outcomes to predict answer quality improvement, generation cost, and corrective potential, followed by validation-based calibration for routing decisions. Experiments on mathematical reasoning problems from GSM8K and Mixture-of-Thoughts show that IDPR improves accuracy over the always-fast policy and several routing baselines at a low slow-call rate.
Critical strengths
-
The paper addresses a practical and well-motivated problem of deciding when additional reasoning is worthwhile, considering both potential corrections and harmful replacements.
-
The proposed framework is straightforward and interpretable, leveraging paired fast–slow outcomes to guide routing decisions.
-
The paper provides statistical significance tests and analyses of corrective and harmful replacements, offering useful insights into the behavior of different routing strategies.
Critical weaknesses
-
Limited methodological novelty. The core ideas of response-conditioned routing and learning from fast–slow outcome differences have been explored in prior work. The proposed method largely combines existing techniques, while its specific methodological contributions remain insufficiently distinguished from related approaches.
-
Insufficient ablations and baseline comparisons. The paper lacks key ablations to validate the necessity and individual contributions of its proposed components, particularly the quality, cost, and correction supervision, given that the final routing decision relies only on predicted quality gain. Moreover, direct comparisons with closely related learned deferral and cascading methods are missing.
-
Limited experimental scope and generalizability. The main experiments rely on a single fast–slow model pair and mathematical problems from GSM8K and Mixture-of-Thoughts, without evaluation across diverse models or task domains. Most additional analyses revisit the same experimental results rather than providing independent validation, leaving the generalizability and robustness of the proposed method unclear.
Questions for authors
-
How does IDPR meaningfully differ from existing learned deferral and cascading methods, given that response-conditioned routing, relative correctness prediction, corrective supervision, and cost-aware decisions have all been explored in prior work? What is the specific methodological contribution beyond combining these existing components, and what evidence establishes its necessity?
-
Given that the final routing decision relies only on predicted quality gain, what are the actual contributions of the auxiliary cost and correction supervision? In particular, can the authors compare quality-only training against the full multi-task design under controlled settings? If these additional components provide negligible or even negative gains, what justifies their necessity in the proposed framework? More broadly, the method introduces multiple components and design choices, yet lacks sufficient ablations and sensitivity analyses to justify key settings, such as loss weights and other hyperparameters.
-
Why does the evaluation primarily focus on heuristic baselines and RouteLLM adaptations, while closely related learned deferral and cascading methods are not directly compared? How can the claimed effectiveness of IDPR be established without comparisons to these more directly relevant approaches under matched experimental settings?
-
Can the authors evaluate IDPR on additional fast–slow model pairs and independent benchmarks beyond mathematical reasoning? Given that the main evaluation relies on a single model pair and a custom in-domain benchmark, the current results seem insufficient to establish whether the observed improvements generalize beyond this specific experimental setting.