- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies selective answering when incorrect responses incur an explicit cost. It proposes Reinforced Hesitation (RH), which uses a ternary reward of +1 for correct answers, 0 for abstention, and −λ for incorrect answers. Training with different penalties produces models with different coverage–error trade-offs. The paper evaluates this behavior on Knights & Knaves, MATH, and medical QA, including transfer from MedMCQA to MedQA. It also introduces cascading across models trained at different penalties and self-cascading through repeated sampling after abstention. These strategies use learned abstention to allocate inference computation and obtain competitive accuracy with fewer model calls than fixed-budget majority voting.
Critical strengths
-
The motivation is compelling and practically meaningful. The paper addresses an important mismatch between accuracy-oriented training and deployment settings where incorrect answers can be substantially more costly than abstentions. I appreciate the explicit treatment of this trade-off and the evaluation across reasoning and knowledge-based tasks. The targeting analysis in Appendix I also provides useful evidence that learned abstentions concentrate on problems where the baseline is more likely to fail.
-
Using abstention to drive adaptive inference is the most appealing contribution. Cascading and self-cascading give learned abstention a concrete computational role. The MATH cascade reaches 89.97% correct with 1.58 average model calls, compared with 91.05% for 16-sample majority voting. Although this does not establish uniform dominance, it is a promising efficiency trade-off. This part of the paper substantially strengthens my interest in the work beyond the reward modification alone.
Critical weaknesses
-
The algorithmic novelty of the training objective is limited relative to closely related work. At λ = 1, RH's content reward coincides with TruthRL's default ternary reward. TruthRL also investigates alternative abstention rewards (Table 5 of its June 2026 version), and the submission acknowledges the relationship to reward sweeps in Jha et al. Thus, both explicit abstention training and adjusting the reward to change the coverage–error trade-off have close precedents. I view the broader risk-specialist analysis and the inference strategies as the main potential contributions; the reward formulation itself provides limited additional novelty.
-
Supporting multiple risk preferences introduces a substantial training and deployment burden. The reported specialist frontier relies on separately trained checkpoints for different penalties, while the prompt-conditioned model in Section 3.5 does not match that frontier. A deployment with one fixed risk preference can use a single specialist, and self-cascading also uses one model. However, supporting several risk preferences or the heterogeneous cascade requires maintaining multiple checkpoints, with additional training, storage, and serving requirements. The RLCR comparison in Appendix E.1, Table 6 provides encouraging evidence of higher utility for RH at nonzero penalties, but these performance gains come with reduced flexibility in changing the risk preference. This remains a practical limitation of the proposed specialist approach.
Questions for authors
Deployment clarification: Could you clarify the aggregate training, storage, and serving costs of the specialist family, and how a deployment should select among existing specialists when its error cost changes? This would help assess the practicality of the proposed approach without requiring a new training algorithm.
Author's response
Thank you for the encouraging review. We agree that the reward formulation has close precedents (Appendix A). Our contribution lies in analyzing risk-specialized models and using abstention at inference.
Training and serving costs. On MATH, each 4B specialist takes 37.7–49.8 GPU-hours to train. All five total 214.0 GPU-hours versus 34.5 for RLCR, or roughly 6×, excluding the shared format warmup used by the specialists. The runs used both H100 and B200 GPUs, so this ratio is indicative. Each bf16 checkpoint occupies about 9.3 GB (46.5 GB for five).
For a fixed deployment error cost, only one specialist needs to be served, at the cost of any single model; self-cascading reuses that model. Only a heterogeneous cascade needs multiple checkpoints available. The depth-four cascade averages 1.53 calls and generates 7.6k characters per problem, compared with 64.5k for 16-sample majority voting. We have not measured latency or peak serving memory.
How many specialists? Fewer than five appear necessary. Adjacent models are often statistically tied: at deployment penalty 5, λ=1, 2, and 5 are indistinguishable in paired tests. The coarse family {0, 1, 10} contains a best or tied-best model at each evaluated penalty on MATH, so three training runs may suffice for those settings.
Choosing a model when costs change. Retraining is not needed. For each specialist, utility is linear in the deployment penalty, U(λ) = P(correct) − λP(wrong). One evaluation pass on development data defines each line; selecting the highest line gives the best estimated specialist at any new cost. On MATH validation problems, this upper envelope switches from the λ=0 specialist to λ=1 at a cost of about 0.17, to λ=5 near 5.7, and to λ=10 above 7.6. Thus the three-model grid covers the evaluated penalties, while finer cost ranges may justify retaining λ=5.
RLCR trades some utility for one adjustable checkpoint. At every nonzero evaluated cost, the matched specialist beats thresholded RLCR by +0.015 to +0.186, with paired confidence intervals excluding zero (Appendix E.1, Table 6). For a known cost, deploy the corresponding upper-envelope specialist; for varying costs, retain a small family and switch models. If one checkpoint is a hard constraint, thresholding RLCR remains a viable alternative with the measured utility loss.