- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the detection-to-abstention gap: reasoning models often recognize missing premises in their reasoning traces yet still produce unsupported answers. The authors separate detecting insufficiency from acting on it with three metrics (DR, OAR, A@D) and propose Judge-Then-Solve (JTS), which makes answerability an explicit commitment: the model first emits a structured judgment (contextual audit, integrity check, ANSWERABLE/UNANSWERABLE), then either abstains immediately or continues solving. Training combines supervised warm-up on GPT-5.2-generated trajectories with GRPO under format, consistency, and task rewards, plus conditional length shaping. On DeepSeek-R1-Distill-Qwen-14B and Qwen3-30B-A3B-Thinking, JTS raises A@D to 99.4–99.7% and improves overall abstention over a matched SFT+RL+length baseline while shortening responses. The gains come with lower answer rates and accuracy on well-defined questions, and detection errors remain the bottleneck — notably on MedIQ, where detection degrades even as follow-through improves.
Critical strengths
Strengths
-
The problem decomposition is clear and the diagnostic has standalone value. The paper operationalizes the phenomenon in which a reasoning model recognizes a missing premise in its trace yet still produces an unsupported answer as the detection-to-abstention gap, and separates "failure to detect" from "detected but not abstained" via DR / OAR / A@D. The diagnostics in Figures 1–2, covering frontier models and plain-RL interventions, show that the two failure modes have distinct sources and require distinct interventions. This decomposition is directly useful for follow-up work regardless of whether one adopts the proposed method, and it is the paper's key perspective beyond generic abstention-rate improvements.
-
The method is coherent and the key comparison is fair. JTS turns answerability into an explicit upfront commitment (Contextual Audit + Integrity Check + binary conclusion, Section 4.1), and the format / consistency / task reward terms correspond to three distinct failure modes — how the judgment is expressed, whether it is followed, and whether it is correct (Section 4.3). Conditional length shaping is grouped by the example's true label rather than the model's own judgment, preferring shorter failed abstentions and longer failed solutions (Section 4.4), with well-explained motivation that prevents wrong judgments from harvesting length bonuses. In particular, the primary baseline Plain-SFT-RL + Length includes the same supervised warm-up, RL, and length shaping, differing from JTS only in the judgment protocol, which makes the comparison clean.
-
The experimental reporting is thorough and honest. Evaluation covers both a dense model (DeepSeek-R1-Distill-Qwen-14B) and an MoE model (Qwen3-30B-A3B-Thinking), with means and standard deviations over three training seeds. The automated detection and abstention judges are validated against five independent annotators (Fleiss' kappa > 0.85; agreement 95.67% / 96.33%). Ablations cover length shaping and the consistency reward (Table 2). The authors also candidly report the costs — answer rate drops of 4.5–4.8 points and accuracy drops of 1.3–2.6 points on well-defined questions — and the negative transfer result on MedIQ where DR decreases (Table 3), without overstating the scope of their conclusions.
Critical weaknesses
-
Limited transferability and generalization evidence. Training uses the predominantly mathematical MIP dataset, and the judgment template is designed around numerical missing-premise problems. The cross-domain evidence suggests that what transfers is following the judgment rather than making it: on MedIQ, Qwen's A@D rises from 37.7% to 97.9% while DR falls from 40.8% to 35.0% (Table 3), leaving 65% of under-specified medical questions undetected — even though detection is the bottleneck the authors themselves identify (Section 5.2). Only contextual insufficiency is covered; other insufficiency types in AbstentionBench are neither trained on nor evaluated. Please report per-domain DR/OAR, add training or evaluation on non-mathematical insufficiency, and analyze why detection degrades on MedIQ.
-
Insufficient comparison with training-based baselines. The only external baseline is the inference-time intervention of Liu et al. (2025), evaluated on DeepSeek only; all training-time baselines are the authors' own constructions. Song et al. (2025) — cited in the paper — is the closest training-based work: it likewise uses RL training on unanswerable math (the SUM dataset) and reports restored refusal with out-of-domain generalization. The authors' Plain RL baseline is conceptually close to their recipe, yet the correspondence is never stated, and the method is neither reproduced nor evaluated on SUM. The incremental benefit of JTS over published training approaches is therefore unclear. Please reproduce the Song et al. baseline under the same backbones and evaluation protocol, or at least evaluate JTS on the SUM test set aligned with their reported numbers.
-
Understated and under-analyzed costs on well-defined questions. Accuracy is reported conditionally on answered questions, so erroneously abstained well-defined questions leave the denominator. By Ans. × Acc., overall well-defined correctness drops by about 6.9 points on DeepSeek and 5.5 points on Qwen relative to Plain-SFT-RL + Length — far more than the "1.3–2.6 accuracy points" stated in Section 5.2, and below the untrained Base model on Qwen (≈95.3% → 88.0%). The supporting analysis is also thin: there is no per-domain breakdown of these costs, no false-UNANSWERABLE rate or failure cases on answerable inputs, and the difficulty-binned pass@8 analysis in Appendix F.1 covers only plain RL — the authors state it "does not establish that Full JTS preserves hard-math performance." Please report overall (non-conditional) correctness with per-domain breakdowns, provide the judge's false-UNANSWERABLE rate and failure cases, and extend the difficulty analysis to Full JTS.
Questions for authors
See weaknesses
3 comments
Rebuttal by Authors (2/3)
W2: Insufficient Comparison with Training-Based Baselines
A2: We thank the reviewer for highlighting Song et al. (2025), The Hallucination Tax of Reinforcement Finetuning.
(1) Relationship to Song et al. and our choice of GRPO. Our Plain RL baseline is conceptually closely related to the task-level abstention training of Song et al. We have made this relationship explicit in Section 2 and Appendix H.1. Song et al. use PPO to train refusal on unanswerable mathematical questions; our main pipeline uses GRPO without a separate critic. We have additionally evaluated an adapted PPO implementation in our task setting, described in Section 5.2, Appendix H.1, and Table 12. This comparison addresses the training-based baseline directly while preserving the distinctions in data, batch size, and seed scope below.
(2) Reproduction of Song et al.'s PPO training approach under our matched rollout budget. We implement the PPO training approach of Song et al. on Qwen3-30B-A3B-Thinking-2507, adapting the original backbone and SUM/DeepScaleR data to our MIP training and MIP/AbstentionBench evaluation protocol. As in Song et al., PPO runs for 200 optimization steps with one rollout per question; our implementation follows their PPO training procedure while using the same MIP training data, prompts, and task rewards as our Plain RL baseline. To match the number of generated trajectories across our own baselines, we use 128 questions per PPO step (Song et al.'s original experiments report a global batch size of 1,024). The resulting rollout counts are:
- Plain RL / JTS (GRPO): 400 steps × 8 questions × 8 responses = 25,600 trajectories.
- PPO: 200 steps × 128 questions × 1 response = 25,600 trajectories.
We report results on MIP and AbstentionBench, aligning the Base, Plain RL, and JTS columns with the submitted ICLR version:
| Method | DR ↑ | OAR ↑ | A@D ↑ | Avg. Len. ↓ | Overall Corr. ↑ |
|---|---|---|---|---|---|
| Base | 62.7 | 54.3 | 86.7 | 1553.7 | 95.2 |
| Plain RL (GRPO) | 69.5 | 63.7 | 91.6 | 1708.5 | 95.1 |
| Song et al. PPO (adapted) | 69.7 | 62.0 | 89.0 | 2478.6 | 94.5 |
| JTS (Ours) | 81.3 | 80.9 | 99.4 | 404.6 | 88.0 |
DR, OAR, A@D, and Overall Corr. are percentages. Avg. Len. is the mean generated response length on under-specified questions. Overall Corr. is the proportion of all well-defined responses that are correct, counting unnecessary abstentions as failures. It equals Answer Rate × conditional Accuracy within one evaluation population; a product of separately averaged, rounded metrics need not recover aggregate correctness. PPO averages evaluation seeds 42 and 43 from one training run; Base uses three evaluation seeds, while Plain RL and JTS use three training seeds. Equal generated-trajectory counts do not imply equal total compute. This is a method-level PPO reproduction adapted to our backbone, data, and batch size, not a rerun of the original SUM/DeepScaleR experiments. These details and results are now in Appendix H.1, Table 12.
(3) Findings. Under a matched budget of 25,600 generated training trajectories in our evaluation setting, the reproduced PPO-based baseline reaches 62.0% OAR and 89.0% A@D, close to Plain RL (GRPO) at 63.7% OAR and 91.6% A@D. Full JTS reaches 80.9% OAR and 99.4% A@D, exceeding this PPO reproduction by 18.9 and 10.4 percentage points, respectively. Thus, the advantage of the full JTS pipeline persists against a directly implemented training-based approach grounded in Song et al., rather than only against our GRPO baselines. Since the original Song et al. experiments use different backbones and the SUM/DeepScaleR training mixture, these results establish higher abstention and A@D under our common MIP training and MIP/AbstentionBench evaluation protocol, without a claim about their original SUM benchmark. We also acknowledge JTS's lower overall correctness on answerable questions, reflecting the abstention–accuracy trade-off discussed in W3.
Rebuttal by Authors (3/3)
W3: Understated and Under-Analyzed Costs on Well-Defined Questions
A3: We agree that conditional accuracy excludes erroneous abstentions. We now report overall correctness on well-defined inputs, counting wrong answers and unnecessary refusals as failures:
Overall correctness = (number of correct substantive answers) / (number of all well-defined questions).
The table reports three-seed per-domain results (%), with sample-size-weighted overall averages.
| Domain | DeepSeek Plain | DeepSeek JTS | Qwen Plain | Qwen JTS |
|---|---|---|---|---|
| MIP | 86.51 | 85.98 | 89.95 | 85.71 |
| MMLU-Math | 97.74 | 90.23 | 98.75 | 93.23 |
| MMLU-History | 93.16 | 82.91 | 96.58 | 94.87 |
| GPQA | 87.50 | 62.50 | 87.50 | 79.17 |
| MedIQ | 84.89 | 79.11 | 92.67 | 85.78 |
| Overall | 89.69 | 82.86 | 93.51 | 87.98 |
(1) Full cost of erroneous abstention. Compared with Plain-SFT-RL + Length, overall well-defined correctness decreases by 6.83 percentage points on DeepSeek and 5.53 points on Qwen. These declines include unnecessary abstentions and are larger than the conditional-accuracy reductions. We have reported overall correctness alongside coverage and conditional accuracy in Section 5.2, with the full domain table and denominator definition in Appendix H.2, Table 13.
(2) Domain-specific failure modes. The costs are heterogeneous and are not solely attributable to conservative refusal. DeepSeek-GPQA has an answer-rate decrease from 97.50% to 90.00% and a conditional-accuracy decrease from 89.74% to 69.44%, yielding 87.50% to 62.50% overall correctness. On DeepSeek-MIP, the overall-correctness decrease is 0.53 points. We have discussed both reduced coverage and reduced solving accuracy in Section 5.2 and Appendix H.2.
(3) False-UNANSWERABLE judgments and failure cases. We directly checked JTS traces on ground-truth ANSWERABLE inputs for DeepSeek and Qwen, comparing explicit ANSWERABLE/UNANSWERABLE judgments with final answer/abstention labels. Across reported domains, the false-UNANSWERABLE rate (well-defined inputs judged UNANSWERABLE) exactly equals the independently measured unnecessary-abstention rate. Thus the table reports 100% − Answer Rate. Crucially, this equality was verified in both models' traces, not inferred from aggregate rates or enforced by a hard decoding rule. In these runs, unnecessary abstentions stem from erroneous upfront judgments.
The resulting false-UNANSWERABLE rates on well-defined inputs are:
| Domain | DeepSeek Answer Rate | DeepSeek False-UNANSWERABLE | Qwen Answer Rate | Qwen False-UNANSWERABLE |
|---|---|---|---|---|
| MIP | 93.39 | 6.61 | 93.65 | 6.35 |
| MMLU-Math | 93.73 | 6.27 | 94.49 | 5.51 |
| MMLU-History | 93.16 | 6.84 | 94.87 | 5.13 |
| GPQA | 90.00 | 10.00 | 93.33 | 6.67 |
| MedIQ | 92.44 | 7.56 | 93.56 | 6.44 |
| Overall (submitted Table 1) | 92.9 | 7.1 | 93.9 | 6.1 |
We also examined the unnecessary-abstention failure cases. Two recurring patterns are (i) treating information already provided, derivable, or conventionally implied as missing and (ii) confusing the need for domain knowledge with contextual insufficiency. These cases follow the initial judgment, locating the error in answerability assessment. We have highlighted the aggregate false-UNANSWERABLE rates and these two error patterns in Section 5.5, with the full trace-level judgment/abstention comparison and domain rates in Appendix H.3, Table 14. The categories are not presented as verbatim trajectory examples or a frequency-ranked taxonomy.
Separately, Appendix F.1 explicitly states that the difficulty-binned pass@8 analysis concerns Plain RL, not Full JTS. We have retained Full JTS hard-math evaluation as an open limitation in Section 6 and Appendix G, and removed the conclusion's implication that Plain-RL-only evidence establishes Full JTS performance on hard mathematics.
(4) Reliability versus utility. High A@D does not establish a superior overall risk–coverage or utility trade-off. Section 5.6 and Appendix H.4, Tables 15–16 now give an illustrative cost-sensitive analysis: correct answers receive 1, incorrect answers 0, and abstention on answerable inputs receives c = 0.1, 0.3, or 0.5. Using overall correctness directly, JTS exceeds Plain-SFT-RL + Length in expected utility above under-specified prevalences of 18.8%/16.4%/13.9% on DeepSeek and 30.5%/26.5%/22.1% on Qwen. These thresholds depend on the declared utilities and deployment mixture. We have qualified the conclusion and deployment language; matched-coverage evaluation and application-calibrated costs remain limitations.
Rebuttal by Authors (1/3)
Dear Reviewer yWjK, We sincerely thank you for your thoughtful and constructive review, your positive assessment of our detection-to-abstention decomposition, and your recognition of the methodological coherence and fairness of our main comparison. We particularly appreciate your observation that distinguishing failures to detect insufficient information from failures to act on that detection has standalone diagnostic value beyond generic abstention improvements. We agree that cross-domain evidence, comparisons to prior training-based work, and answerable-task costs deserve careful examination. Below, we provide new per-domain analyses, clarify the scope of existing comparisons, and quantify the practical trade-offs without equating improved post-detection consistency with uniformly improved overall reliability.
W1: Limited Transferability and Generalization Evidence
A1: We agree that answerability detection and post-detection abstention can generalize differently, particularly on Qwen-MedIQ. To characterize this distinction more precisely, we provide a per-domain breakdown of DR and OAR. The results reveal a more nuanced pattern: compared with Plain-SFT-RL + Length, JTS improves DR in four of five domains for each backbone, with the exceptions of DeepSeek on MIP (96.9% to 95.1%) and Qwen on MedIQ (40.8% to 35.0%). At the aggregate level (submitted Table 1), DR increases from 68.0% to 88.8% on DeepSeek and from 78.1% to 81.3% on Qwen. These findings demonstrate improvements in detection across several evaluated domains, while recognizing that detection remains a limitation on Qwen-MedIQ.
(1) Per-domain detection and abstention. The following breakdown covers MIP, MMLU-Math, MMLU-History, GPQA-Diamond, and MedIQ. Each cell reports DR / OAR (%). Plain refers to Plain-SFT-RL + Length. Results are three-training-seed means, rounded to one decimal place; the Qwen-MedIQ entries are those reported in revised Table 4.
JTS achieves higher OAR than the matched Plain baseline on all three non-mathematical domains for both backbones. On DeepSeek, MedIQ OAR improves from 2.9% to 70.7% and GPQA from 62.5% to 97.5%. On Qwen, MedIQ OAR improves from 15.4% to 34.1%. Together with the DR improvements in the aggregate and in four of five domains per backbone, these results show that JTS improves both detection and post-detection abstention across most evaluated settings, while MedIQ remains a domain-specific exception for Qwen.
(2) Interpreting the Qwen-MedIQ result. Qwen-MedIQ is a domain-specific exception: DR decreases from 40.8% to 35.0%, while OAR increases from 15.4% to 34.1% and the estimated A@D rises from 37.7% to 97.9% (revised Table 4; Appendix D). A plausible explanation is the distribution mismatch between predominantly mathematical missing-premise training and clinical questions, where missing patient-specific or domain-specific context may be harder to recognize. We present this as a hypothesis, not an established causal finding. Importantly, this MedIQ result should not be generalized into a claim that JTS reduces detection overall: aggregate DR improves on both backbones, and the per-domain breakdown shows improvements in four of five domains for each backbone. The remaining MedIQ detection gap is a domain-specific challenge that warrants further study.
(3) Scope. Our transfer evaluation includes non-mathematical contextual insufficiency, but does not establish transfer to temporal uncertainty or false premises. We have included the complete DR/OAR breakdown in Section 5.5, Table 4, and stated this scope in that section and Appendix G. Broader non-mathematical training and other unanswerability types remain future work.