- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes REIN, a reinforcement learning framework that jointly optimizes self-reflection and abstention behaviors to improve the reliability of large reasoning models. REIN structures generation into three stages: reasoning, reflection, and final answer generation. It introduces a reflection-veracity reward to align the model's self-assessment with the correctness of its preliminary answer, together with a boundary-aware abstention reward based on the success of sampled reasoning trajectories. Both objectives are optimized within GRPO.
Experiments on mathematical and commonsense reasoning benchmarks demonstrate improvements in selective accuracy and reductions in false endorsement of incorrect reasoning, while maintaining relatively high answer coverage. The framework also supports single-completion inference without additional verification or iterative refinement at test time.
Critical strengths
The paper addresses the important problem of improving reasoning reliability while avoiding unnecessary abstention. Integrating reflection and abstention into a single structured completion is practically appealing, as it avoids additional inference-time verification or multi-round correction.
The paper evaluates the approach across multiple model families and reasoning benchmarks, with component ablations and diagnostic analyses of reflection and abstention behaviors. The reported improvements in selective accuracy and false endorsement provide useful empirical evidence for the effectiveness of the proposed training strategy.
Critical weaknesses
My primary concern is the limited methodological novelty of the proposed framework. Training language models to self-evaluate and correct their reasoning through reinforcement learning has been explored in prior work such as S²R and Reflect, Retry, Reward. Similarly, learning abstention behaviors through verifiable reinforcement learning has been investigated in approaches such as TruthRL and Abstain-R1.
REIN mainly combines these established directions using a structured reasoning–reflection–answer format and additional reward terms within standard GRPO optimization. While aligning reflection with pre-reflection draft correctness and estimating a policy-relative knowledge boundary are reasonable design choices, the paper does not sufficiently demonstrate why these specific formulations constitute substantial methodological advances over simpler combinations of existing techniques.
Although the component ablations establish that reflection-veracity and abstention rewards contribute to the final results, they do not adequately isolate the necessity of the specific reward designs. For example, it remains unclear whether the group-level boundary estimator is substantially better than simpler confidence-based abstention objectives, or whether aligning reflection to the preliminary draft offers clear advantages over alternative self-evaluation supervision targets. Stronger controlled comparisons would help establish the contribution beyond conventional multi-objective reward shaping.
The boundary-aware reward determines whether a question is beyond the model's capabilities based on whether any of a finite number of sampled completions produces a correct answer. Although the authors acknowledge its policy-relative nature and provide sensitivity analyses, sampling failure does not necessarily indicate an underlying knowledge deficit. It may instead reflect limited exploration, decoding choices, or reasoning failures.
Consequently, the claimed distinction between reasoning-level and knowledge-level hallucinations is not directly established by the proposed boundary estimator. The method may primarily learn to abstain on instances with low sampled success probability, rather than distinguish fundamentally different sources of hallucination.
The reported reduction in H-Proxy is encouraging, but this metric directly measures the reflection behavior targeted by the reflection-veracity reward. Moreover, uncertain reflections are treated as non-endorsements, and the metric measures draft-level assessment rather than the correctness of final responses. Therefore, improvements in H-Proxy do not necessarily imply proportional improvements in factual reliability or reasoning capability.
While the paper includes effective accuracy and some selective prediction analyses, the experimental scope remains concentrated on short-form tasks with verifiable answers. Broader risk–coverage evaluations and tests on open-domain factual reasoning would help demonstrate whether the learned reliability behavior generalizes beyond the training and evaluation setting.
Questions for authors
Q1. What is the fundamental methodological distinction between REIN and prior reinforcement learning approaches for self-verification and abstention, such as S²R, TruthRL, and Abstain-R1? Beyond combining existing objectives, which specific design choice is responsible for the distinctive improvement?
Q2. Could the authors compare the proposed boundary-aware reward against simpler alternatives, such as confidence-based abstention rewards or reward functions based on estimated answer correctness? This would clarify whether the group-level boundary estimation is essential.
Q3. Does the reflection-veracity reward provide significant advantages over supervision based on final-answer correctness or more conventional self-evaluation objectives? A controlled comparison using different reflection supervision targets would strengthen the methodological contribution.
Q4. Since H-Proxy directly measures a behavior optimized by the proposed reward, could the authors provide additional evaluation using independent reliability measures, including the distributions of correct, incorrect, and uncertain reflection stances and more comprehensive risk–coverage analyses?
Q5. Can the authors provide evidence that the framework generalizes to knowledge-intensive or open-domain factual tasks where answer verification is less straightforward? Such results would help establish whether REIN improves general reliability rather than primarily learning task-specific behavior on verifiable reasoning benchmarks.
7 comments
Response to Reviewer b1VE (Part 2)
Could the authors compare the proposed boundary-aware reward against simpler alternatives, such as confidence-based abstention rewards or reward functions based on estimated answer correctness? This would clarify whether the group-level boundary estimation is essential.
We thank the reviewer for raising this question. Confidence (e.g., P(True)) evaluates the model's current answer, whereas Knowledge Boundary estimates whether the current policy can solve a question through multiple sampled trajectories. A low-confidence answer does not necessarily mean that the question is beyond the model's capabilities, just as a high-confidence answer does not guarantee that the model can solve it. Unlike Self-Consistency, which measures agreement among sampled answers, Knowledge Boundary checks whether at least one sampled trajectory produces a verified correct answer.
We also compare seven selective prediction signals in Appendix C. On GSM8K, Reflection Stance achieves an AURC of 0.038, compared with 0.051 for P(True) and 0.040 for Self-Consistency. On MATH, its AURC is 0.257, comparable to Self-Consistency at 0.255. Combining Reflection Stance and Self-Consistency further reduces AURC to 0.031 and 0.246, respectively.
| Selective Score | GSM8K AURC ↓ | MATH AURC ↓ |
|---|---|---|
| P(True) | 0.051 | 0.272 |
| Self-Consistency @16 | 0.040 | 0.255 |
| Reflection Stance | 0.038 | 0.257 |
| Reflection + Self-Consistency | 0.031 | 0.246 |
Finally, we compare independent boundary estimation with same-group estimation. Separating the IDK-free Probe Group from the IDK-enabled Policy Group increases Coverage from 76.4% to 93.8% and Effective Accuracy from 70.2% to 85.0%. This shows that independent boundary estimation reduces unnecessary abstentions while improving overall effective accuracy.
Does the reflection-veracity reward provide significant advantages over supervision based on final-answer correctness or more conventional self-evaluation objectives? A controlled comparison using different reflection supervision targets would strengthen the methodological contribution.
We thank the reviewer for raising this question. We choose the initial draft (), rather than the final answer (), as the supervision target because Reflection directly evaluates the initial reasoning before reflection, while the correctness of the final answer is handled by an independent Answer-Correctness Reward. If () were used to supervise Reflection, it could confuse Reflection's judgment of initial errors with the outcomes of subsequent corrections. For example, an initial draft may be incorrect, but Reflection may correctly identify the error and lead the model to produce a correct final answer. In this case, Reflection's judgment itself is still correct.
The Draft-to-Final Transition analysis in Appendix G supports this design. Among 3,220 generations, 747 initial drafts were incorrect, of which 82 were eventually corrected (Repair Rate: 11.0%). Meanwhile, the proportion of initially correct drafts that were corrupted was only 0.9%. This shows that the correctness of initial drafts and final answers is not equivalent.
Furthermore, after removing the Reflection-Veracity Reward, Final H-Proxy increases from 11.0% to 20.0%, while Coverage remains nearly unchanged (93.8% vs. 93.5%). The Reflection Intervention experiments also show that removing Reflection reduces the Repair Rate from 11.0% to 0.3%; transplanting Reflection from other questions reduces it to 4.1%, while paraphrasing that preserves the original semantics still achieves a Repair Rate of 10.4%.
Response to Reviewer b1VE (Part 3)
Since H-Proxy directly measures a behavior optimized by the proposed reward, could the authors provide additional evaluation using independent reliability measures, including the distributions of correct, incorrect, and uncertain reflection stances and more comprehensive risk–coverage analyses?
We thank the reviewer for pointing out the need to evaluate REIN's reliability beyond H-Proxy. We provide three additional analyses.
(1) Reflection Stance Distribution. Across 3,220 completions, 99.4% have parseable reflection stances: 76.4% Correct, 13.2% Wrong, and 10.4% Uncertain. Conditioned on draft correctness, 97.3% of correct drafts receive a Correct stance, while 92.2% of incorrect drafts are classified as Wrong or Uncertain. This shows that Reflection distinguishes correct and incorrect initial drafts, rather than simply favoring positive endorsements (Appendix G.2, Table 21).
(2) MCC Analysis. We further use the Matthews Correlation Coefficient (MCC) to measure the overall agreement between reflection judgments and answer correctness. MCC accounts for both false positives and false negatives. In Figure 4, REIN achieves 94.3% agreement and an MCC of 0.816. Across models and benchmarks, the macro-averaged agreement and MCC are 92.7% and 0.792, respectively (Figure 4; Appendix J, Figure 6).
(3) Risk–Coverage Analysis. We compare seven selective prediction scores on the same fixed set of single-pass REIN answers across GSM8K and MATH, using final-answer error as the risk measure. On GSM8K, Reflection Stance achieves an AURC of 0.038, outperforming P(True) (0.051) and 16-sample Self-Consistency (0.040). On MATH, its AURC is 0.257, comparable to Self-Consistency (0.255). At 90% and 80% coverage, Reflection Stance achieves selective accuracies of 92.2% and 95.3% on GSM8K, and 58.5% and 62.0% on MATH, respectively. Combining Reflection Stance with Self-Consistency further reduces AURC to 0.031 and 0.246, respectively (Appendix C, Table 10).
These results evaluate REIN from three perspectives: reflection stance quality, overall judgment agreement, and risk–coverage performance.
Can the authors provide evidence that the framework generalizes to knowledge-intensive or open-domain factual tasks where answer verification is less straightforward? Such results would help establish whether REIN improves general reliability rather than primarily learning task-specific behavior on verifiable reasoning benchmarks.
We thank the reviewer for highlighting the need for broader task coverage in our evaluation. To further assess its generalization across task types, we conducted additional experiments on AIME24 and TriviaQA.
On AIME24, REIN improves Sel.Acc. across all four models, from 6.7%–10.0% for the Base models to 16.7%–31.8%, while consistently reducing Final H-Proxy. On TriviaQA, Sel.Acc. also improves across all four models, from 46.0%–58.0% to 66.0%–69.0%, while Final H-Proxy decreases from 31.9%–42.0% to 10.3%–22.5%.
| Backbone | AIME24 Sel.Acc. (Base → REIN) | AIME24 Final H-Proxy (Base → REIN) | TriviaQA Sel.Acc. (Base → REIN) | TriviaQA Final H-Proxy (Base → REIN) |
|---|---|---|---|---|
| Qwen2.5-7B | 10.0 → 31.8 | 33.3 → 13.3 | 52.0 → 68.5 | 34.1 → 16.8 |
| LLaMA-3.1-8B | 6.7 → 20.0 | 39.3 → 18.8 | 58.0 → 69.0 | 31.9 → 10.3 |
| Mistral-7B-v0.3 | 6.7 → 16.7 | 42.9 → 20.0 | 46.0 → 67.0 | 42.0 → 15.7 |
| DeepSeek-R1-Distill-Llama-8B | 6.7 → 21.7 | 35.7 → 16.7 | 50.0 → 66.0 | 36.0 → 22.5 |
These results indicate that REIN’s reliability alignment behavior generalizes beyond the original mathematical and commonsense reasoning tasks to more challenging reasoning problems and knowledge-intensive question-answering scenarios.
Response to Reviewer b1VE (Part 1)
Thanks for your appreciation and suggestions. Here are our point-by-point responses:
REIN addresses hallucination risks arising from reasoning errors and insufficient knowledge by training models to identify erroneous reasoning and decide whether to answer based on their own problem-solving capabilities. We focus on two situations: when a model produces incorrect reasoning but still judges it to be correct, and when a model lacks the knowledge required to solve a problem but still provides an unsupported answer. To this end, REIN combines reflection evaluation with answer decisions, enabling the model to identify reasoning errors, avoid false endorsement, abstain when a problem is beyond its problem-solving capabilities, while retaining the ability to answer solvable problems.
Experiments show that REIN reduces the endorsement of incorrect initial reasoning and improves selective accuracy while maintaining high answer coverage. The entire process is completed within a single generation, without additional reasoning verification or multi-round iterations.
We thank the reviewer for this comment. We clarify the methodological differences between REIN and existing approaches.
Prior work has explored RL-based self-verification and reasoning correction (e.g., S²R (Ma et al., 2025)), as well as RL-based truthful answering and abstention (e.g., TruthRL (Wei et al., 2026) and Abstain-R1 (Zhai et al., 2026)), REIN extends reliability modeling to two related but distinct levels: the reliability assessment of initial reasoning and the assessment of whether the current question is answerable.
Specifically, Reflection Alignment supervises the model to judge the correctness of the initial draft before reflection (y0), enabling the model to correct errors in its own reasoning. Boundary-aware Abstention, on the other hand, learns answering or abstention decisions based on whether the current policy can generate a correct answer to a given question.
A key design of REIN is to separate boundary estimation from the abstention policy. We use an independent IDK-free Probe Group to estimate the solvability of questions under the current policy, and separate it from the IDK-enabled Policy Group that participates in policy gradient updates. This prevents the policy's own abstention behavior from directly affecting boundary estimation.
Coverage increases from 76.4% to 93.8%, while Effective Accuracy increases from 70.2% to 85.0%. This indicates that independent boundary estimation can effectively reduce unnecessary abstentions and improve overall effective accuracy while maintaining high selective accuracy.
To further validate this distinction, we manually examined 112 questions for which all 16 sampled completions were incorrect. Among these questions, 102 (91.07%) were attributed to insufficient knowledge.
Separately, among 747 incorrect initial drafts, 82 were corrected in the final answers (11.0%). Further error-type annotation showed that 79 of these 82 corrected cases (96.34%) involved logical reasoning errors. Together with the Reflection intervention results, these findings support the respective roles of Boundary-aware Abstention in handling knowledge-related failures and Reflection in correcting reasoning errors.
References
Ma, Ruotian, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. "S²R: Teaching LLMs to Self-Verify and Self-Correct via Reinforcement Learning." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22632–22654. Association for Computational Linguistics. 2025.
Wei, Zhepei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Jingxiang Chen, Mohammad Kachuee, Teja Gollapudi, Yiwei Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen-Tau Yih, and Xin Luna Dong. "TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning." Proceedings of the 43rd International Conference on Machine Learning, 306, pp. 134156–134175. PMLR. 2026.
Zhai, Haotian, Jingcheng Liang, and Dongyeop Kang. "Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL." Findings of the Association for Computational Linguistics: ACL 2026, pp. 19674–19695. Association for Computational Linguistics. 2026.