acceptodds
Under review as a conference paper at ICLR 2027

REIN: Bridging the Gap Between Reasoning and Reliability via Reflection and Abstention Alignment

Abstract

Large reasoning models (LRMs) are prone to hallucination, which undermines their reliability and poses challenges for safe deployment. Hallucinations in LRMs arise from two distinct failure sources: reasoning hallucination, where flawed inference steps propagate to an incorrect conclusion, and knowledge hallucination, where the model lacks the requisite factual knowledge to answer the query. To address reasoning hallucination, we propose REIN, an alignment framework that trains LRMs to produce a structured reasoning sequence, , enabling explicit self-reflection before committing to a final answer. To address knowledge hallucination, REIN introduces a reward mechanism that encourages explicit abstention (e.g., "I don't know") when none of the sampled reasoning chains yields a correct answer, allowing the model to refrain from unsupported predictions. Extensive evaluations on mathematical and commonsense reasoning benchmarks show that REIN consistently improves selective accuracy, reduces incorrect-but-self-endorsed responses, and maintains high coverage compared with competitive baselines. Notably, REIN achieves these gains within a single forward pass, without requiring process supervision, inference-time controllers, external search, or multi-round critiques. Experiments on multiple backbones show that REIN reduces the hallucination proxy by 5872% relative to the base models while maintaining 8691% average coverage, and improves selective accuracy on attempted questions by 6.614.2%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large reasoning models, hallucination mitigation, self-verification, reflection alignment, abstention, reinforcement learning, reliability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_b1VEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes REIN, a reinforcement learning framework that jointly optimizes self-reflection and abstention behaviors to improve the reliability of large reasoning models. REIN structures generation into three stages: reasoning, reflection, and final answer generation. It introduces a reflection-veracity reward to align the model's self-assessment with the correctness of its preliminary answer, together with a boundary-aware abstention reward based on the success of sampled reasoning trajectories. Both objectives are optimized within GRPO.

Experiments on mathematical and commonsense reasoning benchmarks demonstrate improvements in selective accuracy and reductions in false endorsement of incorrect reasoning, while maintaining relatively high answer coverage. The framework also supports single-completion inference without additional verification or iterative refinement at test time.

Critical strengths

The paper addresses the important problem of improving reasoning reliability while avoiding unnecessary abstention. Integrating reflection and abstention into a single structured completion is practically appealing, as it avoids additional inference-time verification or multi-round correction.

The paper evaluates the approach across multiple model families and reasoning benchmarks, with component ablations and diagnostic analyses of reflection and abstention behaviors. The reported improvements in selective accuracy and false endorsement provide useful empirical evidence for the effectiveness of the proposed training strategy.

Critical weaknesses

My primary concern is the limited methodological novelty of the proposed framework. Training language models to self-evaluate and correct their reasoning through reinforcement learning has been explored in prior work such as S²R and Reflect, Retry, Reward. Similarly, learning abstention behaviors through verifiable reinforcement learning has been investigated in approaches such as TruthRL and Abstain-R1.

REIN mainly combines these established directions using a structured reasoning–reflection–answer format and additional reward terms within standard GRPO optimization. While aligning reflection with pre-reflection draft correctness and estimating a policy-relative knowledge boundary are reasonable design choices, the paper does not sufficiently demonstrate why these specific formulations constitute substantial methodological advances over simpler combinations of existing techniques.

Although the component ablations establish that reflection-veracity and abstention rewards contribute to the final results, they do not adequately isolate the necessity of the specific reward designs. For example, it remains unclear whether the group-level boundary estimator is substantially better than simpler confidence-based abstention objectives, or whether aligning reflection to the preliminary draft offers clear advantages over alternative self-evaluation supervision targets. Stronger controlled comparisons would help establish the contribution beyond conventional multi-objective reward shaping.

The boundary-aware reward determines whether a question is beyond the model's capabilities based on whether any of a finite number of sampled completions produces a correct answer. Although the authors acknowledge its policy-relative nature and provide sensitivity analyses, sampling failure does not necessarily indicate an underlying knowledge deficit. It may instead reflect limited exploration, decoding choices, or reasoning failures.

Consequently, the claimed distinction between reasoning-level and knowledge-level hallucinations is not directly established by the proposed boundary estimator. The method may primarily learn to abstain on instances with low sampled success probability, rather than distinguish fundamentally different sources of hallucination.

The reported reduction in H-Proxy is encouraging, but this metric directly measures the reflection behavior targeted by the reflection-veracity reward. Moreover, uncertain reflections are treated as non-endorsements, and the metric measures draft-level assessment rather than the correctness of final responses. Therefore, improvements in H-Proxy do not necessarily imply proportional improvements in factual reliability or reasoning capability.

While the paper includes effective accuracy and some selective prediction analyses, the experimental scope remains concentrated on short-form tasks with verifiable answers. Broader risk–coverage evaluations and tests on open-domain factual reasoning would help demonstrate whether the learned reliability behavior generalizes beyond the training and evaluation setting.

Questions for authors

Q1. What is the fundamental methodological distinction between REIN and prior reinforcement learning approaches for self-verification and abstention, such as S²R, TruthRL, and Abstain-R1? Beyond combining existing objectives, which specific design choice is responsible for the distinctive improvement?

Q2. Could the authors compare the proposed boundary-aware reward against simpler alternatives, such as confidence-based abstention rewards or reward functions based on estimated answer correctness? This would clarify whether the group-level boundary estimation is essential.

Q3. Does the reflection-veracity reward provide significant advantages over supervision based on final-answer correctness or more conventional self-evaluation objectives? A controlled comparison using different reflection supervision targets would strengthen the methodological contribution.

Q4. Since H-Proxy directly measures a behavior optimized by the proposed reward, could the authors provide additional evaluation using independent reliability measures, including the distributions of correct, incorrect, and uncertain reflection stances and more comprehensive risk–coverage analyses?

Q5. Can the authors provide evidence that the framework generalizes to knowledge-intensive or open-domain factual tasks where answer verification is less straightforward? Such results would help establish whether REIN improves general reliability rather than primarily learning task-specific behavior on verifiable reasoning benchmarks.

7 comments
AuthorsOpenReview ↗

Response to Reviewer b1VE (Part 1)

Thanks for your appreciation and suggestions. Here are our point-by-point responses:

the limited methodological novelty of the proposed framework

REIN addresses hallucination risks arising from reasoning errors and insufficient knowledge by training models to identify erroneous reasoning and decide whether to answer based on their own problem-solving capabilities. We focus on two situations: when a model produces incorrect reasoning but still judges it to be correct, and when a model lacks the knowledge required to solve a problem but still provides an unsupported answer. To this end, REIN combines reflection evaluation with answer decisions, enabling the model to identify reasoning errors, avoid false endorsement, abstain when a problem is beyond its problem-solving capabilities, while retaining the ability to answer solvable problems.

Experiments show that REIN reduces the endorsement of incorrect initial reasoning and improves selective accuracy while maintaining high answer coverage. The entire process is completed within a single generation, without additional reasoning verification or multi-round iterations.

What is the fundamental methodological distinction between REIN and prior reinforcement learning approaches for self-verification and abstention, such as S²R, TruthRL, and Abstain-R1? Beyond combining existing objectives, which specific design choice is responsible for the distinctive improvement?

We thank the reviewer for this comment. We clarify the methodological differences between REIN and existing approaches.

Prior work has explored RL-based self-verification and reasoning correction (e.g., S²R (Ma et al., 2025)), as well as RL-based truthful answering and abstention (e.g., TruthRL (Wei et al., 2026) and Abstain-R1 (Zhai et al., 2026)), REIN extends reliability modeling to two related but distinct levels: the reliability assessment of initial reasoning and the assessment of whether the current question is answerable.

Specifically, Reflection Alignment supervises the model to judge the correctness of the initial draft before reflection (y0y_0), enabling the model to correct errors in its own reasoning. Boundary-aware Abstention, on the other hand, learns answering or abstention decisions based on whether the current policy can generate a correct answer to a given question.

A key design of REIN is to separate boundary estimation from the abstention policy. We use an independent IDK-free Probe Group to estimate the solvability of questions under the current policy, and separate it from the IDK-enabled Policy Group that participates in policy gradient updates. This prevents the policy's own abstention behavior from directly affecting boundary estimation.

Boundary EstimatorCoverageEffective Accuracy
Independent IDK-free Probe93.8%85.0%
Same-group IDK-enabled76.4%70.2%

Coverage increases from 76.4% to 93.8%, while Effective Accuracy increases from 70.2% to 85.0%. This indicates that independent boundary estimation can effectively reduce unnecessary abstentions and improve overall effective accuracy while maintaining high selective accuracy.

To further validate this distinction, we manually examined 112 questions for which all 16 sampled completions were incorrect. Among these questions, 102 (91.07%) were attributed to insufficient knowledge.

Separately, among 747 incorrect initial drafts, 82 were corrected in the final answers (11.0%). Further error-type annotation showed that 79 of these 82 corrected cases (96.34%) involved logical reasoning errors. Together with the Reflection intervention results, these findings support the respective roles of Boundary-aware Abstention in handling knowledge-related failures and Reflection in correcting reasoning errors.


References

  1. Ma, Ruotian, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. "S²R: Teaching LLMs to Self-Verify and Self-Correct via Reinforcement Learning." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22632–22654. Association for Computational Linguistics. 2025.

  2. Wei, Zhepei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Jingxiang Chen, Mohammad Kachuee, Teja Gollapudi, Yiwei Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen-Tau Yih, and Xin Luna Dong. "TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning." Proceedings of the 43rd International Conference on Machine Learning, 306, pp. 134156–134175. PMLR. 2026.

  3. Zhai, Haotian, Jingcheng Liang, and Dongyeop Kang. "Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL." Findings of the Association for Computational Linguistics: ACL 2026, pp. 19674–19695. Association for Computational Linguistics. 2026.

AuthorsOpenReview ↗

Response to Reviewer b1VE (Part 2)

Could the authors compare the proposed boundary-aware reward against simpler alternatives, such as confidence-based abstention rewards or reward functions based on estimated answer correctness? This would clarify whether the group-level boundary estimation is essential.

We thank the reviewer for raising this question. Confidence (e.g., P(True)) evaluates the model's current answer, whereas Knowledge Boundary estimates whether the current policy can solve a question through multiple sampled trajectories. A low-confidence answer does not necessarily mean that the question is beyond the model's capabilities, just as a high-confidence answer does not guarantee that the model can solve it. Unlike Self-Consistency, which measures agreement among sampled answers, Knowledge Boundary checks whether at least one sampled trajectory produces a verified correct answer.

We also compare seven selective prediction signals in Appendix C. On GSM8K, Reflection Stance achieves an AURC of 0.038, compared with 0.051 for P(True) and 0.040 for Self-Consistency. On MATH, its AURC is 0.257, comparable to Self-Consistency at 0.255. Combining Reflection Stance and Self-Consistency further reduces AURC to 0.031 and 0.246, respectively.

Selective ScoreGSM8K AURC ↓MATH AURC ↓
P(True)0.0510.272
Self-Consistency @160.0400.255
Reflection Stance0.0380.257
Reflection + Self-Consistency0.0310.246

Finally, we compare independent boundary estimation with same-group estimation. Separating the IDK-free Probe Group from the IDK-enabled Policy Group increases Coverage from 76.4% to 93.8% and Effective Accuracy from 70.2% to 85.0%. This shows that independent boundary estimation reduces unnecessary abstentions while improving overall effective accuracy.

Does the reflection-veracity reward provide significant advantages over supervision based on final-answer correctness or more conventional self-evaluation objectives? A controlled comparison using different reflection supervision targets would strengthen the methodological contribution.

We thank the reviewer for raising this question. We choose the initial draft (y0y_0), rather than the final answer (y1y_1), as the supervision target because Reflection directly evaluates the initial reasoning before reflection, while the correctness of the final answer is handled by an independent Answer-Correctness Reward. If (y1y_1) were used to supervise Reflection, it could confuse Reflection's judgment of initial errors with the outcomes of subsequent corrections. For example, an initial draft may be incorrect, but Reflection may correctly identify the error and lead the model to produce a correct final answer. In this case, Reflection's judgment itself is still correct.

The Draft-to-Final Transition analysis in Appendix G supports this design. Among 3,220 generations, 747 initial drafts were incorrect, of which 82 were eventually corrected (Repair Rate: 11.0%). Meanwhile, the proportion of initially correct drafts that were corrupted was only 0.9%. This shows that the correctness of initial drafts and final answers is not equivalent.

Furthermore, after removing the Reflection-Veracity Reward, Final H-Proxy increases from 11.0% to 20.0%, while Coverage remains nearly unchanged (93.8% vs. 93.5%). The Reflection Intervention experiments also show that removing Reflection reduces the Repair Rate from 11.0% to 0.3%; transplanting Reflection from other questions reduces it to 4.1%, while paraphrasing that preserves the original semantics still achieves a Repair Rate of 10.4%.

AuthorsOpenReview ↗

Response to Reviewer b1VE (Part 3)

Since H-Proxy directly measures a behavior optimized by the proposed reward, could the authors provide additional evaluation using independent reliability measures, including the distributions of correct, incorrect, and uncertain reflection stances and more comprehensive risk–coverage analyses?

We thank the reviewer for pointing out the need to evaluate REIN's reliability beyond H-Proxy. We provide three additional analyses.

(1) Reflection Stance Distribution. Across 3,220 completions, 99.4% have parseable reflection stances: 76.4% Correct, 13.2% Wrong, and 10.4% Uncertain. Conditioned on draft correctness, 97.3% of correct drafts receive a Correct stance, while 92.2% of incorrect drafts are classified as Wrong or Uncertain. This shows that Reflection distinguishes correct and incorrect initial drafts, rather than simply favoring positive endorsements (Appendix G.2, Table 21).

(2) MCC Analysis. We further use the Matthews Correlation Coefficient (MCC) to measure the overall agreement between reflection judgments and answer correctness. MCC accounts for both false positives and false negatives. In Figure 4, REIN achieves 94.3% agreement and an MCC of 0.816. Across models and benchmarks, the macro-averaged agreement and MCC are 92.7% and 0.792, respectively (Figure 4; Appendix J, Figure 6).

(3) Risk–Coverage Analysis. We compare seven selective prediction scores on the same fixed set of single-pass REIN answers across GSM8K and MATH, using final-answer error as the risk measure. On GSM8K, Reflection Stance achieves an AURC of 0.038, outperforming P(True) (0.051) and 16-sample Self-Consistency (0.040). On MATH, its AURC is 0.257, comparable to Self-Consistency (0.255). At 90% and 80% coverage, Reflection Stance achieves selective accuracies of 92.2% and 95.3% on GSM8K, and 58.5% and 62.0% on MATH, respectively. Combining Reflection Stance with Self-Consistency further reduces AURC to 0.031 and 0.246, respectively (Appendix C, Table 10).

These results evaluate REIN from three perspectives: reflection stance quality, overall judgment agreement, and risk–coverage performance.

Can the authors provide evidence that the framework generalizes to knowledge-intensive or open-domain factual tasks where answer verification is less straightforward? Such results would help establish whether REIN improves general reliability rather than primarily learning task-specific behavior on verifiable reasoning benchmarks.

We thank the reviewer for highlighting the need for broader task coverage in our evaluation. To further assess its generalization across task types, we conducted additional experiments on AIME24 and TriviaQA.

On AIME24, REIN improves Sel.Acc. across all four models, from 6.7%–10.0% for the Base models to 16.7%–31.8%, while consistently reducing Final H-Proxy. On TriviaQA, Sel.Acc. also improves across all four models, from 46.0%–58.0% to 66.0%–69.0%, while Final H-Proxy decreases from 31.9%–42.0% to 10.3%–22.5%.

BackboneAIME24 Sel.Acc. (Base → REIN)AIME24 Final H-Proxy (Base → REIN)TriviaQA Sel.Acc. (Base → REIN)TriviaQA Final H-Proxy (Base → REIN)
Qwen2.5-7B10.0 → 31.833.3 → 13.352.0 → 68.534.1 → 16.8
LLaMA-3.1-8B6.7 → 20.039.3 → 18.858.0 → 69.031.9 → 10.3
Mistral-7B-v0.36.7 → 16.742.9 → 20.046.0 → 67.042.0 → 15.7
DeepSeek-R1-Distill-Llama-8B6.7 → 21.735.7 → 16.750.0 → 66.036.0 → 22.5

These results indicate that REIN’s reliability alignment behavior generalizes beyond the original mathematical and commonsense reasoning tasks to more challenging reasoning problems and knowledge-intensive question-answering scenarios.

Reviewer_nUGPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes REIN, a reinforcement learning framework designed to improve the reliability of large reasoning models by jointly addressing reasoning hallucinations and knowledge hallucinations.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The method is well motivated and illustrated.

  3. The proposed method achieves promising results.

Critical weaknesses
  1. The absence of correct answers among sampled trajectories does not necessarily indicate a lack of knowledge, as it may result from reasoning failures or sampling uncertainty.

  2. It remains unclear whether the proposed reflection mechanism genuinely improves reasoning or primarily enhances confidence assessment and answer selection.

  3. The additional rollouts required for knowledge-boundary estimation introduce substantial computational costs.

  4. The proposed metric measures false endorsement of incorrect drafts rather than hallucinations in final answers, potentially limiting its effectiveness in evaluating hallucination reduction.

Questions for authors
  1. To what extent does REIN actually correct incorrect reasoning steps, rather than primarily improving the model's ability to assess or abstain from unreliable answers?

  2. Since H-Proxy measures false endorsement of incorrect draft answers, how well does this metric correlate with the hallucination rate of the final generated responses?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nUGP

We thank the reviewer for the constructive comments and hope the following clarifications address the concerns. If there are any remaining questions, we would be happy to provide more details.

W1 Knowledge Boundary Validation

The absence of correct answers among sampled trajectories does not necessarily indicate a lack of knowledge, as it may result from reasoning failures or sampling uncertainty.

To validate the correspondence between the sampled boundary and knowledge hallucinations, we examined 112 questions. Among the 112 examined questions with no correct answer in 16 sampled completions, 102 (91.07%) were annotated as failures attributable to insufficient knowledge.

W2&Q1 Reflection and Reasoning Correction

To what extent does REIN actually correct incorrect reasoning steps, rather than primarily improving the model's ability to assess or abstain from unreliable answers?

Thank you for raising this question. We further analyzed the effectiveness of Reflection in correcting erroneous reasoning. Among 747 incorrect initial drafts, 82 were eventually corrected, yielding a repair rate of 11.0%. Further error-type annotation showed that 79 of these cases (96.34%) involved logical reasoning errors. In addition, the Reflection intervention experiments in Table 3 further demonstrate the contribution of Reflection content to final-answer correction. Removing Reflection reduced the repair rate from 11.0% to 0.3%. Replacing the original Reflection with one from another question reduced the repair rate to 4.1%, whereas paraphrasing the Reflection while preserving its original meaning retained a repair rate of 10.4%. These results show that Reflection helps the model identify and correct logical reasoning errors, and that using question-specific Reflection is important for successful correction.

W3 Computational Cost

The additional rollouts required for knowledge-boundary estimation introduce substantial computational costs.

Thank you for pointing out the additional computational cost introduced by the extra rollouts. As reported in Table 8 of Appendix C, training Qwen2.5-7B with REIN requires 37.9 GPU-hours, compared with 21.4 GPU-hours for vanilla GRPO. This additional cost occurs only during training. The boundary probe is not used at inference time, and REIN generates only a single structured completion without additional sampling or multi-round reflection. According to Appendix I.1.4, Table 24, REIN's inference latency is 1.21× that of greedy decoding, close to 1.19× for Reflection-only.

w4&Q2 Draft-Level vs. Final-Answer Reliability

Since H-Proxy measures false endorsement of incorrect draft answers, how well does this metric correlate with the hallucination rate of the final generated responses?

We thank the reviewer for highlighting the distinction between false endorsement of initial drafts and hallucination risks in final answers. We distinguish two metrics in the revised manuscript: Draft False-Endorsement (DFE) measures the proportion of incorrect initial drafts y0y_0 positively endorsed by Reflection, whereas Final H-Proxy measures the proportion of incorrect final answers y1y_1accompanied by positive Reflection judgments. These metrics differ because incorrect drafts may be corrected or rejected after Reflection. Across four benchmarks on Qwen2.5-7B, DFE is 7.76% (58/747), while Final H-Proxy is 11.53% (55/477). We further report the Matthews Correlation Coefficient (MCC) to measure the overall agreement between Reflection judgments and final-answer correctness. Unlike FH-Proxy, which focuses on false endorsements (False Positives), MCC accounts for both false endorsements and false rejections (False Negatives). On the same evaluation set, the final-answer error rate is 16.13%, and MCC is 0.816. The product of Final H-Proxy and the final-answer error rate is 1.86%, representing the proportion of all substantive responses that are both incorrect and positively endorsed. The corresponding MCC results are reported in Figure 4 and Appendix C.3 (definition and formulation), while Appendix J, Figure 6 jointly visualizes false endorsement and false rejection. REIN achieves a macro-averaged agreement of 92.7% and an MCC of 0.792 in Figure 6.

Reviewer_taWGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes REIN, a framework that jointly trains reflection and abstention to improve reasoning reliability. The model generates a structured completion, <think> → <reflection> → <answer>. A reflection reward supervises the reliability judgment of the draft answer, while multiple forced-answer samples estimate finite-budget answerability and provide supervision for abstention. Training combines SFT initialization with two-phase GRPO, while inference requires a single completion. The authors evaluate REIN on mathematical and commonsense reasoning tasks.

Critical strengths
  • Considering both reasoning hallucinations and knowledge hallucinations is valuable. These failure modes call for different responses: potentially recoverable reasoning errors motivate detection and correction, whereas answers unsupported by the necessary knowledge motivate abstention. By incorporating reflection and abstention into a unified training framework, the paper addresses two complementary requirements for reliable reasoning.
Critical weaknesses
  • The method section needs a clearer progression from the overall framework to individual components, together with better organization within each component.

Section 3 introduces notation, judgment rules, and reward functions before establishing a clear conceptual picture. Section 3.1 mainly defines the output structure and associated notation, rather than explaining what Sections 3.2–3.4 each accomplish, why the components are needed, and how they form a coherent training pipeline. Figure 1 could provide this overview, but it is neither referenced nor explained in the main text.

I suggest that Section 3.1 first walk readers through Figure 1 in intuitive terms: how reflection alignment assesses an individual reasoning trajectory, how boundary-aware abstention uses multiple samples to provide prompt-level supervision, and how these signals are combined through the joint reward and GRPO optimization. The formal definitions would be easier to follow once these relationships are established.

Sections 3.2–3.4 would also benefit from clearer internal organization. Motivation, variable definitions, reward rules, and training arrangements are interleaved without sufficient signposting. Short subheadings could make the purpose and logical structure of each part clearer.

In addition, Figures 1 and 2 are not explicitly referenced or discussed in the main text outside their captions. Figure 1 presents the core pipeline, while Figure 2 presents training dynamics; without accompanying discussion, readers receive little guidance on how these figures support the method description or experimental analysis.

There are also grammatical and sentence-completeness issues. For example, the sentence beginning “Unlike outcome-only preference RL” in lines 131–133 lacks a complete main clause. Combined with the dense technical definitions, such issues further increase the reading burden. The manuscript would benefit from a systematic revision of sentence structure, paragraph transitions, and figure references.

Questions for authors
  • How is the correspondence between the sampled boundary and knowledge hallucinations validated? Failure across all sampled completions can also arise from persistent reasoning difficulties rather than missing knowledge. Could the authors additionally provide an analysis that independently distinguishes knowledge failures from reasoning failures and shows which errors are primarily addressed by each component?

  • Section 4 states that GRPO uses only GSM8K and discusses performance on the other benchmarks as cross-task transfer. However, Appendix I.1.2 indicates that SFT initialization uses training examples from all four benchmarks. Does transfer here refer specifically to transfer beyond the GRPO training domain? Could the authors clarify the data used at each training stage and establish whether the improvements persist without target-task SFT data?

6 comments
AuthorsOpenReview ↗

Response to Reviewer taWG

We thank the reviewer for the constructive comments and hope the following clarifications address the concerns. If there are any remaining questions, we would be happy to provide more details.

W1: Organization of the Method Section.

The method section needs a clearer progression from the overall framework to individual components, together with better organization within each component.

We thank the reviewer for the helpful suggestion to improve the organization of the Method section 3.In the revised PDF, we have reorganized the presentation of our method to improve its clarity and readability. First, in Section 3.1, we introduce the overall REIN framework following Figure 1 and provide a unified explanation of the notations and definitions used throughout the method. We then present the motivations and intuitions behind Reflection Alignment and Boundary-aware Abstention in Sections 3.2 and 3.3, respectively, followed by detailed descriptions of their specific mechanisms. Finally, Section 3.4 presents the reward design and GRPO optimization process. In addition, we have added explicit references to and explanations of Figures 1 and 2 in the main text, clarifying the overall training framework and the evolution of different metrics during training, respectively. We have also systematically reviewed and revised the sentence structures, paragraph transitions, and technical descriptions throughout the manuscript to improve readability and reduce the cognitive burden on readers. These revisions aim to establish a clearer logical connection between the overall motivation, component designs, and optimization procedure of REIN.

Q1: Disentangling Knowledge and Reasoning Failures

How is the correspondence between the sampled boundary and knowledge hallucinations validated? Failure across all sampled completions can also arise from persistent reasoning difficulties rather than missing knowledge. Could the authors additionally provide an analysis that independently distinguishes knowledge failures from reasoning failures and shows which errors are primarily addressed by each component?

We thank the reviewer for raising this question. To validate the correspondence between the sampled boundary and knowledge hallucinations, we manually analysis Among the 112 questions for which all 16 sampled completions were incorrect, 102 (91.07%) were due to insufficient knowledge. In addition, among 747 incorrect initial drafts, 82 were corrected in the final answers, yielding a repair rate of 11.0%. We further analyzed the error types of these 82 cases and found that 79 (96.34%) involved logical reasoning errors. These results show that Boundary-aware Abstention primarily addresses knowledge-related errors, while Reflection primarily corrects logical reasoning errors.

Q2: Training Data and Cross-Task Transfer

We thank the reviewer for pointing out the ambiguity in the description of our training data. The two training stages use different data and serve different objectives. Cold-start SFT uses training examples from GSM8K, MATH, StrategyQA, and ARC-C solely to initialize the structured output format. In the subsequent GRPO stage, the policy is optimized using only the GSM8K training set, with reward signals for answer correctness, reflection veracity, and boundary-aware abstention. Therefore, the cross-task transfer discussed in our paper specifically refers to the transfer of reliability-alignment behaviors learned during the GSM8K-only GRPO stage to other tasks.

To further validate this, we conducted additional experiments on AIME24 and TriviaQA, neither of which was used in SFT or GRPO. As shown in the table, all four models achieve improved Sel.Acc. and reduced FH-Proxy on both benchmarks. These results show that the reliability-alignment behaviors learned by REIN generalize to benchmarks not used during training.

The values in parentheses after Eff.Acc. indicate coverage (%).

BackboneMethodAIME24 Sel.Acc. ↑AIME24 FH-Proxy ↓AIME24 Eff.Acc. ↑TriviaQA Sel.Acc. ↑TriviaQA FH-Proxy ↓TriviaQA Eff.Acc. ↑
Qwen2.5-7BBase10.033.310.052.034.152.0
REIN31.813.323.3 (73.3)68.516.853.43 (78.0)
LLaMA-3.1-8BBase6.739.36.758.031.958.0
REIN20.018.813.3 (66.7)69.010.357.9 (84.0)
Mistral-7B-v0.3Base6.742.96.746.042.046.0
REIN16.720.010.0 (60.0)67.015.756.3 (84.0)
DeepSeek-R1-Distill-8BBase6.735.76.750.036.050.0
REIN21.716.716.7 (76.7)66.022.547.5 (72.0)

Dear Reviewer taWG,

Thank you again for your thoughtful and constructive feedback, which we have carefully addressed in our author response. As the discussion deadline is approaching, we would greatly appreciate any further feedback you may have. If our responses have resolved your main concerns, we would also be grateful if you could update your assessment accordingly.

Sincerely,

The Authors

Reviewer_taWGOpenReview ↗

W1 not well addressed

Dear authors,

Thank you for your rebuttal and revisions. However, my concerns under W1 have not been fully addressed. W1 also highlighted grammatical errors and incomplete sentences. Although you state that you have systematically reviewed and revised the sentence structures, I still find numerous such issues, even more than in the original version. Examples include:

  • Line 131: “Unlike outcome-only preference RL, which is silent about the reasoning trajectory and process-supervised reward models, which require fine-grained step labels and still cannot decide when to abstain.” This remains an incomplete sentence. I explicitly flagged it in my review, but it has not been revised.

  • Line 170: The entire line consists of a single period.

  • Line 193: A sentence-final period is missing.

  • Line 211: A period is missing after “Failure definition”.

  • Line 237: The line should begin with “where”.

  • Line 286: A period is missing after “GRPO Reward Component”.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.