acceptodds
Under review as a conference paper at ICLR 2027

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

Abstract

Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the Alignment Paradox. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors () by more than an order of magnitude (10 to 35). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Confident Hallucinations, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nfs9OpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how post-training affects confidence hallucinations. Across different model families and alignment methods, the authors find that post-training can amplify models' confidence in factually incorrect answers. They then provide a theoretical explanation for this phenomenon and propose Bounded Dynamic Margin DPO to mitigate this alignment paradox.

Critical strengths
  • The problem studied in this paper is of practical significance.
  • The evaluation is relatively comprehensive. The authors conduct investigative experiments across five base and instruction-tuned models and validate the proposed methods on two models in two alignment states.
Critical weaknesses

W1. Inconsistency between the description and implementation of the proposed BDM-DPO

Eq. 6 and Algorithm 1 specify the following loss function:

=-\log\sigma\left(\min(\beta\Delta,M_{\max})\right)$$ where $\Delta$ is the unscaled preference log-ratio difference. However the provided implementation bounds the unscaled margin and multiplies by $\beta$ afterward [[Link1]](https://archive.ph/9upQq) [[Link2]](https://anonymous.4open.science/r/HCE-1B12/src/bdm/loss.py#L115) [[Link3]](https://anonymous.4open.science/r/HCE-1B12/src/bdm/loss.py#L111) [[Link4]](https://anonymous.4open.science/r/HCE-1B12/src/bdm/loss.py#L11-L14), resulting in the following loss function: $$L_{\mathrm{code}} =-\log\sigma\left(\beta\min(\Delta,M_{\max})\right)$$ With $\beta=0.1$, $M_0=3.8$, and $H_{\mathrm{ref}}=0$, the transition in the implemented code occurs around raw margin 3.8, whereas the hard-clamping threshold implied by Eq. 6 in the paper is 38; the corresponding loss floors are approximately 0.521 for the code implementation and 0.022 for the paper’s objective. This is a substantial change in optimization behavior. ### **W2. Limited novelty and insufficient comparisons with prior work** The paper’s central framing "preference alignment can exacerbate overconfidence and that this motivates a training-time remedy" substantially overlaps with existing research. However, the paper does not adequately position and discuss its contributions in relation to existing works: * Taming Overconfidence in LLMs: Reward Calibration in RLHF. (ICLR 2025) * Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach. (ICML 2025) * Trust Me, I’m Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer. (EMNLP 2025) * Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. (EMNLP 2023) For BDM-DPO, comparisons with IPO[1], SLiC-HF[2]’s hinge-based ranking loss, and conservative DPO with label smoothing are particularly relevant: these approaches provide alternatives to indefinitely increasing preference margins, although their objectives are not equivalent to BDM-DPO. > [1] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024 > > [2] SLiC-HF: Sequence Likelihood Calibration with Human Feedback. 2023 ### **W3. Other inconsistencies in the paper** * Inconsistencies between Table 1 and Table 9: | | Table 1 | Table 9 | |---|---|---| | Mistral-7B Gate B, Base → Instruct | 97 → 1,366 | 63 → 1,630 | | Mistral-7B accuracy, Base → Instruct | 52.3% → 41.6% | ~35.8% → 47.6% | | Qwen3-8B ratio | 35.4× | 924/92\$\approx\$10.0x | | Qwen3-14B ratio | 19.8× | 1130/113\$=\$10.0x | * Mathematically inconsistent results in Table 16: For the self-consistency on Qwen-8B, the authors report an FPR@95TPR of 0. However, for the same scores, labels, and evaluation set, the monotonicity of the ROC curve implies that $\mathrm{AUROC} \geq 0.95(1-\mathrm{FPR@95TPR})$. Therefore, an FPR@95TPR of 0 would require an AUROC of at least 0.95, whereas the paper reports an AUROC of 0.516
Questions for authors
  • Can you clarify the identified inconsistencies meantioned in the weakness section?

  • Which loss function is actually used to obtain the experimental results presented in the paper?

  • In Appendix A.4, the authors report differences in statistics between a pilot subset (n = 2,000) and the full datasets. I'm confused about the purpose of this pilot subset and the motivation for reporting these differences.

  • For Table 2, could the authors rerun C3 using exactly the same instructions and few-shot examples as C1? Appendix A.5 shows that their prompts differ, confounding parameter updates with prompt effects. Moreover, Table 2 attributes 49–85% of the increase to the template. How does this support the claim that “parameter updates account for the vast majority”?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nfs9 (1/2)

Thank you for your careful review, including the check of our code. Changes are marked in blue.

W1 / Q2 (which loss was used). You are right, and we apologize for the mismatch. The BDM results were produced by the implemented loss, L=−log⁡σ(β⋅min⁡(Δ,M))\mathcal{L} = -\log\sigma(\beta \cdot \min(\Delta, M)) with M=M0exp⁡(−γH)M = M_0 \exp(-\gamma H). Here HH is the reference model's mean token entropy over the prompt, normalized by ln⁡∣V∣\ln|V|, and β=0.1\beta = 0.1, M0=3.0M_0 = 3.0, γ=2.0\gamma = 2.0 outside the sweeps of Appendix F. Section 5.2 (Eq. 3) and Algorithm 1 (Appendix F.3) now state this loss and these values, and the defaults in the repository's loss.py match them. As you computed, the two forms optimize differently, and Table 4 measures this. It also trains the form with β\beta inside the minimum, −log⁡σ(min⁡(βΔ,M))-\log\sigma(\min(\beta\Delta, M)), with M0=3.8M_0 = 3.8, γ=1.0\gamma = 1.0 and first-token entropy, so this run also differs in these settings. After ten epochs it has a held-out margin of 21.9 and 2,538 errors at fixed coverage, close to DPO with β=0.2\beta = 0.2 (22.9 and 2,141). The implemented form ends at 3.4 and 1,661, against 1,615 initially. The repository now contains the scripts behind the revision's results with a map from each table and figure to its script.

W2 (positioning; IPO, SLiC-HF, cDPO). Thank you for these references and baselines. We agree that this framing overlaps with prior work. The revision credits the degradation to that prior work and asks which answers receive the added confidence (general response). Section 2 now discusses the four papers. Leng et al. (2025) correct the reward model to reduce verbalized overconfidence in PPO, and Xiao et al. (2025) restore multiple-choice confidence reliability after preference alignment with an ECE-based regularizer. Tian et al. (2023) show that a verbalized confidence can be more reliable after RLHF. A confidence threshold at decoding reads the token probabilities of a free-form answer instead (their geometric mean pˉ\bar{p}), which is what we measure and what our mitigation acts on (Section 5.5). Simhi et al. (2025) show that models can be certain and wrong on questions they can answer. In our tracing, answers that were correct before are 7–14% of the new confident errors when only the weights change, and 73–89% are wrong answers the base model already gave (Section 3.4). We trained IPO, cDPO (ϵ=0.1\epsilon = 0.1) and an SLiC-HF-style hinge loss for ten epochs with the same data and evaluation. The initial checkpoint has 1,615 errors among its 7,308 most confident answers and 47.8% accuracy.

ObjectiveHeld-out margin, epoch 10Errors among top 7,308, epoch 10Accuracy, epoch 10
IPO0.71,64147.7%
BDM3.41,66147.6%
cDPO (ϵ=0.1\epsilon = 0.1)4.31,73947.3%
Hinge on the DPO margin (SLiC-HF style, without its regularizer)8.31,78047.3%
DPO, β=0.2\beta = 0.222.92,14145.0%
BDM, β\beta inside the min21.92,53844.4%
DPO, β=0.1\beta = 0.141.12,62342.9%

IPO, cDPO, the hinge and BDM stay close to the initial checkpoint, and across 42 checkpoints the errors track the held-out margin (Spearman ρ=0.94\rho = 0.94, an association); none adds more than a weak preference for correct answers (Table 28). BDM is now reported as one bounded objective among these, and the revision's mitigation acts on which answers gain confidence instead (Section 5.5). As an RL stage from four initial checkpoints (two seeds each), it leaves 36–61% fewer confident errors and 33–57% fewer errors at fixed coverage than standard GRPO on the same questions. We also applied SLiC-HF's hinge to our labeled groups in place of the Brier reward (RL stage from the Tulu 3 DPO checkpoint, two seeds per setting; Appendix K.3). Without its NLL term, 295–347 of the 4,738 most confident answers are wrong, against 224 and 213 for our loss and 376 and 417 for standard GRPO. With the NLL term, the ranking falls below that of the initial checkpoint (1,049 and 2,104 against 467).

AuthorsOpenReview ↗

Response to Reviewer nfs9 (2/2)

W3 (a) (Table 1 vs. Table 9). Thank you for catching this. Table 8 of the revision now comes from the Table 1 run and adds up to it, so all four rows of your comparison agree with Table 1.

Table 1Revised Table 8
Mistral-7B Gate B, base → instruct97 → 1,36617 + 80 → 516 + 850 (TriviaQA + PopQA)
Mistral-7B accuracy, base → instruct52.3% → 41.6%52.3% → 41.6% (instruct 62.8% and 25.0%)
Qwen3-8B ratio35.4×50 → 1,768 (35.4×)
Qwen3-14B ratio19.8×104 → 2,061 (19.8×)

W3 (b) (Table 16). You are right, and thank you for the careful check. In the revised Table 16, self-consistency has FPR@95TPR = 1.000 and an AUPR near the share of errors (0.903 and 0.909, against 89% and 91%), i.e., chance level. Every row satisfies AUROC≥0.95(1−FPR@95TPR)\mathrm{AUROC} \geq 0.95(1-\mathrm{FPR@95TPR}). The table now states the sample of every row (Appendix D).

Q3 (the pilot subset). Thank you for asking. The 2,000-question subset (1,000 per benchmark) was used only for the 13-configuration grid (Table 21), because every configuration needs its own training and evaluation; all other results use all 22,267 questions. We report the two rates to show that the subset is representative for this grid. With the zero-shot prompt of Appendix F, the initial model makes confident errors on 7.00% of the subset and 7.25% of all questions. The subset is now described where the grid is introduced (Appendices A.3 and F.4).

Q4 (the C3 prompt; the share of the weights). Thank you for both questions, which led us to rewrite Section 3.3. Following your suggestion, C3 now uses exactly the C1 prompt. Both conditions render the same instruction, the same eight few-shot examples and the same four negative examples with the same function. The prompt string of each of the 22,267 questions is therefore identical for the base and the instruct model (Appendix A.4 shows it), and both are counted with the Table 1 pipeline. The prompt is therefore no longer confounded with the weights in the C1 → C3 comparison. With only the weights changed, Gate B errors rise 4.9–17.5× in every family (Mistral-7B 97 → 1,134; Table 2).

You are also right that the shares do not support "the vast majority". Along C1 → C3 → C2, the weights account for 25–83% of the base-to-chat increase per family (49% overall) and the template for the remaining 17–75% (51% overall), and the split depends on this order. Section 3.3 now reports these shares and draws only the fixed-prompt conclusion, that the weights alone multiply confident errors in every family and the chat template adds a further increase. The source tracing of Section 3.4 uses the same fixed prompt, so its finding that 73–89% of the new confident errors are the base model's own wrong answers does not depend on the template.

We hope the corrections and clarifications above settle the points you raised, and we would be glad to answer any further questions.

Thank you for the responses. While the substantial revisions and additional experiments address some technical issues, my overall assessment remains unchanged.

Specifically, the authors acknowledge that the core loss function was incorrectly defined in the original submission, invalidating a substantial portion of the original theoretical analysis, including Theorem 1 (Unbounded Margin Divergence on Factual Errors) and several corollaries. Consequently, the original theoretical explanation has been substantially narrowed, and several conclusions previously used to motivate BDM are no longer maintained. Meanwhile, the main mitigation contribution now shifts toward the newly introduced Brier-reward method in Section 5.5, with a different training objective and theoretical proposition. Given the extent of these revisions and the changes to the core narrative and central claims (as can be seen in the revised manuscript, where nearly the entire paper has been rewritten and highlighted in blue), I believe this paper is more suitable for major revision and resubmission. Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Clarification on the Revision

Thank you for your quick reply despite our late responses, and for noting that the revisions and additional experiments address some of the technical issues. We would like to clarify two points.

First, the error you found was in how Section 5.2 stated the BDM loss (Eq. 6). All reported BDM experiments were run with the implemented loss, which the revision now states (Eq. 3). Theorem 1 and its proof concerned the standard DPO loss and did not use Eq. 6, so this correction did not affect them. We restated Theorem 1 per preference pair (Remark 1) because gradients of different pairs can conflict, as Reviewer TATw noted. We then measured the margins of the full model. These measurements still motivate a bound on the margin. After ten epochs, the held-out margin is 41.1 under standard DPO and 3.4 under BDM (Table 4). Across seven objectives, errors at fixed coverage rise with the margin (Spearman ρ=0.94\rho = 0.94 over 42 checkpoints).

Second, the largest additions come from Reviewer TATw's question on the post-training data. The data experiment it led to (Section 5.4) shows that the data decide which answers gain confidence, and Section 5.5 builds on this result. Most of the other blue text marks corrections and narrower statements that the reviews asked for.

The main result and the method of the submission are kept. Table 1 is unchanged, and BDM remains in Sections 5.2–5.3 with the same result as in the submission, 35.3% fewer confident errors than standard DPO at epoch 8. Following your suggestion, Table 4 compares BDM with IPO, cDPO and an SLiC-HF-style hinge. These objectives also stop rewarding a larger margin, and all of them keep confident errors near the initial level. The mitigation therefore has two parts. A bound on the margin, with BDM as one form, stops the growth of confident errors. Training on the model's own answers labeled correct or incorrect, with a Brier reward, lowers errors at fixed coverage below the initial level (Section 5.5).

Thank you again for your careful review and for checking our code.

Reviewer_sDtFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reports that instruction-tuned models produce many more factual errors at sequence confidence above 0.95 than their base checkpoints, on PopQA and TriviaQA across five model pairs. It shows that part of the increase survives when the chat template is removed, that the errors are stable under resampling, and that logit-lens margins on confident outputs grow in the last layers. Five inference-time defenses remove under 9 percent of these errors. The paper then proposes BDM-DPO, a DPO variant that clamps the implicit reward margin at an entropy-dependent ceiling, and reports that over ten epochs of DPO on 3,000 UltraFeedback pairs it keeps confident errors and accuracy near the starting checkpoint while standard DPO degrades.

Critical strengths
  1. The paper asks the right question for deployment. Most calibration work reports ECE and stops. This paper counts the errors that cross an abstention threshold, which are the ones that reach the user, and shows that alignment moves errors across that line. The gate framing is simple and I expect it to be reused.

  2. It ties three things together that are usually studied apart: calibration loss under alignment, the failure of post-hoc detectors on aligned models, and the training objective.

Critical weaknesses
  1. The central finding is known, and the paper does not say so. That RLHF and preference tuning degrade calibration is documented in the GPT-4 technical report, in Zhu et al. 2023 on calibration and alignment, in Tian et al. 2023, and in Leng et al. 2024 on reward calibration. None are cited. Theorem 1 restates the observation that motivated IPO, which the paper cites as Azar et al. 2024.

  2. The comparison and the fix are both off target. Base models here are 8 to 11 points more accurate than their instruct versions on the same queries, which the paper does not discuss.

Questions for authors
  1. Please reconcile the Gate B counts for Mistral-7B and Qwen3-8B between Table 1 and Table 2. Which numbers are correct, and what differs between the two runs?

  2. Given Table 21, what does BDM-DPO add over standard DPO with beta of 0.2, or over standard DPO with early stopping, across the ten epochs of Table 5?

  3. For instruct outputs under the chat template, is the sequence confidence computed over the entity tokens only or over the full generation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer sDtF (1/2)

Thank you for your detailed review and for recognizing the deployment relevance of counting the errors that cross an abstention threshold. Your comments on prior work and on the fix led to the main changes of the revision. Changes are marked in blue.

W1 (the finding is known; Theorem 1 restates the motivation of IPO). We agree, and thank you for pointing us to this literature. The revision cites OpenAI (2023), Zhu et al. (2023), Tian et al. (2023), Leng et al. (2025) and Kadavath et al. (2022), credits the degradation to these papers, and restates Theorem 1 as a per-pair remark credited to IPO (Remark 1, Section 5.1). These works measure how reliable confidence is over a population of answers. The revision asks a question they leave open, namely which answers receive the added confidence, and answers it by following the same answer to the same question from the base to the instruct model. With the prompt held fixed, most new confident errors are wrong answers the base model already gave, and few were correct before (Section 3.4, Table 3).

FamilyNew confident errorsBase model's own wrong answerCorrect before
Mistral-7B1,10078%11%
Qwen3-8B85673%14%
LLaMA-3.1-8B16686%8%
Mistral-Nemo-12B43789%7%
Qwen3-14B52973%14%

Under one shared prompt, 79–99% of the new confident errors of every stage of Tulu 3, Tulu 2.5 and Zephyr are the preceding checkpoint's own wrong answers. The preference for correct answers is at most weak. Of a correct and a wrong answer to two questions that start equally confident, the correct one ends more confident with probability 0.60–0.68 when only the weights change and 0.53–0.65 at the public stages, against 0.52–0.55 without change. Which answers gain confidence depends on the data. With the model, budget and objective (RPO) fixed, data that reinforce the model's own wrong answers raise confident errors from 1,613 to 3,379, whereas data that contrast them with correct answers lower them to 43 (Section 5.4).

W2 (base models are more accurate; the comparison and the fix are off target). Thank you for raising the accuracy gap. Section 3.2 now discusses it. Lower accuracy alone would raise confident errors only in proportion to the number of wrong answers. Yet the share of wrong answers that are confident rises from 0.4–1.0% for the base models to 4.8–16.4% for the instruct models. Questions that the chat models newly get wrong account for 19–23% of their confident errors, and the rest are on questions the base model also misses. Under the base prompt the accuracy gap shrinks to 0.2–4.3 points, yet the weights alone multiply confident errors 4.9–17.5× (Q1 in part 2).

We also agree on the fix. Bounding the margin slows the increase but adds little preference for correct answers (Q2 in part 2), so the revision adds a mitigation that acts on which answers gain confidence (Section 5.5). It trains on groups of the model's own answers to verifiable questions, labeled correct or incorrect, with the reference answer added when none is correct, and rewards each answer with a Brier score on its confidence. As an RL stage it compares with standard GRPO on the same questions as follows (two seeds, a / b).

Initial checkpointConfident errors: initial → GRPO → oursErrors at fixed coverage: initial → GRPO → ours
Tulu 3 SFT185 → 534 / 700 → 343 / 316185 → 143 / 125 → 84 / 84
Tulu 3 DPO467 → 723 / 699 → 367 / 379467 → 376 / 417 → 224 / 213
Tulu 3 RLVR616 → 769 / 920 → 415 / 390616 → 525 / 505 → 313 / 286
Mistral-7B-Instruct1,613 → 1,043 / 1,149 → 581 / 4461,613 → 1,372 / 1,349 → 593 / 599

Errors at fixed coverage are the wrong answers among a model's kk most confident answers, with kk the initial checkpoint's number of confident answers, so they compare rankings at the same coverage. AUROC rises from 0.792–0.842 to 0.871–0.879. Added to a generic SFT or preference stage on 3,000 UltraFeedback pairs, the same loss leaves 45–58% fewer errors at fixed coverage than the initial checkpoint, with about as many confident correct answers.

AuthorsOpenReview ↗

Response to Reviewer sDtF (2/2)

Q1 (Table 1 vs. Table 2). Thank you for catching this inconsistency. The Table 1 counts are the correct ones. Table 2 now computes every column with the Table 1 pipeline (the same judge and the same Gate B checks) for all five families, so its base and chat columns equal Table 1 and the two tables agree. C3 uses the instruct weights with exactly the base prompt of C1.

FamilyC1: baseC3: instruct, base promptC2: instruct, chat templateC3 / C1C2 / C1
Mistral-7B971,1341,36611.7×14.1×
Qwen3-8B508751,76817.5×35.4×
LLaMA-3.1-8B381865964.9×15.7×
Mistral-Nemo-12B564765608.5×10.0×
Qwen3-14B1045892,0615.7×19.8×

With the prompt held fixed, the weights alone multiply confident errors 4.9–17.5×.

Q2 (BDM vs. DPO with β=0.2\beta = 0.2 or early stopping over ten epochs). Thank you for suggesting these baselines. We ran both (Section 5.3, Table 4; Tables 31 and 34 in Appendix I). The table gives confident errors (1,615 for the initial checkpoint).

Epoch 135810Accuracy, epoch 10
DPO, β=0.1\beta = 0.11,6302,0472,2312,4092,13742.9%
DPO, β=0.2\beta = 0.21,6201,9052,2052,2862,12645.0%
BDM1,5911,5991,6701,5591,48047.6%

Doubling β\beta delays the increase but does not prevent it. Early stopping on 500 held-out pairs selects epoch 1 (β=0.1\beta = 0.1; 1,630) and epoch 2 (β=0.2\beta = 0.2; 1,599), comparable to BDM. What BDM adds is that it needs no stopping epoch, like IPO, cDPO and a hinge loss (Section 5.3). None of them adds more than a weak preference for correct answers, so the revision keeps BDM as one bounded objective and adds a mitigation that acts on which answers gain confidence (Section 5.5).

Q3 (entity tokens or full generation). Thank you for asking. It is the full generation, end-of-sequence token excluded, with at most 10 new tokens (Section 3.1). Computed over the answer tokens only (the first line, without a leading "Answer:" label and final punctuation), the number of confident errors changes by at most 1% for the instruct models of Mistral-7B, Qwen3-8B, Mistral-Nemo-12B and Qwen3-14B. Of the Mistral-7B-Instruct outputs, 12.9% reach the token limit (at most 1.2% for the other models), and 79 of its 1,366 confident errors come from such outputs. Without these outputs the instruct-to-base ratios are 10.0–35.3× (13.3× instead of 14.1× for Mistral-7B). With the first-token probability instead of the sequence confidence, the increase is 24–52× (Appendix J).

We hope these responses resolve your concerns, and we would be happy to clarify any remaining points.

Reviewer_TATwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into if there is any effect on hallucinations when doing post training on LLMs. They show that instruction tuned models have higher confidence hallucinations compared to their base counterpart. They go on to analyze why this may be happening and come up with BDM-DPO. Which will keep the margin from increasing too much when training.

Critical strengths
  • This paper identifies an interesting paradox, showing findings on how post training can lead to higher confidence hallucinations.
  • They analyzed this phenomenon over different model families and sizes for generalizability.
Critical weaknesses
  • The paper does not dive deep into how post training datasets may play a role in the hallucinations. If the data being used to post train on has false information this could also cause the model to hallucinate.
  • The paper dives into many different aspects but their solution is a bit simple and heuristic.
Questions for authors
  • Can authors dive deeper into how post training datasets can affect the model?
  • Paper showed that when the model is predicting correctly with high confidence the margins increase as well in the later layers. Why this is hallucination specific rather than a general post-training effect?
  • How Theorem 1 proofs the full model increases margins to infinity, when gradients from different samples may conflict?
4 comments
AuthorsOpenReview ↗

Response to Reviewer TATw (1/2)

Thank you for your constructive review and for finding the paradox interesting. Your two main suggestions, to study the role of the post-training data and to go beyond a heuristic fix, shaped the revision. Changes are marked in blue.

W1 / Q1 (the role of the post-training data). Thank you for this suggestion. The resulting experiment is one of the main results of the revision (Section 5.4, Table 5, Appendix H) and shaped the mitigation in Section 5.5. From Mistral-7B-Instruct, with the budget fixed (3,000 examples from the TriviaQA training split, one epoch), the datasets differ in what they reinforce (RPO, i.e., DPO plus an NLL term on the chosen answer). Two of the datasets contain false information, as you suggested. H3 reinforces the model's own wrong answers, and H3′ reinforces wrong answers of another model (LLaMA-3.1-8B base) that differ from the model's own greedy answer and samples. "Correct above wrong" is the probability that a correct answer ends more confident than a wrong one of the same prior confidence (0.52 without change).

DataObjectiveConfident errorsErrors at fixed coverageAccuracyCorrect above wrong
Initial–1,6131,61348.0%–
H1: known questions, correct answerSFT / RPO143 / 207840 / 82951.4 / 51.8%0.73 / 0.74
H2: the model's own wrong answer vs. the correct oneSFT / RPO45 / 43638 / 63152.4 / 52.2%0.75 / 0.76
H3: the model's own wrong answer reinforcedSFT / RPO / DPO2,175 / 3,379 / 3,8591,517 / 1,677 / 2,05046.9 / 46.4 / 45.1%0.59 / 0.57 / 0.56
H3′: another model's wrong answer reinforcedDPO / RPO2,256 / 443,247 / 1,16541.8 / 47.6%0.57 / 0.71

The results support your hypothesis in a specific form. Reinforcing the model's own wrong answers raises its confident errors by a third under SFT and more than doubles them under RPO and DPO, and 1,564 (RPO) and 1,794 (DPO) of them are its own earlier wrong answers, now above 0.95. Reinforcing another model's wrong answers acts differently. It raises few of the model's own wrong answers above 0.95 (13–512, against 728–1,794 for H3). Under DPO it changes the answer form and gives the worst ranking, and under RPO it mainly lowers the confidence level, with a ranking between the initial checkpoint and H2. Contrasting the model's own wrong answers with correct ones gives the best ranking, with 631 errors at fixed coverage against 1,677 when they are reinforced. Its few confident errors partly reflect a lower confidence level, which RPO also gives on H3′, so the ranking columns carry the data effect. False data are one route but not the only one. More epochs of SFT on correct answers also raise confident errors again (H1 has 143 after one epoch and 1,444 after five, against 1,613 initially; Appendix I). In public pipelines, under one prompt, 79–99% of each stage's new confident errors are pre-existing (Section 3.4). Inside a generic preference stage, adding 300 contrast pairs also lowers errors at fixed coverage more than adding 300 pairs on known questions (1,052 against 1,237, and 1,716 for the stage alone; Table 49). Repeating the four RPO runs with the pairs in another order gives the same comparisons (for example, 631 and 656 errors at fixed coverage for H2, against 1,165 for H3′ in both runs; Appendix H).

W2 (the solution is simple and heuristic). We agree that BDM is heuristic, and this comment led us to a principled mitigation. The revision keeps BDM as one bounded objective (Section 5.2) and shows that over ten epochs it behaves like IPO, cDPO and a hinge loss (1,475–1,560 confident errors at epoch 10, against 2,137 for standard DPO and 1,615 initially). These bounded objectives keep confident errors near the initial level but add little preference for correct answers (Section 5.3, Table 4). The new mitigation trains on groups of the model's own answers labeled correct or incorrect, with the reference answer added when none is correct. It rewards each answer with a Brier score on its confidence pˉ\bar{p}, the geometric mean of its token probabilities (Section 5.5). Proposition 1 states what this reward optimizes before group normalization. Unlike a binary reward, it penalizes probability concentrated on a wrong answer. As an RL stage from four initial checkpoints (two seeds each), it ends with 36–61% fewer confident errors and 33–57% fewer errors at fixed coverage than standard GRPO (Table 6). Added to a generic SFT or preference stage, it ends with 45–58% fewer errors at fixed coverage than the initial checkpoint (Section 5.5).

AuthorsOpenReview ↗

Response to Reviewer TATw (2/2)

Q2 (late-layer margins also grow for correct answers; why is this specific to hallucination?). Thank you for this question; answering it led to the central finding of the revision. The effect is indeed not specific to hallucination. Post-training raises the answers a model already gives, correct and wrong alike. It still multiplies confident errors because few wrong answers start above 0.95, so a rise shared by correct and wrong answers moves many wrong answers across the threshold. With the prompt held fixed, 73–89% of the new confident errors are the base model's own wrong answers. Among the answers the instruct model keeps, a correct answer ends more confident than a wrong one of the same prior confidence with probability 0.53–0.63, against 0.53–0.55 when the base model is compared with itself (Section 3.4). Section 4.2 now presents the late-layer margin in this light. Instruct models add a step at the last layer that base models lack (+1.35 against −0.21 for Mistral-7B), to confident errors as well as to confident correct answers. The two differ by ∣d∣≤0.76|d| \le 0.76 at every layer except one layer of the Qwen3-8B base model (1.02). At 85% and 95% of depth and at the last layer, errors mostly have the lower margin (dd from −0.69-0.69 to +0.09+0.09; Appendix C.1).

Q3 (how Theorem 1 implies a diverging margin despite conflicting gradients). You are right that it does not. Theorem 1 concerns a single preference pair, and because gradients from different pairs can conflict, it says nothing about the full model. We have restated it as Remark 1 (Section 5.1), a per-pair statement credited to IPO (Azar et al., 2024), and measure the full model instead (Section 5.3, Table 4; Appendix I). The mean margin on 500 held-out pairs grows from 7.0 after one epoch to 41.1 after ten epochs of standard DPO, while IPO, BDM, cDPO and the hinge stay at 0.7–8.3. Across seven objectives, errors at fixed coverage track the margin (Spearman ρ=0.94\rho = 0.94 over 42 checkpoints). Accuracy falls along with the margin, so this is an association, not a causal estimate.

The data experiment you asked for became a central part of the revision, and we would be glad to discuss any of its results further.

The revision meaningfully strengthens the paper, particularly the data analysis and the new method. The authors also corrected the overclaims and provided clearer explanations. My concerns are mostly addressed, and I think this paper provides interesting insights to this area that further support my positive judgment of weak acceptance.

Thank you for taking the time to read the revision. We are glad to hear that it has strengthened the paper and addressed your concerns. Thank you again for your insightful suggestion to examine how the post-training data affect the model. It led us to the controlled data experiment in Section 5.4, which became one of the main results of the paper. We appreciate your engagement throughout the review process.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thorough reviews. Every one of their comments has helped us improve the quality of the paper. Reviewer TATw's question about the role of the post-training data led to a controlled data experiment, whose result motivated the mitigation that the revision adds. Reviewers sDtF and nfs9 pointed us to the closest prior work and to baselines (IPO, cDPO, SLiC-HF, a larger β\beta and early stopping), and comparing these baselines with BDM showed what a bound on the margin can and cannot change. We apologize for the late response, as the new experiments took most of the discussion period.

All three reviews ask, in different forms, what is new beyond the known observation that post-training makes confidence less reliable, and whether the proposed bound addresses the cause. The revision keeps the structure of the submission and its main result (Table 1). It cites this prior work (OpenAI, 2023; Zhu et al., 2023; Tian et al., 2023; Leng et al., 2025; Xiao et al., 2025; Simhi et al., 2025) and asks a question it leaves open, namely which answers receive the confidence that post-training adds. Changes are in blue, and Sections 3.4, 5.4 and 5.5 are new.

We use three terms below. pˉ\bar{p} is the geometric mean of the answer-token probabilities. A confident error is a wrong answer with pˉ≥0.95\bar{p} \ge 0.95. Errors at fixed coverage are the wrong answers among a model's kk most confident answers, with kk the initial checkpoint's number of confident answers.

  1. Post-training raises the answers a model already gives, with at most a weak preference for correct ones (Sections 3.3–3.4). We follow the same answer to the same question across checkpoints. With the prompt held fixed, the weights alone multiply confident errors 4.9–17.5×, and 73–89% of the new confident errors are a wrong answer the base model already gave. Under one shared prompt, at every stage of three public pipelines (Tulu 3, Zephyr and Tulu 2.5), 79–99% come from the preceding checkpoint's own wrong answers. Of a correct and a wrong answer to two questions that start equally confident, the correct one ends more confident with probability 0.60–0.68 when only the weights change and 0.53–0.65 at the stages of public pipelines. For a checkpoint that does not change, this probability is 0.52–0.55.

  2. Which answers gain confidence depends on the data, and how far they rise tracks the margin (Sections 5.3–5.4). With the model, objective and budget fixed, data that reinforce the model's own wrong answers double its confident errors under RPO (1,613 → 3,379). A correct answer then ends above a wrong one with probability 0.57. Data that contrast these wrong answers with correct ones raise this probability to 0.76 and make 631 errors at fixed coverage, against 1,677. Following the reviewers' suggestions, we trained IPO, cDPO, an SLiC-HF-style hinge and DPO with β=0.2\beta = 0.2 for ten epochs, next to DPO and two forms of BDM. Errors at fixed coverage follow the preference margin (Spearman ρ=0.94\rho = 0.94 over 42 checkpoints, an association). A bound such as BDM keeps confident errors near the initial level but adds little preference for correct answers, so the revision keeps BDM as one bounded objective among these.

  3. A mitigation that acts on which answers gain confidence (Section 5.5). The model's own answers to checkable questions are labeled correct or incorrect, and the reference answer is added when none is correct. Each answer receives a Brier-score reward on its confidence pˉ\bar{p}, the quantity a confidence threshold reads. As an RL stage after the public SFT, DPO and RLVR checkpoints of Tulu 3 and from Mistral-7B-Instruct (two seeds each), it leaves 36–61% fewer confident errors and 33–57% fewer errors at fixed coverage than standard GRPO on the same questions. On PopQA, which no run trains on, it leaves 14–37% fewer errors at fixed coverage. Added to a generic SFT or preference stage (two seeds), it leaves 45–58% fewer errors at fixed coverage than the initial checkpoint, with about as many confident correct answers. The preference stage keeps about the same held-out preference accuracy (70.2% against 70.8%). SFT on the reference answers lowers the whole confidence level instead, and over 5–50% coverage its area under the risk–coverage curve (lower is better) is higher than that of our RL stage in all 16 pairs of seeds (Appendix K.3). The scope is short-answer questions with verifiable answers.

The anonymous repository now contains the scripts behind the revised results, with a map from each table and figure to its script. The replies to each reviewer give the section and table for each of their points. We would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.