- Rating
- 1
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how post-training affects confidence hallucinations. Across different model families and alignment methods, the authors find that post-training can amplify models' confidence in factually incorrect answers. They then provide a theoretical explanation for this phenomenon and propose Bounded Dynamic Margin DPO to mitigate this alignment paradox.
Critical strengths
- The problem studied in this paper is of practical significance.
- The evaluation is relatively comprehensive. The authors conduct investigative experiments across five base and instruction-tuned models and validate the proposed methods on two models in two alignment states.
Critical weaknesses
W1. Inconsistency between the description and implementation of the proposed BDM-DPO
Eq. 6 and Algorithm 1 specify the following loss function:
=-\log\sigma\left(\min(\beta\Delta,M_{\max})\right)$$ where $\Delta$ is the unscaled preference log-ratio difference. However the provided implementation bounds the unscaled margin and multiplies by $\beta$ afterward [[Link1]](https://archive.ph/9upQq) [[Link2]](https://anonymous.4open.science/r/HCE-1B12/src/bdm/loss.py#L115) [[Link3]](https://anonymous.4open.science/r/HCE-1B12/src/bdm/loss.py#L111) [[Link4]](https://anonymous.4open.science/r/HCE-1B12/src/bdm/loss.py#L11-L14), resulting in the following loss function: $$L_{\mathrm{code}} =-\log\sigma\left(\beta\min(\Delta,M_{\max})\right)$$ With $\beta=0.1$, $M_0=3.8$, and $H_{\mathrm{ref}}=0$, the transition in the implemented code occurs around raw margin 3.8, whereas the hard-clamping threshold implied by Eq. 6 in the paper is 38; the corresponding loss floors are approximately 0.521 for the code implementation and 0.022 for the paper’s objective. This is a substantial change in optimization behavior. ### **W2. Limited novelty and insufficient comparisons with prior work** The paper’s central framing "preference alignment can exacerbate overconfidence and that this motivates a training-time remedy" substantially overlaps with existing research. However, the paper does not adequately position and discuss its contributions in relation to existing works: * Taming Overconfidence in LLMs: Reward Calibration in RLHF. (ICLR 2025) * Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach. (ICML 2025) * Trust Me, I’m Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer. (EMNLP 2025) * Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. (EMNLP 2023) For BDM-DPO, comparisons with IPO[1], SLiC-HF[2]’s hinge-based ranking loss, and conservative DPO with label smoothing are particularly relevant: these approaches provide alternatives to indefinitely increasing preference margins, although their objectives are not equivalent to BDM-DPO. > [1] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024 > > [2] SLiC-HF: Sequence Likelihood Calibration with Human Feedback. 2023 ### **W3. Other inconsistencies in the paper** * Inconsistencies between Table 1 and Table 9: | | Table 1 | Table 9 | |---|---|---| | Mistral-7B Gate B, Base → Instruct | 97 → 1,366 | 63 → 1,630 | | Mistral-7B accuracy, Base → Instruct | 52.3% → 41.6% | ~35.8% → 47.6% | | Qwen3-8B ratio | 35.4× | 924/92\$\approx\$10.0x | | Qwen3-14B ratio | 19.8× | 1130/113\$=\$10.0x | * Mathematically inconsistent results in Table 16: For the self-consistency on Qwen-8B, the authors report an FPR@95TPR of 0. However, for the same scores, labels, and evaluation set, the monotonicity of the ROC curve implies that $\mathrm{AUROC} \geq 0.95(1-\mathrm{FPR@95TPR})$. Therefore, an FPR@95TPR of 0 would require an AUROC of at least 0.95, whereas the paper reports an AUROC of 0.516Questions for authors
-
Can you clarify the identified inconsistencies meantioned in the weakness section?
-
Which loss function is actually used to obtain the experimental results presented in the paper?
-
In Appendix A.4, the authors report differences in statistics between a pilot subset (n = 2,000) and the full datasets. I'm confused about the purpose of this pilot subset and the motivation for reporting these differences.
-
For Table 2, could the authors rerun C3 using exactly the same instructions and few-shot examples as C1? Appendix A.5 shows that their prompts differ, confounding parameter updates with prompt effects. Moreover, Table 2 attributes 49–85% of the increase to the template. How does this support the claim that “parameter updates account for the vast majority”?
4 comments
Response to Reviewer nfs9 (2/2)
W3 (a) (Table 1 vs. Table 9). Thank you for catching this. Table 8 of the revision now comes from the Table 1 run and adds up to it, so all four rows of your comparison agree with Table 1.
| Table 1 | Revised Table 8 | |
|---|---|---|
| Mistral-7B Gate B, base → instruct | 97 → 1,366 | 17 + 80 → 516 + 850 (TriviaQA + PopQA) |
| Mistral-7B accuracy, base → instruct | 52.3% → 41.6% | 52.3% → 41.6% (instruct 62.8% and 25.0%) |
| Qwen3-8B ratio | 35.4× | 50 → 1,768 (35.4×) |
| Qwen3-14B ratio | 19.8× | 104 → 2,061 (19.8×) |
W3 (b) (Table 16). You are right, and thank you for the careful check. In the revised Table 16, self-consistency has FPR@95TPR = 1.000 and an AUPR near the share of errors (0.903 and 0.909, against 89% and 91%), i.e., chance level. Every row satisfies . The table now states the sample of every row (Appendix D).
Q3 (the pilot subset). Thank you for asking. The 2,000-question subset (1,000 per benchmark) was used only for the 13-configuration grid (Table 21), because every configuration needs its own training and evaluation; all other results use all 22,267 questions. We report the two rates to show that the subset is representative for this grid. With the zero-shot prompt of Appendix F, the initial model makes confident errors on 7.00% of the subset and 7.25% of all questions. The subset is now described where the grid is introduced (Appendices A.3 and F.4).
Q4 (the C3 prompt; the share of the weights). Thank you for both questions, which led us to rewrite Section 3.3. Following your suggestion, C3 now uses exactly the C1 prompt. Both conditions render the same instruction, the same eight few-shot examples and the same four negative examples with the same function. The prompt string of each of the 22,267 questions is therefore identical for the base and the instruct model (Appendix A.4 shows it), and both are counted with the Table 1 pipeline. The prompt is therefore no longer confounded with the weights in the C1 → C3 comparison. With only the weights changed, Gate B errors rise 4.9–17.5× in every family (Mistral-7B 97 → 1,134; Table 2).
You are also right that the shares do not support "the vast majority". Along C1 → C3 → C2, the weights account for 25–83% of the base-to-chat increase per family (49% overall) and the template for the remaining 17–75% (51% overall), and the split depends on this order. Section 3.3 now reports these shares and draws only the fixed-prompt conclusion, that the weights alone multiply confident errors in every family and the chat template adds a further increase. The source tracing of Section 3.4 uses the same fixed prompt, so its finding that 73–89% of the new confident errors are the base model's own wrong answers does not depend on the template.
We hope the corrections and clarifications above settle the points you raised, and we would be glad to answer any further questions.
Thank you for the responses. While the substantial revisions and additional experiments address some technical issues, my overall assessment remains unchanged.
Specifically, the authors acknowledge that the core loss function was incorrectly defined in the original submission, invalidating a substantial portion of the original theoretical analysis, including Theorem 1 (Unbounded Margin Divergence on Factual Errors) and several corollaries. Consequently, the original theoretical explanation has been substantially narrowed, and several conclusions previously used to motivate BDM are no longer maintained. Meanwhile, the main mitigation contribution now shifts toward the newly introduced Brier-reward method in Section 5.5, with a different training objective and theoretical proposition. Given the extent of these revisions and the changes to the core narrative and central claims (as can be seen in the revised manuscript, where nearly the entire paper has been rewritten and highlighted in blue), I believe this paper is more suitable for major revision and resubmission. Therefore, I maintain my original rating.
Clarification on the Revision
Thank you for your quick reply despite our late responses, and for noting that the revisions and additional experiments address some of the technical issues. We would like to clarify two points.
First, the error you found was in how Section 5.2 stated the BDM loss (Eq. 6). All reported BDM experiments were run with the implemented loss, which the revision now states (Eq. 3). Theorem 1 and its proof concerned the standard DPO loss and did not use Eq. 6, so this correction did not affect them. We restated Theorem 1 per preference pair (Remark 1) because gradients of different pairs can conflict, as Reviewer TATw noted. We then measured the margins of the full model. These measurements still motivate a bound on the margin. After ten epochs, the held-out margin is 41.1 under standard DPO and 3.4 under BDM (Table 4). Across seven objectives, errors at fixed coverage rise with the margin (Spearman over 42 checkpoints).
Second, the largest additions come from Reviewer TATw's question on the post-training data. The data experiment it led to (Section 5.4) shows that the data decide which answers gain confidence, and Section 5.5 builds on this result. Most of the other blue text marks corrections and narrower statements that the reviews asked for.
The main result and the method of the submission are kept. Table 1 is unchanged, and BDM remains in Sections 5.2–5.3 with the same result as in the submission, 35.3% fewer confident errors than standard DPO at epoch 8. Following your suggestion, Table 4 compares BDM with IPO, cDPO and an SLiC-HF-style hinge. These objectives also stop rewarding a larger margin, and all of them keep confident errors near the initial level. The mitigation therefore has two parts. A bound on the margin, with BDM as one form, stops the growth of confident errors. Training on the model's own answers labeled correct or incorrect, with a Brier reward, lowers errors at fixed coverage below the initial level (Section 5.5).
Thank you again for your careful review and for checking our code.
Response to Reviewer nfs9 (1/2)
Thank you for your careful review, including the check of our code. Changes are marked in blue.
W1 / Q2 (which loss was used). You are right, and we apologize for the mismatch. The BDM results were produced by the implemented loss, L=−logσ(β⋅min(Δ,M)) with M=M0exp(−γH). Here H is the reference model's mean token entropy over the prompt, normalized by ln∣V∣, and β=0.1, M0=3.0, γ=2.0 outside the sweeps of Appendix F. Section 5.2 (Eq. 3) and Algorithm 1 (Appendix F.3) now state this loss and these values, and the defaults in the repository's loss.py match them. As you computed, the two forms optimize differently, and Table 4 measures this. It also trains the form with β inside the minimum, −logσ(min(βΔ,M)), with M0=3.8, γ=1.0 and first-token entropy, so this run also differs in these settings. After ten epochs it has a held-out margin of 21.9 and 2,538 errors at fixed coverage, close to DPO with β=0.2 (22.9 and 2,141). The implemented form ends at 3.4 and 1,661, against 1,615 initially. The repository now contains the scripts behind the revision's results with a map from each table and figure to its script.
W2 (positioning; IPO, SLiC-HF, cDPO). Thank you for these references and baselines. We agree that this framing overlaps with prior work. The revision credits the degradation to that prior work and asks which answers receive the added confidence (general response). Section 2 now discusses the four papers. Leng et al. (2025) correct the reward model to reduce verbalized overconfidence in PPO, and Xiao et al. (2025) restore multiple-choice confidence reliability after preference alignment with an ECE-based regularizer. Tian et al. (2023) show that a verbalized confidence can be more reliable after RLHF. A confidence threshold at decoding reads the token probabilities of a free-form answer instead (their geometric mean pˉ), which is what we measure and what our mitigation acts on (Section 5.5). Simhi et al. (2025) show that models can be certain and wrong on questions they can answer. In our tracing, answers that were correct before are 7–14% of the new confident errors when only the weights change, and 73–89% are wrong answers the base model already gave (Section 3.4). We trained IPO, cDPO (ϵ=0.1) and an SLiC-HF-style hinge loss for ten epochs with the same data and evaluation. The initial checkpoint has 1,615 errors among its 7,308 most confident answers and 47.8% accuracy.
IPO, cDPO, the hinge and BDM stay close to the initial checkpoint, and across 42 checkpoints the errors track the held-out margin (Spearman ρ=0.94, an association); none adds more than a weak preference for correct answers (Table 28). BDM is now reported as one bounded objective among these, and the revision's mitigation acts on which answers gain confidence instead (Section 5.5). As an RL stage from four initial checkpoints (two seeds each), it leaves 36–61% fewer confident errors and 33–57% fewer errors at fixed coverage than standard GRPO on the same questions. We also applied SLiC-HF's hinge to our labeled groups in place of the Brier reward (RL stage from the Tulu 3 DPO checkpoint, two seeds per setting; Appendix K.3). Without its NLL term, 295–347 of the 4,738 most confident answers are wrong, against 224 and 213 for our loss and 376 and 417 for standard GRPO. With the NLL term, the ranking falls below that of the initial checkpoint (1,049 and 2,104 against 467).