- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates an important yet overlooked issue in LLM post-training: reward-based post-training often focuses on final-answer correctness, while the preceding reasoning may contain hallucinations that can become reinforced through post-training. The authors define self-generated hallucinations and a memorization score, and evaluate the extent to which several models memorize these hallucinations during post-training on two datasets. The evaluation examines sentence-level hallucination rates, hallucination log-probabilities, the effect of backtracking, the relationship with question difficulty, and the effect of the KL penalty, yielding several findings.
Critical strengths
- The research question is interesting and addresses an important yet overlooked issue in LLM post-training: optimizing for final-answer correctness may cause models to memorize hallucinations in the preceding reasoning.
- The evaluation is relatively comprehensive, examining sentence-level hallucination rates, hallucination log-probabilities, the effect of backtracking, the relationship with question difficulty, and the effect of the KL penalty.
Critical weaknesses
- Some findings lack explanations of their underlying causes.
- Some details are missing or unclear.
Questions for authors
-
I am somewhat confused by the definition of the “point of hallucination”: “the training time step at which hallucination (h) was generated by the policy.” Can the model not generate hallucinations at multiple training time steps? Does this refer to the first training time step at which (h) appears? If so, since the hallucination is generated by the model itself before post-training, shouldn't (p(h)) be 1 or a relatively small value?
-
Do the distractor options in the multiple-choice questions (MCQs) include the factually correct answer to the question?
-
Some experimental observations lack explanations. For example, why is hallucination memorization stronger on QuALITY, and why do memorization scores increase with question difficulty? The explanation for why the KL penalty reduces hallucination memorization also seems somewhat contradictory. The authors suggest that it suppresses memorization by discouraging the updated policy from drifting too far from its initialization. However, the initial model itself produces hallucinations. Wouldn't keeping the updated policy close to this initial model make the post-trained model more prone to hallucination?
-
Regarding “recall at the end of epoch 2,” does epoch 2 correspond to the final stage of post-training? The main text does not report the training details, making this difficult to determine.
3 comments
Rebuttal from authors (part 2)
Q3. Explanations for experimental observations (QuALITY vs. TriviaQA, question difficulty, KL penalty).
-
Why is memorization stronger on QuALITY? We hypothesize that the difference in model initialization and training setup might explain this trend. For TriviaQA, we conduct RL training on a model that has already undergone extensive general-purpose RL. As these initial policies have already-sharpened distributions, the self-hallucinations may already have high likelihood under the policy, leaving less headroom for further increase. In contrast, for QuALITY, we conduct RL training starting from a model that has not previously undergone RL. Therefore, the initial policies in this case may have relatively flatter distributions, and newly generated facts, including hallucinations, may have lower likelihood under the policy. Subsequent RL updates may sharpen these distributions, increasing the likelihood of these generated facts more than in the case of TriviaQA. We explain this in our paper in lines 259–263 but will add further clarification.
-
Why does memorization increase with question difficulty? We believe that this is due to how GRPO computes advantages. Harder questions correspond to those with fewer correct rollouts. Since GRPO normalizes rewards within each group, these correct rollouts receive larger positive advantages than correct rollouts for easier questions. Hence, models may reinforce hallucinations from these rollouts more strongly, yielding higher memorization scores in Figure 3. We will include this explanation in our paper.
-
Clarification regarding KL penalty. We believe that there might be some confusion in the understanding of our claim here. Let us explain the intuition behind our hypothesis that a KL penalty may potentially reduce this kind of memorization. Recall that, in our paper, we measure if the memorization of the specific hallucinations a model generates during training increases.
Suppose a model places uniform probability on 20 different hallucinations and randomly samples one in its rollout, and receives a positive advantage. The update, without any KL penalty, would attempt to shift the models' distribution to increase the likelihood of the hallucination that was sampled. This increases the memorization of that particular fact compared to when it was sampled. However, a KL regularization tries to ensure that the distribution remains close to the initial uniform distribution, and may therefore reduce the extent of memorization of that hallucinated fact.
Importantly, we emphasize that we do not claim that KL regularization suppresses memorization. Rather, we presented this as a hypothesis based on the above reasoning and empirically investigated it. Our results show that the memorization effect persists, with only a modest average reduction of 0.52 points.
Q4. Clarification regarding the number of training epochs.
For our main experiments (Tables 1, 3, 4, 5, and Figures 2 and 3), we train all models for one epoch of RL, following standard GRPO implementations, and save 8 equally spaced checkpoints.
In contrast, for the forgetting ablation in Section 4 (lines 374–377, 397–401; Figure 4), we continue post-training for a second epoch and save two additional checkpoints (corresponding to epochs 1.5 and 2) to evaluate whether previously memorized hallucinations from the first epoch are forgotten under further training. We mention this in the Appendix (lines 806–808) but will add it to the main text for better clarity.
Dear authors,
Thank you for the rebuttal, which answered my questions. I will maintain my original positive rating. Best.
Rebuttal from authors (part 1)
We thank the reviewer for their valuable feedback. We are glad that the reviewer found our research question interesting and important, and our evaluation comprehensive. We address the main questions below.
Q1. Definition of the point of hallucination.
You are absolutely right that the model generates hallucinations at different points during training. Let us clarify our definition of p(h), i.e., the point of hallucination.
During training, a model generates rollouts for a different batch of questions in its training data at each time step. Let us suppose that the model sees question qi at time step t=10 of training, and the corresponding rollout contains hallucination hi. This rollout receives a reward, and the model is updated accordingly. Under our definition, the point of hallucination for hi, i.e., p(hi) is 10. In particular, rollouts (and hallucinations) are generated by the model throughout training. Therefore, different hallucinations can and do have different points of hallucination.
In short, the point of hallucination (p(hi)) for each hallucinated fact (hi) corresponds to the training step at which the model encounters the associated question, generates hi in its rollout, and receives a training signal that reinforces it.
Q2. Validity of distractor options and the presence of the factually correct ground truth.
This is a good question! For QuALITY, we found that the ground truth answer did not appear in distractors. This is because of two reasons: often the hallucinated fact concerned a completely new event or property for which there did not exist a corresponding ground truth in the story. Furthermore, we used GPT-4.1-mini to generate distractors without providing it access to the story, although it might have seen these in its training data. For TriviaQA, we sampled 200 random MCQs from our HalluSet-train and annotated whether any of the distractor options were the ground truth. We found that the factual ground truth appeared in 39/200 questions (19.5%). In fact, we deliberately do not enforce the exclusion of factually correct distractors, as their presence makes our results stronger: if the model increasingly selects the hallucination despite having the correct answer available, it provides stronger evidence of hallucination memorization. Among these 39 questions, the number of times the model selects the factual ground truth decreases from 27 initially to 15 at the end of epoch 1, while the number of times it selects the hallucination over the ground truth increases from 8 to 14, respectively.