acceptodds
Under review as a conference paper at ICLR 2027

LLMs Memorize Their Own Hallucinations During Post-Training

Abstract

Reasoning domains have shaped the post-training recipes of frontier models even though they are trained and deployed in domains beyond math and code. In this paper, we study the consequences of directly adopting reasoning-centric training recipes, e.g., the use of outcome-based rewards for RL, injection of “cognitive” behaviors such as proposing and backtracking from hypotheses in reasoning chains, etc., for knowledge. We hypothesize that these decisions, while useful for reasoning, inadvertently encourage models to memorize hallucinations they themselves generate in their long reasoning chains when trained on knowledge tasks. To study this, we train models with standard post-training, extract their self-generated hallucinations during RL, and track memorization throughout the rest of training. Our experiments on six open models across two model families and two architectures confirm our hypothesis. We find that models' recall of hallucinations reinforced during training increases by 4.83% on average, compared to just 0.17% for self-hallucinations that receive no training signal. In fact, models even memorize false facts that they immediately retract in their reasoning; a backtracking template inherited from reasoning-centric training that has no benefit for knowledge recall. Furthermore, we find that memorization is strongest on the hardest training questions, and rewarding a hallucination makes the model less likely to abstain from it later (by 7.23% on average). Overall, our work identifies and systematically studies a previously unexamined pathway through which models learn hallucinations during training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, hallucinations, memorization, reasoning models, RLVR, GRPO

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kNevOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an important yet overlooked issue in LLM post-training: reward-based post-training often focuses on final-answer correctness, while the preceding reasoning may contain hallucinations that can become reinforced through post-training. The authors define self-generated hallucinations and a memorization score, and evaluate the extent to which several models memorize these hallucinations during post-training on two datasets. The evaluation examines sentence-level hallucination rates, hallucination log-probabilities, the effect of backtracking, the relationship with question difficulty, and the effect of the KL penalty, yielding several findings.

Critical strengths
  • The research question is interesting and addresses an important yet overlooked issue in LLM post-training: optimizing for final-answer correctness may cause models to memorize hallucinations in the preceding reasoning.
  • The evaluation is relatively comprehensive, examining sentence-level hallucination rates, hallucination log-probabilities, the effect of backtracking, the relationship with question difficulty, and the effect of the KL penalty.
Critical weaknesses
  • Some findings lack explanations of their underlying causes.
  • Some details are missing or unclear.
Questions for authors
  • I am somewhat confused by the definition of the “point of hallucination”: “the training time step at which hallucination (h) was generated by the policy.” Can the model not generate hallucinations at multiple training time steps? Does this refer to the first training time step at which (h) appears? If so, since the hallucination is generated by the model itself before post-training, shouldn't (p(h)) be 1 or a relatively small value?

  • Do the distractor options in the multiple-choice questions (MCQs) include the factually correct answer to the question?

  • Some experimental observations lack explanations. For example, why is hallucination memorization stronger on QuALITY, and why do memorization scores increase with question difficulty? The explanation for why the KL penalty reduces hallucination memorization also seems somewhat contradictory. The authors suggest that it suppresses memorization by discouraging the updated policy from drifting too far from its initialization. However, the initial model itself produces hallucinations. Wouldn't keeping the updated policy close to this initial model make the post-trained model more prone to hallucination?

  • Regarding “recall at the end of epoch 2,” does epoch 2 correspond to the final stage of post-training? The main text does not report the training details, making this difficult to determine.

3 comments
AuthorsOpenReview ↗

Rebuttal from authors (part 1)

We thank the reviewer for their valuable feedback. We are glad that the reviewer found our research question interesting and important, and our evaluation comprehensive. We address the main questions below.

Q1. Definition of the point of hallucination.

You are absolutely right that the model generates hallucinations at different points during training. Let us clarify our definition of p(h)p(h), i.e., the point of hallucination.

During training, a model generates rollouts for a different batch of questions in its training data at each time step. Let us suppose that the model sees question qiq_i at time step t=10t=10 of training, and the corresponding rollout contains hallucination hih_i. This rollout receives a reward, and the model is updated accordingly. Under our definition, the point of hallucination for hih_i, i.e., p(hi)p(h_i) is 10. In particular, rollouts (and hallucinations) are generated by the model throughout training. Therefore, different hallucinations can and do have different points of hallucination.

In short, the point of hallucination (p(hi))(p(h_i)) for each hallucinated fact (hi)(h_i) corresponds to the training step at which the model encounters the associated question, generates hih_i in its rollout, and receives a training signal that reinforces it.



Q2. Validity of distractor options and the presence of the factually correct ground truth.

This is a good question! For QuALITY, we found that the ground truth answer did not appear in distractors. This is because of two reasons: often the hallucinated fact concerned a completely new event or property for which there did not exist a corresponding ground truth in the story. Furthermore, we used GPT-4.1-mini to generate distractors without providing it access to the story, although it might have seen these in its training data. For TriviaQA, we sampled 200 random MCQs from our HalluSet-train and annotated whether any of the distractor options were the ground truth. We found that the factual ground truth appeared in 39/200 questions (19.5%). In fact, we deliberately do not enforce the exclusion of factually correct distractors, as their presence makes our results stronger: if the model increasingly selects the hallucination despite having the correct answer available, it provides stronger evidence of hallucination memorization. Among these 39 questions, the number of times the model selects the factual ground truth decreases from 27 initially to 15 at the end of epoch 1, while the number of times it selects the hallucination over the ground truth increases from 8 to 14, respectively.

AuthorsOpenReview ↗

Rebuttal from authors (part 2)

Q3. Explanations for experimental observations (QuALITY vs. TriviaQA, question difficulty, KL penalty).

  1. Why is memorization stronger on QuALITY? We hypothesize that the difference in model initialization and training setup might explain this trend. For TriviaQA, we conduct RL training on a model that has already undergone extensive general-purpose RL. As these initial policies have already-sharpened distributions, the self-hallucinations may already have high likelihood under the policy, leaving less headroom for further increase. In contrast, for QuALITY, we conduct RL training starting from a model that has not previously undergone RL. Therefore, the initial policies in this case may have relatively flatter distributions, and newly generated facts, including hallucinations, may have lower likelihood under the policy. Subsequent RL updates may sharpen these distributions, increasing the likelihood of these generated facts more than in the case of TriviaQA. We explain this in our paper in lines 259–263 but will add further clarification.

  2. Why does memorization increase with question difficulty? We believe that this is due to how GRPO computes advantages. Harder questions correspond to those with fewer correct rollouts. Since GRPO normalizes rewards within each group, these correct rollouts receive larger positive advantages than correct rollouts for easier questions. Hence, models may reinforce hallucinations from these rollouts more strongly, yielding higher memorization scores in Figure 3. We will include this explanation in our paper.

  3. Clarification regarding KL penalty. We believe that there might be some confusion in the understanding of our claim here. Let us explain the intuition behind our hypothesis that a KL penalty may potentially reduce this kind of memorization. Recall that, in our paper, we measure if the memorization of the specific hallucinations a model generates during training increases.

    Suppose a model places uniform probability on 20 different hallucinations and randomly samples one in its rollout, and receives a positive advantage. The update, without any KL penalty, would attempt to shift the models' distribution to increase the likelihood of the hallucination that was sampled. This increases the memorization of that particular fact compared to when it was sampled. However, a KL regularization tries to ensure that the distribution remains close to the initial uniform distribution, and may therefore reduce the extent of memorization of that hallucinated fact.

    Importantly, we emphasize that we do not claim that KL regularization suppresses memorization. Rather, we presented this as a hypothesis based on the above reasoning and empirically investigated it. Our results show that the memorization effect persists, with only a modest average reduction of 0.52 points.



Q4. Clarification regarding the number of training epochs.

For our main experiments (Tables 1, 3, 4, 5, and Figures 2 and 3), we train all models for one epoch of RL, following standard GRPO implementations, and save 8 equally spaced checkpoints.

In contrast, for the forgetting ablation in Section 4 (lines 374–377, 397–401; Figure 4), we continue post-training for a second epoch and save two additional checkpoints (corresponding to epochs 1.5 and 2) to evaluate whether previously memorized hallucinations from the first epoch are forgotten under further training. We mention this in the Appendix (lines 806–808) but will add it to the main text for better clarity.

Dear authors,

Thank you for the rebuttal, which answered my questions. I will maintain my original positive rating. Best.

Reviewer_8nYAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether large language models memorize their own hallucinations during reinforcement learning with outcome-based rewards. The authors focus on false factual statements generated within reasoning trajectories that ultimately receive positive rewards, and examine whether such statements become more likely to be recalled after training.

To investigate this phenomenon, the paper introduces a memorization metric that tracks changes in the probability of recalling hallucinated statements before and after their reinforcement during training. Experiments across six models and two knowledge-oriented datasets, QuALITY and TriviaQA, reveal that self-generated hallucinations are increasingly memorized during RL training. Interestingly, the effect persists even when the model subsequently corrects the hallucinated statement within the same reasoning trajectory. Additional analyses investigate the relationship between memorization, question difficulty, abstention behavior, and KL regularization.

The findings highlight a potential limitation of outcome-based RL: rewarding successful reasoning trajectories may inadvertently reinforce incorrect intermediate factual statements.

Critical strengths

S1. The paper provides an interesting perspective on hallucination by studying how models may acquire and reinforce false factual knowledge during post-training, rather than focusing solely on hallucinations at inference time. The observation that incorrect statements can be memorized even when they are subsequently corrected within the same reasoning trajectory is particularly interesting. This finding challenges the assumption that successful final outcomes or explicit self-correction necessarily prevent intermediate errors from being reinforced. I consider this an important contribution to understanding the limitations of outcome-based RL.

S2. The paper provides a relatively comprehensive empirical investigation across multiple models and knowledge tasks. The use of held-out hallucination controls and checkpoint-level memorization measurements helps distinguish the observed phenomenon from general changes in model output probabilities. The analyses of backtracking, question difficulty, abstention behavior, and KL regularization further strengthen the findings. Together, these experiments provide useful evidence that the identified phenomenon is systematic rather than an isolated observation.

Critical weaknesses

W1. The experimental pipeline relies heavily on LLM-based extraction and verification of hallucinated statements, as well as automatically generated multiple-choice questions to measure subsequent memorization. Although this is a reasonable and scalable evaluation strategy, the accuracy of the extracted hallucination labels and the quality of the constructed probes are central to the paper's conclusions. Without systematic human validation or additional controls for distractor quality and answer-position biases, it remains unclear to what extent the measured memorization effects could be influenced by annotation noise or evaluation artifacts.

Furthermore, the MCQ-based memorization metric does not directly establish that models are more likely to reproduce these hallucinations during open-ended generation. The token-level probability analyses partially address this concern, but direct free-form evaluations would strengthen the practical implications of the findings.

W2. The paper presents compelling evidence that self-generated hallucinations become more likely to be recalled during RL training. However, the extent to which this phenomenon specifically results from positive outcome rewards, rather than the broader effect of training exposure to self-generated statements, could be more directly investigated. While the held-out controls are useful, additional comparisons involving alternative training objectives or different reward assignments would help isolate the proposed mechanism.

Similarly, the observation that backtracked hallucinations are memorized is interesting, but it remains unclear whether this behavior is primarily driven by outcome-based reinforcement, reasoning trajectory length, or the repeated occurrence of incorrect factual content. More controlled experiments would help strengthen the causal interpretation.

Questions for authors

Q1. Have the authors conducted human validation of the automatically extracted hallucinated statements and their corresponding backtracking labels? Could the authors report annotation accuracy on a representative subset, along with the number and distribution of hallucinated statements used in each experimental setting?

Q2. How sensitive are the reported memorization effects to the construction of MCQ distractors and answer-option ordering? More importantly, do the memorized hallucinations also become more likely to appear in free-form generation, rather than only in the constructed multiple-choice probes?

Q3. Could the authors compare outcome-based RL against alternative training settings, such as supervised training on the same trajectories or training with different reward assignments? Such comparisons would help determine whether the memorization effect is specifically attributable to positive outcome rewards rather than exposure to generated false statements during optimization.

Q4. How does hallucination memorization vary with reasoning trajectory length, the frequency of hallucinated statements, and the number of subsequent corrections? Could these factors explain part of the observed memorization effect, particularly for difficult questions?

Q5. Could the authors provide confidence intervals or variance estimates for the memorization differences, particularly across different random seeds? Additionally, have the authors explored whether simple factuality-aware or process-level reward modifications can reduce the observed effect without substantially degrading downstream task performance?

6 comments
AuthorsOpenReview ↗

Rebuttal from authors (part 1)

We thank the reviewer for their valuable feedback. We are glad that the reviewer found our perspective on how models acquire and reinforce false knowledge interesting, particularly the memorization of backtracked hallucinations, and our empirical investigation comprehensive. We address the questions below.

Q1. Human validation of hallucination and backtracking labels.

We performed several rounds of spot checks while developing our HalluSet generation pipeline, refining the prompts through several rounds of prompt engineering and manual spot-checking. We found that the combination of our prompts and the frontier LLM we use (GPT 4.1-mini) to be very performant for our task. We discuss human validation of the two components the reviewer inquired about below.

Human validation of hallucinated facts. We human-validated that the facts that the pipeline identifies as hallucinated are in fact non-factual (reported in Appendix A.4 under "Reliability of LLM-based annotation"). Recall that for QuALITY, detecting hallucinations merely involves comparing against a short story (~5700 tokens on avg.) and frontier LLMs have already been shown to be strong at this (Shaham et al., 2023; Jacovi et al., 2025). Therefore, our human validation focuses on the more challenging TriviaQA setting that involves verifying facts against the internet. One of the authors manually verified 200 randomly sampled facts from TriviaQA's HalluSet, i.e., those labeled as hallucinated, against the web. We found a precision of 98% on this subset.

Human validation of backtracking labels. We conduct an additional manual evaluation of 200 randomly sampled examples from TriviaQA. We manually verified whether claims labeled as backtracked or non-backtracked were correctly classified. We found 87% precision for backtracked claims and 91% for non-backtracked claims, showing that the LLM-annotations were reliable. In fact, the error cases above were genuinely ambiguous cases where the model, say, expressed high uncertainty, which can be viewed as implicit backtracking behavior.

AuthorsOpenReview ↗

Rebuttal from authors (part 2)

Q2. Sensitivity to MCQ distractors and option ordering, and free-form generation.

This is a good point! We agree that it would be valuable to confirm that our findings are not sensitive to these effects. We followed the reviewer's suggestion and investigated the sensitivity of our memorization scores against (1) MCQ distractor construction and (2) answer-option ordering.

1. Sensitivity to distractor construction. For Qwen3-4B and Olmo-3-7B-Think, we regenerated new distractor options for the questions in HalluSet-train. We then recomputed the memorization scores using these new MCQs. The table below reports these new scores and the old ones from Table 1 of the paper for comparison.

TaskQwen3-4BOlmo-3-7B-Think
Quality (Table 1)5.855.11
Quality (New run)5.725.30
TriviaQA (Table 1)3.194.24
Trivia QA (New run)3.274.03

We observe only small differences in memorization scores, with a mean absolute change of 0.15, indicating that our results are not sensitive to the distractors used.

2. Sensitivity to answer-option ordering. For all MCQs in HalluSet-train, we randomly shuffled the options and repeated inference across all models and datasets, keeping the questions and distractors unchanged.

TaskQwen3-4BQwen3-8BQwen3-14BQwen3-30B-A3BOlmo-3-7B-ThinkOLMoE-1B-7B
Quality (Table 1)5.856.145.524.445.116.88
Quality (New run)5.776.255.294.605.206.72
TriviaQA (Table 1)3.193.713.563.774.245.58
Trivia QA (New run)3.073.883.493.694.325.81

Once again, the scores remain very similar, with a mean absolute change of 0.13 and a maximum change of 0.23.

We thank the reviewer for this suggestion and we will include our paper as they strengthen our empirical evidence!

We also want to highlight that, conceptually, our memorization score's design is already robust to such potential confounders. This is because it does not report absolute scores for a specific checkpoint but rather the change in this score for different checkpoints throughout training. Consider an extreme case where the test dataset contains only implausible distractors. In that case, it is possible that the absolute MCQ accuracy might be artificially high when evaluating a single checkpoint. However, this effect will cancel out when computing the difference. Similarly, any underlying positional bias effects would also get cancelled out (as an additional check, we verified that the MCQ generation pipeline diversifies the position at which the hallucination is placed, so this was not a concern). Together, these design choices, along with our additional experiments, show that our results are robust to variations in distractor construction and answer-option ordering.

Free-form generation As the reviewer states, we also conducted a likelihood-based evaluation to support the conclusions we draw from MCQs. These results showed that the probability of hallucinations in HalluSet-train increases substantially after training. We agree that free-form generation would be the most pertinent setting, but evaluating reasoning models in that setting robustly is extremely challenging.

Consider the original TriviaQA task – given a question about an entity, the model often generates hallucinations for related as well as unrelated entities that might be alternate candidates. In some trajectories, it might hallucinate a new, say, date of birth, while in the other it might hallucinate a location. Ideally, free-form evaluation would measure hallucination recall in the same setting. But ensuring that the reasoning trajectory generates hallucinations for the same properties is challenging, and any constrained decoding would lead to off-distribution samples. Therefore, we believe that our evaluation balanced controllability and robustness.



Q3. Isolating the role of positive outcome rewards from exposure to false statements.

As we understand it, the reviewer's concern is that the memorization we observe could be due to any exposure to false statements during training, and not necessarily because of the positive outcome rewards assigned to trajectories with hallucinations. To address this point, our submission also reports memorization scores for hallucinations extracted from trajectories with an incorrect final answer (called HalluSet-neg), i.e., a zero reward. These are hallucinations that the LLM was exposed to during training, but have a negative trajectory-level advantage. Memorization scores for these were reported in Table 13 in our Appendix. We copy-paste them below as well.

TaskQwen3-4BQwen3-8BQwen3-14BQwen3-30B-A3BOlmo-3-7B-ThinkOLMoE-1B-7B
Quality0.611.15-0.40-1.30-0.23-0.86
TriviaQA-3.01-2.18-2.38-2.55-0.09-1.69
AuthorsOpenReview ↗

Rebuttal from authors (part 3)

(Continued from previous comment.)

As we can see in the above table, memorization scores here are either negligible or negative. This shows that memorization does not happen merely due to exposure to false statements during training. Instead, they need to be associated with a correct final answer, and consequently, a positive advantage to be memorized.

To clarify, in our paper, we report memorization scores for three HalluSets:

  1. HalluSet-train – these are hallucinations generated by the model during training and receive a positive reward that the update optimizes. (memorization scores in Table 1)
  2. HalluSet-heldout – these are hallucinations generated by the same model checkpoints, but the model is not exposed to them during training. (memorization scores in Table 1)
  3. HalluSet-neg – similar to (1), these are hallucinations generated by the model during training, but their corresponding trajectories receive a negative reward. (memorization scores in Table 13)

We compare memorization scores of (1) and (2) in the main paper to show that exposure during training is necessary. In addition to that, the comparison between (1) and (3) shows that memorization only occurs when hallucinated statements receive a positive outcome reward, and that exposure alone does not cause memorization. We will move results with HalluSet-neg to the main paper to illustrate this.

Suggested SFT training experiment: We admit we are slightly confused about the exact experiment that the reviewer has suggested. As we understand it, the suggested experiment is to conduct SFT training on the same trajectories that were generated during RL. As SFT is always conducted on "good" demonstrations, this would mean performing rejection sampling, i.e., identifying the subset of trajectories with the correct final response and conducting SFT on those. Our confusion is that this uses the same outcome-only way of evaluating response correctness. As such, the model will be exposed to false statements during training because of the outcome-based rewards. Therefore, we are unsure that this would disentangle the effect of "positive outcome rewards" from "exposure to generated false statements during optimization". If the reviewer has a different experiment in mind, please let us know, and we would be happy to run those.

Nevertheless, we ran experiments on SFT w/ rejection sampling on two models by re-using the trajectories from GRPO and applying filtering using outcome-based correctness. We find that, as expected, the memorization scores remained positive (5.01 and 4.47 for QuALITY for Qwen3-4B and Olmo-3-7B-Think, respectively, and 3.05 and 4.03 for these two models for TriviaQA) and comparable to GRPO. However, we emphasize that this is due to both exposure and outcome-based correctness evaluation.



Q4. Effect of trajectory length and hallucination frequency.

Thank you for the question. We conducted additional experiments to study how hallucination memorization correlates with reasoning trajectory length and the number of hallucinated statements.

Reasoning trajectory length. We compute the Pearson correlation between reasoning length (in words) and memorization score across all six models on both datasets. We find no correlation, suggesting that memorization strength does not align with the length of the thinking traces.

TaskQwen3-4BQwen3-8BQwen3-14BQwen3-30B-A3BOlmo-3-7B-ThinkOLMoE-1B-7B
Quality0.05050.04370.07850.04830.02330.0657
TriviaQA-0.0520.0491-0.109-0.091-0.04570.0448

Frequency of hallucinated facts. For each hallucination h in HalluSet-train, we use GPT-4.1-mini to identify the number of times the same hallucination was repeated in the corresponding reasoning trace. We conduct this analysis on TriviaQA, where reasoning traces frequently contain backtracking and the model repeats/revisits factual claims. We skip QuALITY for this analysis as we found that facts are rarely repeated in a trajectory.

We report both the Pearson and Spearman rank correlation between the repetition frequency of h and the fact's memorization (While running this analysis, we realized that generally, over 80% of the hallucinations occur 1–3 times within the reasoning traces. Since this is concentrated in a small range of discrete values, we also include Spearman rank correlations.)

CorrelationQwen3-4BQwen3-8BQwen3-14BQwen3-30B-A3BOlmo-3-7B-ThinkOLMoE-1B-7B
Pearson0.08330.04670.01240.07120.04590.0971
Spearman0.09440.03780.01530.06310.04120.0888

In both settings, we find weak correlations, suggesting that repetition of a hallucination is not a strong predictor of its subsequent memorization.

(Response continued in the next comment.)

Reviewer_LczmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether outcome-based reinforcement learning causes LLMs to memorize hallucinated facts generated in their own reasoning traces, even when the final answers are correct. The authors extract self-generated hallucinations from GRPO training rollouts and track their recall across training checkpoints using multiple-choice probes. Further analyses show that models memorize even backtracked hallucinations and become less likely to abstain after reinforcement.

Critical strengths
  • Clear and well-motivated investigation of how outcome-based RL reinforces false facts in correct reasoning trajectories.
  • Interesting findings show that backtracked hallucinations are memorized equally strongly, while abstention decreases by 7.23 points.
Critical weaknesses
  • The paper primarily measures hallucination recall through constructed MCQs, while the practical significance of the reported memorization increase remains unclear.

  • Although the title broadly refers to post-training, the experiments focus exclusively on GRPO.

  • Prior works such as KnowRL have already identified and proposed methods to mitigate the reinforcement of incorrect reasoning. The paper doesn't evaluate whether existing methods can prevent hallucination memorization.

Questions for authors

Do the generated MCQs include factually correct answers, and are all options independently verified? How do the authors ensure that "None of the above" is a valid abstention option?

4 comments
AuthorsOpenReview ↗

Rebuttal from authors (part 1)

We thank the reviewer for their valuable feedback and effort in reviewing our paper! We are glad that the reviewer found our work clear and well-motivated, and our findings on memorization of backtracked hallucinations and abstention interesting. We address the main concerns below.

W1. Reliance on MCQ probes for evaluation.

We want to highlight that we do conduct a non-MCQ-based evaluation to establish the memorization effect (see results corresponding to Fig. 2). We run experiments that track whether the log-probability of generating the hallucinations, conditioned on the corresponding question, increases during training. Figure 2 reports these for checkpoints before and after the point of hallucination. Our results showed that the probability of hallucinations in HalluSet-train increases substantially at the hallucination point, and this persists as training progresses. This setup is a close proxy for free-form generation, and our results here both indicate a practical consequence and support the main conclusions from the MCQ-based evaluation. We use the MCQ-based evaluation in further ablations as they allow for more controlled experiments.

Our paper honestly and directly engages with both the advantages of the MCQ-based evaluation and its limitations (discussed in lines 199-234). As we state above, our design choice was motivated both by robustness of evaluation, as well as prior literature on probing LLMs for knowledge (Wang et al., 2024; Du et al., 2026; Rein et al., 2023).



W2. Focusing exclusively on GRPO.

Our primary focus is on investigating the consequences of using an outcome-based reward for knowledge tasks. We use GRPO as our representative post-training algorithm because it is the most commonly used one in post-training recipes in recent literature. That said, we ran additional experiments to confirm that our findings do generalize beyond GRPO. Specifically, we re-ran our main experiments on two models using PPO as the RL algorithm, with the same experimental setup as in our paper. The table reports the memorization scores for HalluSet-train after PPO, as well as GRPO from Table 1 of the paper for comparison.

Our results show that PPO leads to similar memorization of hallucinations as GRPO, with memorization scores comparable to those of GRPO. Additionally, we see similarity in other trends, such as relatively higher memorization scores for QuALITY than for TriviaQA. This provides empirical evidence that the phenomenon we highlight in this paper is not specific to GRPO. We will include results with PPO on the full set of models and datasets in our paper.

TaskQwen3-4BOlmo-3-7B-Think
Quality (Table 1 - GRPO)5.855.11
Quality (PPO)5.904.80
TriviaQA (Table 1 - GRPO)3.194.24
Trivia QA (PPO)3.563.82

Conceptually, we can understand why PPO and GRPO training led to similar conclusions. The underlying reason for the memorization we study is that trajectories with hallucinations may be assigned the maximum reward as long as the final outcome is correct. So, even though PPO and GRPO use different ways of estimating the advantage and objectives, they both aim to optimize the expected value of the same flawed reward.



AuthorsOpenReview ↗

Rebuttal from authors (part 2)

W3. Evaluation of existing mitigation methods such as KnowRL.

This is a good point! In this paper, our focus was on highlighting a previously understudied source of hallucination memorization under conventional training recipes. These standard recipes today do not include fine-grained verification of reasoning trajectories, possibly due to the associated costs (we discuss these below). Our paper, by highlighting this new issue with outcome-only rewards, serves to provide additional incentive for such reward designs.

We do view KnowRL and similar variants as promising solutions to mitigate the kind of memorization we highlight (Section 5; Discussion and Implications, and Section 6; Related Work). However, we found that it was challenging to adopt off-the-shelf KnowRL-like approaches without making significant modifications.

The main limitation we faced was cost. These approaches roughly follow a two-stage pipeline to compute the fraction of facts that are hallucinated to use as additional reward: extract atomic facts and then verify them against an evidence corpus. However, this pipeline can be prohibitively expensive, particularly for the current generation of models that generate very long reasoning chains.

Let us provide a back-of-the-envelope cost estimate for the above pipeline using the LLM used in KnowRL, namely gpt-4o-mini. Each of our six models is trained on 3,609 questions with eight rollouts per question on two datasets. Therefore, we would need to run this pipeline on 173,232 reasoning traces per dataset. Based on the average reasoning lengths in our experiments, let us use a fairly conservative estimate of ~6 atomic facts per reasoning for QuALITY and ~10 for TriviaQA, with each fact containing ~15 tokens. For QuALITY, each fact can be verified against the reference story (averaging 5,700 tokens). However, TriviaQA requires verifying arbitrary facts against the internet. In our initial experiments for constructing HalluSet, we attempted to verify facts without internet access using frontier LLMs but found the process highly error-prone. Moreover, we found that facts often contain information that is not verifiable against Wikipedia (for which offline dumps exist) and require broader internet access for reliability. Therefore, for TriviaQA, we assume web-based verification using web_search ($10 per 1K calls + a fixed 8K-token block per call at the input price). Note that this mirrors the pipeline we converged on to create a reliable HalluSet.

We report the costs for verification using two settings: verifying each fact in a separate LLM call or verifying all facts in a trajectory together (the latter can be error-prone if facts are diverse and require a large number of distinct web articles for verification). The table shows the additional costs incurred by this pipeline for each dataset, across all six models.

DatasetPer-fact verificationPer-response verification
Quality$913.94$172.07
TriviaQA$19,469.90$2005.78

Note that these are estimates for a single final run whereas our research required us to run these same experiments multiple times as we finessed our methodology. We also want to emphasize that these high costs are actually conservative estimates, assuming relatively few facts per response, short verification prompts (10 tokens long) and outputs (1 token long), and only one web-search call per verification request.

Notably, the verify-then-extract pipeline also increases latency due to the addition of these API calls per generation step. A locally hosted model of adequate performance (say Qwen3-32B) can reduce API costs but would still require retrieval from the internet, adding to latency, as well as higher GPU resources for training.

We believe that it is due to these considerations that such approaches have not found broader adoption despite showing promise. However, our work identifies a compelling reason for why these hallucinations are harmful, and we hope it will drive future investigations into cost-efficient mitigation strategies.



AuthorsOpenReview ↗

Rebuttal from authors (part 3)

Q1. Validity of MCQ options and of "None of the above" as an abstention option.

We performed several rounds of spot checks while developing our HalluSet generation pipeline. We manually verified 200 examples via spot-checking that the frontier LLMs we used were effective at generating plausible distractors. We provide pointwise responses to your specific questions below:

Do the generated MCQs include factually correct answers? This is a good question! For QuALITY, we found that the ground truth answer did not appear in distractors. This is because of two reasons: often the hallucinated fact concerned a completely new event or property for which there did not exist a corresponding ground truth in the story. Furthermore, we used GPT-4.1-mini to generate distractors without providing it access to the story, although it might have seen these in its training data. For TriviaQA, we sampled 200 random MCQs from our HalluSet-train and annotated whether any of the distractor options were the ground truth. We found that the factual ground truth appeared in 39/200 questions (19.5%). In fact, we deliberately do not enforce the exclusion of factually correct distractors, as their presence makes our results stronger: if the model increasingly selects the hallucination despite having the correct answer available, it provides stronger evidence of hallucination memorization. Among these 39 questions, the number of times the model selects the factual ground truth decreases from 27 initially to 15 at epoch 1, while the number of times it selects the hallucination over the ground truth increases from 8 to 14, respectively.

Validity of "None of the above" as an abstention option. We appreciate the reviewer's point about our abstention setup. To investigate, we manually annotated the MCQ distractors for the same 200 questions as above. We found that only 14% of these contained the ground truth answer. (Note that we randomly select a distractor option to replace with "None of the Above"; this might eliminate the ground truth and hence this number is lower than the 19.5% earlier.) This relatively small percentage cannot explain the large drops in probability of generating "None of the Above", as reported in Table 4.




References

Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., ... & Chen, W. (2024). Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37, 95266-95290.

Du, X., Yao, Y., Ma, K., Wang, B., Zheng, T., Liu, M., ... & Zhang, G. (2026). Supergpqa: Scaling llm evaluation across 285 graduate disciplines. Advances in Neural Information Processing Systems, 38.

Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., ... & Bowman, S. R. (2023). Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022.

Thanks for the clarifications. My original score reflects my recommendation properly, and I choose to keep it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.