acceptodds
Under review as a conference paper at ICLR 2027

CLOSING THE HALLUCINATION WINDOW IN DIFFUSION MODELS

Abstract

Diffusion models learn structure from noise, yet not always all of it, and this leads to hallucination. While the memorization and generalization regimes of their training dynamics have been well studied, we study how hallucination fits into these dynamics. We believe that by better understanding how hallucination occurs from an optimization perspective, we can better design regularization methods and, by extension, architectures for diffusion models that could offset hallucination during training. We find that denoising loss has an elbow, i.e., it plateaus significantly before the model actually generalizes . Consequently, hallucination can be distinguished from generalization only as an asynchrony in the optimization dynamics when the denoising loss is hierarchically resolved and rescaled. This gap, which we call the hierarchical lag, is the training dynamics signature of hallucination. We further show for a diffusion transformer trained on a random hierarchy model how a simple regularization to decrease this unintended “lag” can accelerate generalization and shrink the hallucination window in training time by an order of a magnitude.

Primary area:
generative models
Keywords:
Diffusion, Hallucination, Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5j7rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

While a recent line of work has started to tackle the dichotomy between generalization and memorization in diffusion models, the authors of this work instead address the issue of hallucination. They define hallucination as the generation of samples that are locally coherent but globally incoherent (think of a hand with 6 fingers). The authors distinguish four distinct regimes: bad generation (both local and global inconsistencies), generalization (both locally and globally consistent), hallucination (locally consistent but globally inconsistent) and memorization (verbatim repetition of the training samples). The picture they propose is that when a model is trained, it starts in the bad generation regime and ends in the memorization regime (Bonnaire et al. 2025), while the window in between can be either generalization or hallucination. In this intermediate window of training times, both the train and test losses plateau. The authors claim that it is during this plateau that the model learns the global constraints, hence it goes from hallucination to generalization. More precisely, the authors define two times: TelbT_{\mathrm{elb}}, which corresponds to the beginning of the plateau and is also the time at which the model starts to learn the local rules, and TglobalT_{\mathrm{global}}, at which the model starts to learn the global rules. The window [Telb,Tglobal][T_{\mathrm{elb}}, T_{\mathrm{global}}] is the hallucination window.

To back their claims, the authors study the Random Hierarchy Model (RHM) introduced in Petrini et al. 2023 and studied in the context of diffusion in several papers (Sclocchi et al. 2024, Garnier-Brun et al. 2025, …\dots). Created to model the hierarchical nature of real data, it has local and global rules at the same time. The loss itself can be decomposed into a term accounting for the global rules and another one accounting for the local rules.

When training a denoiser on the RHM, the local term nlocaln_{\mathrm{local}} decreases first, and the global term nglobaln_{\mathrm{global}} only decreases one or two orders of magnitude later in terms of gradient steps. This indicates that the model first learns to generate locally consistent samples before learning global consistency, hence generating hallucinated samples before samples faithful to the distribution.

In order to mitigate this, the authors propose the "Hierarchical Lag Regularizer". This regularizer adds a penalty to the loss that forces the global term to be smaller than the local term. The experiments done by the authors show that it reduces the hallucination window.

Critical strengths
  • The authors study a simple synthetic model with a clear separation between hallucination and generalization.
  • The authors propose and study a regularization that specifically mitigates hallucination. As far as I know this is the first paper studying doing this.
Critical weaknesses
  • No experiments on real data and no applicability to real data. Overall, the paper is just analyzing one experiment. Although the ideas are interesting, it is more the beginning of an article than a complete article. I advise the authors to continue working on this project and add more content.
  • For each set of parameters, the authors only did two runs and took only λ=0.1\lambda=0.1 for the regularizer strength.
Questions for authors
  • Why not study several values of λ\lambda and the effect of tuning λ\lambda? This would be a valuable addition to this work.
  • Why is there a wide white region in Fig. 5 and Fig. 6? Also, the plot is super hard to read and has too much information.
  • In Fig. 1, the authors are plotting 2 quantities, but there is one curve with different colors. What are the authors plotting exactly? If it is two quantities, then it is misleading.
  • Why only 2 seeds for numerical experiments? Would increasing the number of seeds over which the observables are averaged make the curves in the right panel of Fig. 3 less noisy?
  • The regularizer the authors studied works only for the RHM. Can the authors deduce one to mitigate hallucinations for general data distributions?
  • The authors say that TelbT_{\mathrm{elb}} and TglobalT_{\mathrm{global}} are separated by one or two orders of magnitude. What does this separation depend on: the data distribution, the choice of model architecture, the parameters (v,m,s,L)(v,m,s,L) of the RHM?
Reviewer_NV7tOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors study hallucination in diffusion models. They define a hallucination as a sample that is locally correct, but globally incorrect. They use a random hierarchical model, which gives them access to the true posterior and KL divergences through belief propagation. They identify two metrics, n_local and n_global, and assert that the gap between these needs to be small for hallucinated samples to be suppressed. After investigating hallucinations in the model itself, they proceed to use the introduced metric as a regularizer and show that this indeed suprresses hallucinated samples.

Critical strengths
  • I find the concept of local correctness and global incorectness a very interesting starting point to study hallucination.
  • The authors work quite thoroughly through a setup using analytical models, while that raises questions about generalization (see weaknesses), it does allow for some very interesting insights using KL divergences.
Critical weaknesses
  • The authors should spend some time explaining belief propagation and how it is used for diffusion models. Especially in this field I think it would serve the authors to spend some time introducing it.
  • Overall, I find the setup very difficult to follow. The paper ties a lot of weight to n_local and n_global and while those concepts are defined on paper, there is not a lot of intuition for how they come about. It would be great if the authors could conect figure 2 better to the derivation resulting in the equations on line 221
  • I find the conclusion 450-452 quite strong given the authors' particular choice of example model. Can the authors make an argument of why this should broadly generalize?
Questions for authors
  • Can the authors define "locality" in their setup? It seems to me there needs to be some kind of connection between variables, like there is in an image. Is that simply introduced through the hierarchy? It is not so clear to me what it means for something to be local in these hierarchical models.
  • Is the top row of figure 4 the same as figure 3?
  • How do the authors decide if a sample is good, bad or hallucinated? In particular for Figs 5 and 6
Reviewer_f4FbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that there exists a "hullucination window" in the training process of diffusion models under a "random hierarchy model" (RHM). Specifically, the diffusion model learns the "global" denoising rule at a smaller speed compared to the "local" denoising rule, hence demonstrating a loss gap of these two. Within this gap, they argue diffusion model hullucinates. To address this issue, they add an extra regularizer that directly penalizes this gap. Empirically, they demosntrate the gap is alleviated and the models generalize better under the regularization.

Critical strengths
  1. The observation of the hullucination gap is novel and interesting, which can potentially motivate future research.

  2. The proposed simple regularization method leads to better generalization under RHM, which further validates the claim.

Critical weaknesses
  1. I find there is a conceptual gap between the observed discripancy between n_local and n_global, and the hullucination. It seems a smaller gap between n_local and n_global does not necessarily imply the model can generalize well. It is possible that both n_local and n_global are very large, while their difference is small. In such case, the model is not good but the gap is small.

  2. Related to my first point, I feel adding the regularizer may not necessarily lead to better generalization, since a small gap between n_local and n_global seems to be a necessary but not sufficient condition for good generalization. Moreover, it is possible introducing the extra regularization can be harmful for the overall standard denoising objective.

  3. All the observations and experiments are based on the RHM. It is unclear whether those conclusions hold on the real-world models.

Questions for authors
  1. Is the gap between n_local and n_global a good indicator of generlaization or hullucination? Is it possible the gap is small but both n_local and n_global are large?

  2. How do you ensure the added regularization can prompte generalization? Can you give some theoretical justification? For example, does the global optimizer of the regularized objective still equivalent to the correct posterior-mean?

  3. How can your result generalize beyond RHM? What if none of your observations hold on real-world setting?

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time in evaluating this paper. Since there is only one day left for this stage, could you please read the rebuttal and provide a response? This is important for the review process.

AC

Closing Statement to AC nFyH

We are thankful to the AC and the reviewers for their time and efforts during the rebuttal period. We are particularly grateful that all reviewers recognized the novelty of our work and found it interesting.

Concerns addressed during the rebuttal

  • (NV7t) was primarily concerned about the lack of exposition on the RHM setup. We added a belief-propagation introduction, a worked geometric example (Fig. 8), and an interpretation of nlocaln_{\text{local}} and nglobaln_{\text{global}} (Appendices B, F). NV7t found the answers and additions reasonable and raised their score.
  • (f4Fb) was concerned about the distinction between lag minimization and generalization, and about the theoretical soundness of the regularizer. We agreed that a small lag alone does not imply generalization. Based on their valid suggestion, we also added Proposition E.1, showing that the regularizer preserves the population-optimal denoiser.
  • (5j7r) was concerned about the experimental robustness of the study and the correctness of the illustration in Fig. 1(a). To increase robustness, we increased the number of seeds from 2 to 10 (Figs. 3–6) and added a sweep over the regularization strength λ and a DiT vs. U-Net comparison (Appendix I). Fig. 1(a) was corrected exactly as the reviewer described: two distinct curves, with the memorized fraction starting at zero. Their last comment predates the latest revision, which also includes an extension to an image setting beyond the RHM (Appendix J).

Applicability beyond the RHM

We added the partially controlled image task Quarter-MNIST (Appendix J) to address a concern common to the reviewers: that the study is limited to the RHM. Quarter-MNIST shows that the hierarchical lag and the hallucination window also appear in image data. Extending the study to uncontrolled real-world datasets such as ImageNet is harder, because local and global validity cannot be checked exactly there. We acknowledge this limitation and consider it an important direction for future work.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.