acceptodds
Under review as a conference paper at ICLR 2027

Correcting Diffusion Hallucinations via Uncertainty-guided Trajectory Steering

Abstract

Diffusion models have emerged as state-of-the-art image generative models, yet they still suffer from *hallucinations*, i.e., unrealistic artifacts or implausible structures absent from the training data distribution. Prior work attributes this issue to *mode interpolation*, where the model generates samples that lie between nearby data modes but fall outside the actual data support. Motivated by this perspective, we propose a lightweight image editing method for correcting hallucinations in diffusion-generated images. The key idea is to steer the sampling trajectory along directions where the diffusion model’s posterior mean estimator exhibits the highest uncertainty, thereby pushing the generation process away from unsupported, uncertain regions and toward nearby realistic modes, while largely preserving the original semantic content. Notably, our method requires neither backpropagation nor external supervision. Experiments across multiple datasets show that it consistently reduces hallucinations while remaining faithful to the original image, achieving a more favorable validity–fidelity trade-off than existing techniques.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion model, hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G1SrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Uncertainty-guided Trajectory Steering (UTS), a training-free method for correcting hallucinations in diffusion-generated images. The method estimates uncertainty in the posterior mean predictor using multiple noisy samples and identifies steering directions through PCA. These directions are then used to steer the denoising trajectory toward nearby realistic modes. The evaluation is conducted on MNIST, synthetic shapes, hands, and a limited text-to-image setting.

Critical strengths
  1. Clear presentation. The paper is well written and easy to follow, with an intuitive motivation and a straightforward algorithm.

  2. Practical applicability. The proposed method is training-free and does not require backpropagation or external supervision. Its simplicity and ease of implementation potentially offer practical value for correcting hallucinations in pretrained diffusion models.

Critical weaknesses
  1. Insufficient justification of the uncertainty formulation and PCA-based steering. The paper attributes uncertainty to two fundamentally different sources: mode interpolation and the denoising process. However, this distinction is neither sufficiently justified nor theoretically analyzed, and heavily corrupted inputs do not necessarily lead to high-variance outputs. Also, the PCA-based mode directions may not be suitable for resolving mode interpolation when clean predictions exhibit a superposition of multiple plausible modes (e.g., duplicated fingers or overlapping motions), as can occur in practice.

  2. Limited experimental evaluation and comparison with related methods. The experimental evaluation is primarily conducted on small-scale datasets, with only a limited text-to-image setting. More comprehensive experiments on modern text-to-image or text-to-video diffusion models are needed to demonstrate practical applicability. Furthermore, the proposed method shares similar motivations and approaches with existing inference-time guidance methods, particularly Tangential Amplifying Guidance (Cho et al., ICML 2026). Without comparisons against closely related methods, it is difficult to assess the novelty and relative effectiveness of the proposed approach.

Questions for authors
  1. Uncertainty formulation. What theoretical or empirical evidence supports separating uncertainty into mode interpolation and the intrinsic denoising process? In particular, why should heavily corrupted inputs necessarily produce higher variance in the posterior mean predictions?

  2. Effectiveness of PCA-based mode directions. When a intermediate, clean x0 prediction exhibits a superposition of multiple plausible modes (e.g., duplicated fingers or overlapping motions, as discussed in DiffusionBrowser (Hong et al., 2025)), can the leading principal component effectively identify the competing modes and guide the prediction toward a single realistic mode? Could the authors provide additional analysis or experiments on such cases?

  3. Comparison with related guidance methods. How does the proposed method differ from Tangential Amplifying Guidance (Cho et al., ICML 2026), which shares similar motivations and approaches? How does UTS compare empirically with other inference-time unconditional guidance methods, such as SAG (Hong et al., ICCV 2023), PAG (Ahn et al., ECCV 2024), ICG (Sadat et al., ICLR 2025), AG (Karras et al., NeurIPS 2024), and SEG (Hong, NeurIPS 2024)? Such comparisons would help clarify the novelty and effectiveness of the proposed method.

  4. Scalability to practical settings. Can the authors provide more comprehensive evaluations on modern text-to-image or text-to-video diffusion models to demonstrate the scalability and practical applicability of the proposed approach?

Reviewer_KrfhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UTS edits generated images using DDIM inversion and trajectory steering. At selected timesteps, it re-noises the current clean estimate, extracts the leading PCA direction of multiple denoiser predictions, and orients the perturbation using the model score. Two-Gaussian analyses motivate mode-direction recovery and a classifier-guidance interpretation. Experiments test validity-fidelity trade-offs on MNIST, rule-based shapes, generated hands, and DeepFloyd-IF images from one handshake prompt.

Critical strengths
  1. The method makes a specific, useful methodological move: estimating uncertainty across re-noised predictions at a fixed timestep, then using its dominant direction for score-oriented correction. This goes beyond diagnosing mode interpolation [R1] or estimating denoiser uncertainty [R2]. It does not require the externally specified modes/classifier used by Dynamic Guidance [R3]. DI and equal-norm randomized steering provide relevant controls showing that inversion or arbitrary perturbation alone does not explain the reported gains.

  2. The evaluation is more substantial than selected correction examples. Table 1 shows strong rule-based shape validity and higher hypervolume on three of four datasets. Figure 5 studies steering windows, strength, and PCA sample count. Appendices D.5-D.8 report runtime, higher-PC visualizations, diversity, and prevention with additional samplers. These studies support a working correction mechanism in controlled settings and help identify its computational and diversity trade-offs.

[R1] Aithal, Maini, Lipton, and Kolter (NeurIPS 2024). Understanding Hallucinations in Diffusion Models through Mode Interpolation.

[R2] Manor and Michaeli (ICLR 2024; preprint 2023). On the Posterior Distribution in Denoising: Application to Uncertainty Quantification.

[R3] Triaridis et al. (2025 preprint; revised June 2026). Mitigating Diffusion Model Hallucinations with Dynamic Guidance.

Critical weaknesses
  1. The central real-image hallucination-reduction claim lacks a validated outcome measure. Hands and Shaking Hands both select examples and score edits with MediaPipe detection confidence (Section 5; D.1), without independently establishing that higher confidence means fewer implausible structures or malformed fingers. A hand-tracking score [R4] may improve with pose or detectability without correcting the claimed error. Reusing the proxy is not itself invalid and UTS does not directly optimize it; the issue is unverified construct validity. Qualitative examples and exact shape rules do not close this gap, particularly for the single-prompt text-to-image evaluation.

  2. The claimed correction-fidelity advantage is not yet fully auditable. Table 1 averages scores over method-specific hyperparameter grids, so those means are not matched operating points. Hypervolume can address trade-offs, but its normalization/reference point is unspecified; D.1 defines shape fidelity by preserving shape types whereas Table 1 labels fidelity as LPIPS. High shape validity should not be conflated with the best trade-off: its hypervolume is slightly below DG's. Formula inconsistencies also require correction, but are secondary: repairing the two-Gaussian theorem would not validate real-image metrics or establish beneficial corrections at comparable fidelity.

[R4] Zhang et al. (2020). MediaPipe Hands: On-device Real-time Hand Tracking.

Questions for authors
  1. Which MediaPipe output and missed-/multiple-hand aggregation are used? Can an independently labeled subset validate this proxy and demonstrate fewer implausible structures at matched fidelity, rather than only improved detectability?

  2. Please define the per-image MNIST score, shape fidelity, and hypervolume normalization/reference point, and provide the actual trade-off points. How were DG's mode definitions/classifier or text branches implemented, and are conclusions stable under comparable tuning budgets?

  3. Which index was used in the square-root diffusion coefficient: t-1 in Algorithm 1 or t in the accompanying text? Please reconcile Theorem 4.1 with its proof, including the (1-P) denominator factor and probability convention, and supply the fixed-estimate conditional-PCA argument with the finite-perturbation approximation stated explicitly.

Reviewer_n6V5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper summary: This paper proposes a method for correcting diffusion hallucination. The overall idea is based on an assumption: hallucinated samples typically lie between different modes of the real data. The introduced theory proposes an updating direction towards the data mode, and by updating the diffusion denoising trajectory of each timestep towards that direction, the method can correct the hallucinated samples. Their method is training-free and can be injected into the inference of any pretrained diffusion model.

Paper contributions:

  1. This paper proposes UTS, a training-free method that can be injected into the inference time of a diffusion model to correct the hallucinated samples. UTS is based on the mode interpolation assumption.
  2. The experiments show that UTS achieves consistent improvement over the baselines.
Critical strengths

Paper strength:

  1. Simple and well-motivated framework: The method is based on a mode interpolation assumption, and the approach is intuitive: find the direction to the data mode. The proposed theory also supports this idea. 2 Experiments: Experiments somehow show the effect of the proposed framework with the baselines.
Critical weaknesses

There are several weaknesses in this paper:

  1. Evaluation protocol: This paper chooses samples with the lowest scores from MNIST and Hands to evaluate the method. The validity metric is the same as the metric used to choose the samples. I feel this evaluation protocol is not a good protocol, since the worst samples can improve their scores after editing. Also, the improvement of the metric on the worst samples does not indicate that the hallucination is fixed. I suggest conducting a survey for human judgment as another metric to verify whether the hallucinated samples are fixed.
  2. Minor improvement for Shaking Hands: In Table 1, compared to baseline DI, most of the gains (Valid score) are from simpler datasets like MNIST, Shapes, and Hands, while the improvement on Shaking Hands is minor, which might suggest UTS does not improve much on complex datasets.
  3. Not good Fid: In Table 1, the Fid results of UTS is not as good compared to other baselines.
  4. Overstate text-to-image benchmark: The introduction states the evaluation on a text-to-image benchmark; however, the text-to-image dataset only contains a single prompt: "Shaking Hands" for the entire set, which overstates the evaluation scope of this paper.
  5. Missing baselines: Together with DDIM inversion, SD Edit [1] shares a very close denoising mechanism with this work. SD Edit predicts x0x_0 from xtx_{t}, and renoise to get xt−1x_{t-1}. This can be a "no target direction" compared with UTS, since we do not need to compute the δ\delta update direction in SDEdit.
  6. Mode interpolation assumption: The reason the direction δ\delta is proposed is based on the theoretical results of the Gaussian mixture with two modes. However, for complicated data, this assumption does not hold. Therefore, choosing the direction δ\delta as the highest principal component might not guarantee a correct update direction.

[1] https://arxiv.org/abs/2108.01073

Questions for authors

See weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.