acceptodds
Under review as a conference paper at ICLR 2027

How Does Denoising Guidance Encourage Creativity in Diffusion Models?

Abstract

Diffusion models can generate images that go beyond the examples seen during training, but where does this creativity come from? We show that an important part of it can emerge in the *tail of denoising* with denoising guidance. Classifier-free guidance after substantial object structure has formed can preserve pose and layout while guiding the semantic content of generated images. We analyze the tail of deterministic DDIM sampling and show that its local geometry is governed by a *conditional PCA* whose dominant directions depend jointly on the intermediate denoising state and the guidance class. Iterative denoising amplifies these directions through a power-iteration mechanism. We test these predictions using only post-training analysis of a pretrained DiT model trained on ImageNet, and observe the predicted class- and image-dependent geometry. Our results identify a simple geometric mechanism through which guidance can encourage compositional creativity in diffusion models.

Primary area:
generative models
Keywords:
Creativity, Diffusion Models, Guidance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Kw6SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how denoising guidance contributes to compositional creativity in diffusion models. The authors analyze late-stage deterministic DDIM sampling and connect denoising dynamics to conditional PCA and power iteration on the denoiser Jacobian. Experiments on a pretrained ImageNet DiT model are presented to illustrate how switching class conditions during denoising can alter semantic content while preserving parts of the intermediate image structure.

Critical strengths

The paper investigates an interesting question regarding the relationship between denoising dynamics and compositional creativity. Connecting local denoiser geometry with class-dependent semantic adaptation offers a potentially useful perspective for understanding diffusion models.

Critical weaknesses
  1. A gap between the theoretical analysis and the claimed guidance mechanism. The paper motivates its contribution through classifier-free guidance and creative generation, but the central conditional PCA analysis is derived for the ideal denoiser with guidance weight w=0w=0. It is therefore unclear whether the proposed theoretical interpretation extends to classifier-free guidance with nonzero guidance scales. Moreover, the power-iteration analysis relies on a local approximation whose validity depends on the Jacobian remaining sufficiently stable along the trajectory. The paper should clarify these assumptions and provide stronger evidence that the theoretical conclusions apply to the actual guided sampling process.

  2. Insufficient comparisons with existing methods and analytical baselines. Although the paper includes comparisons between standard and time-frozen denoising, it lacks systematic comparisons with existing guidance-switching and compositional generation approaches. Prior methods investigating condition switching and time-dependent semantic control are discussed, but their behaviors are not directly compared under consistent experimental settings. Additional comparisons with fixed switching schedules, different guidance strengths, and existing concept-switching methods would help establish whether the observed behavior represents a distinctive mechanism rather than a previously known consequence of conditional denoising.

  3. Insufficient quantitative evaluation of the central creativity claim. The experimental evidence relies heavily on visual examples, Jacobian eigendirections, projection measurements, and a limited ResNet-50 classification experiment. These analyses provide some support for the proposed geometric interpretation but do not directly establish that guidance produces more creative or compositionally novel images. In particular, classification accuracy only measures whether the generated image resembles the target class; it does not quantify whether semantic information from the original class is preserved or meaningfully combined with the new class. More direct metrics for compositional novelty, structural preservation, semantic integration, and human-perceived creativity are needed.

  4. Limited experimental generalizability. The empirical investigation is primarily conducted on a single ImageNet-trained DiT model, with several analyses focusing on a small number of class combinations, such as dogs and eagles. It remains unclear whether the identified conditional PCA and power-iteration behavior generalizes to modern text-conditioned diffusion models, different architectures, or more complex multi-concept compositions. Furthermore, the paper does not sufficiently analyze how switching time, guidance strength, and semantic similarity between source and target classes affect the proposed mechanism.

Questions for authors

See weakness

4 comments

Thank you very much for your time reviewing the submission. To address your concerns, we uploaded a revision with changes highlighted in dark blue.

Insufficient quantitative evaluation of the central creativity claim

We provided a quantitate definition of creativity with experiments on 1000 generated images from 61 image classes of Image Net in the revised Section 2.

In particular, classification accuracy only measures whether the generated image resembles the target class; it does not quantify whether semantic information from the original class is preserved or meaningfully combined with the new class.

The preserved part from the original class (source class) is term yy which is obtained by denoising guidance of the source class. Our theoretical result in line 349 (of the revised pdf) are surprising: the emerging creativity are obtained by summing up a noisy image from one class a PCA from another class. These images are can be classified as target class TT with nearest neighbor from source class SS; see Table 1.

More direct metrics for compositional novelty, structural preservation, semantic integration, and human-perceived creativity are needed.

Please check the proposed metric in the revised Section 2 and let us know if this metric is convincing enough for blending two image classes. Note that there is no consensus on the notion of creativity. Here, we are studying specific notion of creativity: generating images classified as a target image class TT with nearest neighbor training image from the source class SS. Our motivation is that switching guidance can generate such images as shown in Figure 2 (and Table 1).

Limited experimental generalizability.

Our theoretical analysis shows that the bound holds for any model satisfying the Lipschitz conditions in Appendix C.1 and C.2. Although the analysis is motivated by our experiments, it goes beyond them: it gives sufficient conditions under which the same result holds, so it is not tied to the specific models we tested.

Prior work is often model-specific as well. For example, Kadkhodaie et al. (2024), an ICLR 2024 award-winning paper, and Bonnaire et al. (2025), a NeurIPS 2025 best paper, each focus on a single model setting. Thus, studying a single model in depth is well accepted by the community, as the two works with this limitations awarded best paper awards in ICLR and NeurIPS. Our work offers this depth and, in addition, a general analysis that extends beyond the models we tested.

Our added more experimental results on 1000 images from 61 image classes in the revised Section 2. We analyzed the semantic similarity between source and target in Section 6, switching time is the focus of our analysis as we discussed in Proposition 4.1.

A gap between the theoretical analysis and the claimed guidance mechanism.

We analyzed the guidance switch mechansim proposed in prior studies (Liew et al., 2022; Choi et al., 2024): It starts from denoising toward source class S and the switch to the target class T. Our question is why this specific denoising mechansim generates images from class T whose closest training images from S (see Figure 2). Note that w≠0w \neq 0 is often used for improving FID score and guide samples towards a single image class. What we argue here is that we can mix images from different classes with the guidance switch mechansim. The goal of original classifier-free guidance paper was not mixing images from different classes. We added results in Table 1 confirming that without switching denoising does not combine different image classes.

Insufficient comparisons with existing methods and analytical baselines.

We are not proposing a novel method, we are analyzing the guidance switch mechansim and explain how it generates images that combines elements from two different image classes. The switch mechansim is studied in prior works (Liew et al., 2022; Choi et al., 2024). Our claim is not that the guidance switch is the best or a better method to encourage creativity. Here, we explain how it works and generate images combining two classes. We provide extensive comparisons to related works studying the origin of generalization and creativity in diffusion models in Section 5.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

I appreciate the authors' addition of quantitative experiments and Table 1, which I have carefully reviewed. However, the theoretical analysis still mainly revolves around w=0, and the stability of CFG and Jacobian is not fully explained; the comparison with existing switching methods is also too interpretive and lacks a systematic approach; more importantly, I haven't seen many more combinatorial innovation results like those in Fig. 6 in the updated PDF. My core concerns remain, so I apologize, but I still prefer to retain the original rating.

We have extensively revised the manuscript to address your concerns. In particular, Figures 4-5 in the main text and Figures 8–12 in the appendix present several new experiments. We hope you will reconsider your evaluation in light of these updates. Thank you.

Reviewer_KbjjOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper explains the creative process of diffusion models. The idea is to stitch two different denoisers from different classes together. The evaluation seems to be focused on visual analysis of some example images (especially the bald eagle & maltese dog pair) and lacks a clear explanation as well as a numeric / qualitative core. ???

Suggestion: Maybe mark (start and stop) of the “tail of the denoising process” in Fig. 1. / what about the other areas? I thing there are names for the phase in the literature. Terms like “compositional creativity” are introduced but not used in the rest of the paper.

Critical strengths
  • The paper is well written and easy to follow
Critical weaknesses
  • I am not really sure what is the novelty here. As far as I can see the paper uses conditioning and switches the conditioning to several possible classes mid-generation.
  • It is already well known that in the first few steps of the diffusion process the general image composition is defined, while the later steps add image details.
  • I am not sure what the benefit of using the Jacobian instead of "normal" denoising is.
  • It is unclear how exactly arrive at the frozen intermediate image such that guiding using the Jacobian really works.
Questions for authors

Q1: What is the benefit of using the Jacobian? Could you briefly explain this?
Q2: How do we arrive at the intermediate image? Can we just guide using a random class?

5 comments

Thank you very much for your time reviewing the submission. To address your concerns, we uploaded a revision with changes highlighted in dark blue.

The evaluation seems to be focused on visual analysis of some example images (especially the bald eagle & maltese dog pair) and lacks a clear explanation as well as a numeric / qualitative core. Maybe mark (start and stop) of the “tail of the denoising process” in Fig. 1. / what about the other areas? I thing there are names for the phase in the literature. Terms like “compositional creativity” are introduced but not used in the rest of the paper.

We defined the notion of compositional creativity in the revised Section 2 and added quantitive results for 1000 generated from 61 image classes in Table 1 of the revised Section 2. We also linked observations to our analysis in this section.

The idea is to stitch two different denoisers from different classes together. I am not really sure what is the novelty here. As far as I can see the paper uses conditioning and switches the conditioning to several possible classes mid-generation.

The idea of denoising switch guidance is not from us and it is previously used in (Liew et al., 2022; Choi et al., 2024). Here, we explain how this mechansim can blend images from two classes: the output is decomposed to one term that only depends on the source class SS, another term that only depends on the target image class TT and an error term controlled by the switch time. We provided more details in Section 2 of the revised pdf.

It is already well known that in the first few steps of the diffusion process the general image composition is defined, while the later steps add image details.

The novelty here is explaining how by changing the guidance we can generate images that are resembling two image classes at the same time: we can control details with the guidance switch mechansim and that can effectively change the image class. The idea is inspired by observation in Figure 2 where we observed that closest images of generated with guidance switch are different from the image class. For example, the closest training image of the first generated eagle is a dog (not an eagle). We provided quantitive results in Table 1 confirming that denoising switch is capable of such blending images. Our main contribution is the establish analyze of this observation as discussed in the revised Section 2.

I am not sure what the benefit of using the Jacobian instead of "normal" denoising is. Q1: What is the benefit of using the Jacobian? Could you briefly explain this?

Denoising Jacobian is a tool to analyze the denoising mechansim. Consider multiple steps denoising with a non-linear neural network. Due to the non-linearity of input-output of neural networks, it is very difficult to study multiple denoising steps. Even a single step is not easy to analyze. What is significant about our analysis (in Section 4) is showing the highly non-linear composition of denoising steps can be well approximated with a linear power iteration on the denoising Jacobean. The Jacobean has a very nice link to a covariance matrix obtained by reweighing samples within a class as discussed in Section 5.

It is unclear how exactly arrive at the frozen intermediate image such that guiding using the Jacobian really works. Q2: How do we arrive at the intermediate image?

The intermediate image is obtained by denoising guidance conditioned on a source image class SS. Since this intermediate is obtained by denoising with SS, we expect it mainly resembles images from this class. When we switch the guidance to image class target TT, then additional denoising steps add a PCA term to this image that obtained by a covariance matrix conditioned on TT. The combined terms create images classified as TT with nearest neighbor in SS (see the revised Section 2).

Can we just guide using a random class?

Yes, we can guide using any source image class to start denoising, yet the target class may depends on the source. We discussed conditions when this switch leads to creativity in Section 6.

I thank the authors for their detailed answer.

While some of my concerns have been appropriately addressed, I am still unsure about the novelty, or rather the new insight, provided by the new paper. I understand that switching to different classes during generation is defined as creative in this work. However, I am not sure why the analysis is a novel insight. What follows from this analysis, and what is different from previous work that has proposed switching mechanisms?
In the contribution section, it is written, "We test these insights on the pretrained DiT model of Peebles & Xie (2023), trained on ImageNet (Deng et al., 2009), and show experimentally that our approximation accurately captures the tail denoising behavior of large-scale pretrained models." Could you clarify why the insight that we can approximate the denoising behavior is useful?

AuthorsOpenReview ↗

Author response

Thank you very much for considering our response.

I am still unsure about the novelty, or rather the new insight, provided by the new paper. Could you clarify why the insight that we can approximate the denoising behavior is useful?

Related literature on generalization in diffusion models provide new insights (not a new method) including the best paper of ICLR 2023 (Kadkhodaie et al., 2024) and the best paper of NeurIPS 2025 (Bonnaire et al., 2025). These works do not propose a new denoising method nor they provide a new generative methods. The first paper shows that diffusion models learn adaptive bases for denoising and the 2nd shows that early stopping contribute to generalization. While these papers do not design a new algorithm, nor enhance benchmarks, they are recognized as best papers of two leading machine learning conferences and advance the knowledge of generative models. This is due to the fact that optimal generative models are known to only regenerated training samples (Kamb & Ganguli, 2025); indeed, we do not know exactly how diffusion model generate images different from their training samples.

Akin to the above seminal works, we also provide a new complementary insights on how diffusion create images different from training samples. Our novelty compared to these works is showing denoising guidance mechansim can generate images whose closest neighbor are from a source image class S, while they are classified in the target image class T (see Figure 2 and Table 1). Why this is important? This shows images combines components from different image classes, thus it is different from training samples. Then we analyze the underline mechansim of blending two image classes. How diffusion models combine components from two different images? One may think that this process is very complicated, but we show is a simple linear combination of two images: a noisy image from the source class + a principal component of reweighed samples from the target class + an approximate term vanishing with the switch time.

However, I am not sure why the analysis is a novel insight. What follows from this analysis, and what is different from previous work that has proposed switching mechanisms?

The proposed work only use denoising switch for fairness adjustment and downstream applications. These prior works aim at using guidance mechansim rather than studying its impact on generalization and creativity. Here, we explain how denoising switch can encourage creative image generation by a additive image decomposition: a noisy image + a PCA can approximate the mechansim of denoising switch.

The novelty of our work can be assessed in the light of prior works on generalization of diffusion model showing that generalization emerges in large scale learning (Kadkhodaie et al., 2024), with early stopping (Bonnaire et al., 2025), and locality of CNN filters (Kamb & Ganguli, 2025). Here, we prove that the guidance mechansim significantly contributes to generalization and creative image generation.

Reviewer_RY5hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how changing the conditioning class during diffusion sampling can alter semantic content while retaining aspects of an intermediate image’s spatial structure. It proposes an interpretation of late-stage denoising through power iteration on the denoiser Jacobian, connected to conditional PCA, and illustrates this interpretation using a pretrained DiT model.

Critical strengths

The central observation is interesting and intuitive: an intermediate image can retain spatial structure while subsequent conditioning changes how that structure is completed.

Critical weaknesses
  1. The notion of creativity needs a more precise definition and evaluation. Although the paper describes its focus as compositional creativity, I remain unsure what operational criterion distinguishes a creative output from an image that simply changes or blends semantic content. It is also unclear how the degree of creativity should be quantified. Since creativity is central to the paper’s motivation and claims, this ambiguity makes it difficult to assess how strongly the presented evidence supports the conclusions.

  2. The mathematical explanation is not sufficiently clear to me. I find the qualitative observation intuitive, but the connection between the mathematical analysis and the claimed creative behavior is harder to follow. In particular, I would appreciate a clearer account of what the power-iteration and conditional-PCA interpretation establishes about that behavior, and which parts remain intuitive interpretations. As presented, the explanation feels more heuristic than fully justified to me. My concern is about the explanatory link and its scope; I am not identifying a specific error in the proofs.

Questions for authors
  1. How should compositional creativity be operationally defined and measured in this setting? Which aspects of the presented evaluation directly assess this definition?

  2. Could the authors clarify the precise connection between the mathematical results and the observed creative behavior, distinguishing the conclusions established by the analysis from the interpretations suggested by the examples?

3 comments

Thank you very much for your time reviewing the submission. To address your concerns, we uploaded a revision with changes highlighted in dark blue.

The notion of creativity needs a more precise definition and evaluation.

We added a definition of creativity to Section 2 of the revised submission, together with a quantitative creativity score. An image is considered creative if it is classified as the target class TT while its nearest-neighbor training sample belongs to the source class SS. The nearest-neighbor condition ensures that the overall shape of the image (the canvas) comes from one class, while the details that drive classification (the brush) come from another, so the image is a blend of the two classes. We also added experimental results on 1000 generated images, which confirm that the denoising switch is capable of producing such images.

The mathematical explanation is not sufficiently clear to me.

We added a paragraph on the theory, together with an outline, to the revised Section 2. It explains how the denoising switch creates images that combine two different classes: the denoising output decomposes into a component from the source class SS, a component from the target class TT, and an error term controlled by the switch time. We are happy to clarify any remaining ambiguities during the discussion period.

Q.1: How should compositional creativity be operationally defined and measured in this setting? Which aspects of the presented evaluation directly assess this definition?

The definition is inspired by the observation in Figure 2 where generated image classes are different from the class of their nearest neighbor image in the training set. Following your advice, we formulate the notion mathematically and evaluated on 1000 generated images in Table 1.

Q.2: Could the authors clarify the precise connection between the mathematical results and the observed creative behavior, distinguishing the conclusions established by the analysis from the interpretations suggested by the examples?

We added a paragraph on "theory" in Section 2 to address this concern.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

We have extensively revised the manuscript to address your concerns. In particular, Figures 4-5 in the main text and Figures 8–12 in the appendix present several new experiments. We hope you will reconsider your evaluation in light of these updates. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.