- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The work proposes thinking about images as continous location-to-pixel-value fields, rather than as grids of numbers, in the context of diffusion models. Following this view, they show how common inductive biases such as locality and translation equivariance can be seen naturally through field properties. Next, they show that diffusion sampling using local and equivariant denoisers yields reaction-diffusion-like PDEs, provide analytic solution to linear denoisers, and show that non-linearities yield important properties like symmetry breaking. While not missing gaps within its discussed concepts, the paper has very little experimental results and there is very little connection to general trained diffusion models, so the benefit of thinking about models in this manner is not very supported.
Critical strengths
- The paper is clear and well-written
- Moving beyond discrete grids is attractive from a mathematical point of view
Critical weaknesses
- Critical lack of experiments, with the only experimental results in Figure 1
- Lack of connection and insights derived for diffusion models trained in practice, which limits the utility of the work's proposed theoretical framework
- Did not cite "How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Models" Luca Ambrogioni https://arxiv.org/pdf/2603.20092, which has very similar contributions regarding pattern formation.
Questions for authors
- Could the authors include more experimental results, particularly using strong public diffusion models e.g. Flux, to their proposed framework?
- Could the authors include any specific insights gained about diffusion models trained in practice that are particularly well-framed by their proposed framework? In general, I think a more solid empirical anchor with diffusion models trained in practice would greatly strengthen the work.
5 comments
Rebuttal (part 2)
(continued)
-
Trained CNN-based denoisers support spatial pattern formation at small noise scales. To check whether denoisers at small noise scales (with the aforementioned spatial instabilities) support pattern formation, we considered an interesting twist on sampling: instead of doing sampling as usual, we considered fixed-noise-scale denoisers and performed sampling using Langevin dynamics. This is useful because it allows us to attribute pattern-forming behavior to a denoiser at a fixed noise scale rather than to a more complex interaction between denoisers at different noise scales. We did this for the aforementioned FFHQ32 and CIFAR-10 models (which we trained), as well as for a pretrained ImageNet-64 (EDM2) denoiser (whose quality is a bit better). All models supported the development of high-contrast stripes, and we found that this was related to spatial instability structure. This yields pictures reminiscent of our Fig. 3, but which come from realistic, trained diffusion models.
-
Trained text-to-image models exhibit spatial pattern formation behavior for certain prompts at low noise scales. What about bigger, more 'realistic' diffusion models? Do they look like they do spatial pattern formation at low noise scales? We suspect that this should not always be true; rather, it should depend on whether the relevant data (+ prompt) has natural spatial structure. Motivated by this idea, we looked at two publicly available models: GPIC JiT (1.12B parameters) and FLUX.2 klein base (3.88B parameters). We repeated the fixed-noise Langevin experiment for prompts related to natural spatial structure, like "White clouds in a blue sky." and "A close-up of leopard fur with black spots."; we wanted to see if structure like stripes emerged here too. We indeed found nontrivial spatial pattern formation dynamics, including the formation of spots, stripes, and hexagonal latices, which depended on initialization much like conventional reaction-diffusion (i.e., dynamics can manipulate and amplify existing structure, but it's harder to create totally new structure). In FLUX, approximately uniform initial states with constrained latent-channel means can develop spatial patterns whose characteristic scales depend on noise level, sometimes exhibiting initial amplification of intermediate spatial frequencies followed by coarsening-like behavior. These experiments demonstrate that spatial pattern formation is relevant to diffusion models relevant beyond small CNNs and analytically constructed examples.
-
Trained denoisers can support spatial pattern formation, but the Ginzburg-Landau model appears to be a bad description. Another way we tried getting more quantitative was to fit the Ginzburg-Landau (GL) model, one of the models we discussed in Sec. 6, to trained denoisers at low noise scales. The GL model's distinguishing feature is that it is local and assumes a specific low-order polynomial form for the denoiser's nonlinearity. Unfortunately, we found that the FFHQ32 and CIFAR-10 models we trained are not well described by the GL model---not because they aren't local (they are local according to our experiments above), but because the assumed nonlinearity doesn't fit well. We have moved toward thinking that the examples in Sec. 6 are more 'proofs of principle' rather than anything else, and that while they capture qualitative features of spatial pattern formation in real diffusion models, real diffusion models are unlikely to look exactly like them except in special cases. We are working to do additional fits to say more about real denoisers' nonlinearities. Our analysis so far suggests that the nonlinearities learned by real denoisers are more complex; for example, they can look like the gradients of more heavy-tailed (rather than polynomial) potential functions.
Overall, we are much more confident that:
- Denoisers trained on distributions with spatial structure (like natural images) can be local and support pattern formation (e.g., stripes, spots, lattices) at low noise scales. (Examples: FFHQ32, CIFAR-10, ImageNet-64, GPIC JiT, and FLUX.2 klein base)
- This pattern formation behavior can be directly linked to spatial instability, as our framework suggests.
Rebuttal (part 3)
(continued)
Other concerns
Did not cite "How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Models" Luca Ambrogioni https://arxiv.org/pdf/2603.20092, which has very similar contributions regarding pattern formation.
Thank you for the pointer to this extremely interesting paper. We found it after this paper was already mostly written; it was a complete oversight on our part that we forgot to cite it! We agree that there's a lot of parallels with our work, and we think that our work is complementary to Ambrogioni's. He explains a specific phenomenon (spatial pattern formation) as being related to spontaneous symmetry breaking and the phenomenon of low-frequency modes "softening" around a critical noise scale. While he works within the standard discrete diffusion model framework, to get his main results he uses approximations that are essentially equivalent to working in our 'field-theoretic' regime. We think it would be very interesting to rederive some of his results in terms of our formalism, since we think it provides a natural language for discussing the issues he discusses in that paper.
In any case, we think that his work provides good motivation for ours. Ours focuses more on fleshing out the formalism, while his shows what you can do with it.
Rebuttal (part 1)
Thank you for your careful reading of our paper and for your helpful comments. We are certain that engaging with them has greatly improved our work, and we hope that the rebuttal below addresses your concerns; if it does not, please let us know, and we will do our best to improve things further!
On predictions and experimental depth
We agree that a major weakness of the submitted version is that there is much more focus on the fields and pattern formation formalism than on that formalism's predictions and related experimental results. While we think that this formalism is worth developing and presenting in some detail---a nontrivial task given some of the mathematics involved!---we agree that focusing mostly on the formalism leaves the reader with obvious questions. Does sampling with real diffusion models really look like spatial pattern formation? If so, when, and what kind of guidance does our theory provide?
We are happy to report that we have greatly improved the connection between our formalism and real diffusion models via extensive new experiments. We will include many of these in the updated PDF (albeit in an unpolished form given the limited rebuttal time). Here are some highlights / key points:
Trained CNN-based denoisers are approximately local for small noise scales. Fig. 1 of our paper showed that CNN-based (UNet) denoisers are approximately linear early in training, but become considerably less local later. We wanted to see if these denoisers still look approximately local after they have been trained, since this implies that sampling in this regime looks like spatial pattern formation. We trained UNets on FFHQ32 and CIFAR-10 and fit several simple functions to the learned denoisers. We found that although the learned denoisers are definitely not linear, they are fairly local (looking at a 7x7 window is good enough to explain their behavior). As a side note, they didn't seem to be well-described as the gradient of an energy function.
Spatial stability analysis reveals spatial instabilities in trained CNN-based models. One claim of our framework is that linear stability analysis, including dispersion relations, can help characterize which spatial structures are amplified or suppressed during sampling. Motivated by this idea, we examined the Jacobians of denoisers trained on FFHQ32 and CIFAR-10 and the associated dispersion relations. We found positive growth rates (i.e., unstable directions) across a wide range of noise levels, something inconsistent with a simple linear/Gaussian approximation. These unstable directions did not correspond to single Fourier modes, but combinations of them, which suggests sensitivity to nontrivial spatial perturbations. Which spatial structures were most amplified changed as a function of noise scale; at lower noise levels, edges and fine details were amplified. These instabilities suggest pattern-forming behavior (i.e., what gets most amplified determines most of the final pattern), which we examined more closely in the next bullet point.