acceptodds
Under review as a conference paper at ICLR 2027

Field-theoretic diffusion models and sampling as spatial pattern formation

Abstract

Although images can be viewed as large vectors, such a representation fails to exploit nontrivial spatial structure: for example, nearby pixel locations tend to have similar colors and intensities. Similarly, theoretical analyses of image-based diffusion models tend to portray the input of denoisers and score functions as a grid of numbers rather than as a continuous object. In this work, we argue that the sampling dynamics of image-based diffusion models (and related concepts, like locality and translation equivariance) are usefully understood in terms of an idealization in which space is treated as continuous, which makes them a kind of spatial pattern formation process. To make this perspective concrete, we introduce and formalize field-theoretic diffusion models, whose central objects are fields: mappings from a real-valued pixel space to a real-valued color space. We consider field-theoretic diffusion models interesting both because they model the structure of architectures with strong spatial inductive biases, like convolutional neural networks, and because they provide a bridge between diffusion model theory and the theory of spatial pattern formation. We present theoretical results concerning sampling, including exact solutions for linear model sampling dynamics and brief results on the sampling dynamics of specific nonlinear models. The continuum description yields a novel picture of sampling in terms of reaction-diffusion-like dynamics, where different 'chemicals' drift, diffuse, and interact. We illustrate this intuition using a variety of examples. Our work demonstrates the theoretical benefits of reasoning about sampling via a continuous-space description.

Primary area:
generative models
Keywords:
diffusion, generative models, theory, linear models, image models, sampling dynamics, reaction-diffusion, pattern formation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yzRpOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The work proposes thinking about images as continous location-to-pixel-value fields, rather than as grids of numbers, in the context of diffusion models. Following this view, they show how common inductive biases such as locality and translation equivariance can be seen naturally through field properties. Next, they show that diffusion sampling using local and equivariant denoisers yields reaction-diffusion-like PDEs, provide analytic solution to linear denoisers, and show that non-linearities yield important properties like symmetry breaking. While not missing gaps within its discussed concepts, the paper has very little experimental results and there is very little connection to general trained diffusion models, so the benefit of thinking about models in this manner is not very supported.

Critical strengths
  • The paper is clear and well-written
  • Moving beyond discrete grids is attractive from a mathematical point of view
Critical weaknesses
  • Critical lack of experiments, with the only experimental results in Figure 1
  • Lack of connection and insights derived for diffusion models trained in practice, which limits the utility of the work's proposed theoretical framework
  • Did not cite "How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Models" Luca Ambrogioni https://arxiv.org/pdf/2603.20092, which has very similar contributions regarding pattern formation.
Questions for authors
  • Could the authors include more experimental results, particularly using strong public diffusion models e.g. Flux, to their proposed framework?
  • Could the authors include any specific insights gained about diffusion models trained in practice that are particularly well-framed by their proposed framework? In general, I think a more solid empirical anchor with diffusion models trained in practice would greatly strengthen the work.
5 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your careful reading of our paper and for your helpful comments. We are certain that engaging with them has greatly improved our work, and we hope that the rebuttal below addresses your concerns; if it does not, please let us know, and we will do our best to improve things further!

On predictions and experimental depth

Critical lack of experiments, with the only experimental results in Figure 1

Lack of connection and insights derived for diffusion models trained in practice, which limits the utility of the work's proposed theoretical framework

Could the authors include more experimental results, particularly using strong public diffusion models e.g. Flux, to their proposed framework?

Could the authors include any specific insights gained about diffusion models trained in practice that are particularly well-framed by their proposed framework? In general, I think a more solid empirical anchor with diffusion models trained in practice would greatly strengthen the work.

We agree that a major weakness of the submitted version is that there is much more focus on the fields and pattern formation formalism than on that formalism's predictions and related experimental results. While we think that this formalism is worth developing and presenting in some detail---a nontrivial task given some of the mathematics involved!---we agree that focusing mostly on the formalism leaves the reader with obvious questions. Does sampling with real diffusion models really look like spatial pattern formation? If so, when, and what kind of guidance does our theory provide?

We are happy to report that we have greatly improved the connection between our formalism and real diffusion models via extensive new experiments. We will include many of these in the updated PDF (albeit in an unpolished form given the limited rebuttal time). Here are some highlights / key points:

  • Trained CNN-based denoisers are approximately local for small noise scales. Fig. 1 of our paper showed that CNN-based (UNet) denoisers are approximately linear early in training, but become considerably less local later. We wanted to see if these denoisers still look approximately local after they have been trained, since this implies that sampling in this regime looks like spatial pattern formation. We trained UNets on FFHQ32 and CIFAR-10 and fit several simple functions to the learned denoisers. We found that although the learned denoisers are definitely not linear, they are fairly local (looking at a 7x7 window is good enough to explain their behavior). As a side note, they didn't seem to be well-described as the gradient of an energy function.

  • Spatial stability analysis reveals spatial instabilities in trained CNN-based models. One claim of our framework is that linear stability analysis, including dispersion relations, can help characterize which spatial structures are amplified or suppressed during sampling. Motivated by this idea, we examined the Jacobians of denoisers trained on FFHQ32 and CIFAR-10 and the associated dispersion relations. We found positive growth rates (i.e., unstable directions) across a wide range of noise levels, something inconsistent with a simple linear/Gaussian approximation. These unstable directions did not correspond to single Fourier modes, but combinations of them, which suggests sensitivity to nontrivial spatial perturbations. Which spatial structures were most amplified changed as a function of noise scale; at lower noise levels, edges and fine details were amplified. These instabilities suggest pattern-forming behavior (i.e., what gets most amplified determines most of the final pattern), which we examined more closely in the next bullet point.

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

  • Trained CNN-based denoisers support spatial pattern formation at small noise scales. To check whether denoisers at small noise scales (with the aforementioned spatial instabilities) support pattern formation, we considered an interesting twist on sampling: instead of doing sampling as usual, we considered fixed-noise-scale denoisers and performed sampling using Langevin dynamics. This is useful because it allows us to attribute pattern-forming behavior to a denoiser at a fixed noise scale rather than to a more complex interaction between denoisers at different noise scales. We did this for the aforementioned FFHQ32 and CIFAR-10 models (which we trained), as well as for a pretrained ImageNet-64 (EDM2) denoiser (whose quality is a bit better). All models supported the development of high-contrast stripes, and we found that this was related to spatial instability structure. This yields pictures reminiscent of our Fig. 3, but which come from realistic, trained diffusion models.

  • Trained text-to-image models exhibit spatial pattern formation behavior for certain prompts at low noise scales. What about bigger, more 'realistic' diffusion models? Do they look like they do spatial pattern formation at low noise scales? We suspect that this should not always be true; rather, it should depend on whether the relevant data (+ prompt) has natural spatial structure. Motivated by this idea, we looked at two publicly available models: GPIC JiT (1.12B parameters) and FLUX.2 klein base (3.88B parameters). We repeated the fixed-noise Langevin experiment for prompts related to natural spatial structure, like "White clouds in a blue sky." and "A close-up of leopard fur with black spots."; we wanted to see if structure like stripes emerged here too. We indeed found nontrivial spatial pattern formation dynamics, including the formation of spots, stripes, and hexagonal latices, which depended on initialization much like conventional reaction-diffusion (i.e., dynamics can manipulate and amplify existing structure, but it's harder to create totally new structure). In FLUX, approximately uniform initial states with constrained latent-channel means can develop spatial patterns whose characteristic scales depend on noise level, sometimes exhibiting initial amplification of intermediate spatial frequencies followed by coarsening-like behavior. These experiments demonstrate that spatial pattern formation is relevant to diffusion models relevant beyond small CNNs and analytically constructed examples.

  • Trained denoisers can support spatial pattern formation, but the Ginzburg-Landau model appears to be a bad description. Another way we tried getting more quantitative was to fit the Ginzburg-Landau (GL) model, one of the models we discussed in Sec. 6, to trained denoisers at low noise scales. The GL model's distinguishing feature is that it is local and assumes a specific low-order polynomial form for the denoiser's nonlinearity. Unfortunately, we found that the FFHQ32 and CIFAR-10 models we trained are not well described by the GL model---not because they aren't local (they are local according to our experiments above), but because the assumed nonlinearity doesn't fit well. We have moved toward thinking that the examples in Sec. 6 are more 'proofs of principle' rather than anything else, and that while they capture qualitative features of spatial pattern formation in real diffusion models, real diffusion models are unlikely to look exactly like them except in special cases. We are working to do additional fits to say more about real denoisers' nonlinearities. Our analysis so far suggests that the nonlinearities learned by real denoisers are more complex; for example, they can look like the gradients of more heavy-tailed (rather than polynomial) potential functions.

Overall, we are much more confident that:

  1. Denoisers trained on distributions with spatial structure (like natural images) can be local and support pattern formation (e.g., stripes, spots, lattices) at low noise scales. (Examples: FFHQ32, CIFAR-10, ImageNet-64, GPIC JiT, and FLUX.2 klein base)
  2. This pattern formation behavior can be directly linked to spatial instability, as our framework suggests.
AuthorsOpenReview ↗

Rebuttal (part 3)

(continued)

Other concerns

Did not cite "How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Models" Luca Ambrogioni https://arxiv.org/pdf/2603.20092, which has very similar contributions regarding pattern formation.

Thank you for the pointer to this extremely interesting paper. We found it after this paper was already mostly written; it was a complete oversight on our part that we forgot to cite it! We agree that there's a lot of parallels with our work, and we think that our work is complementary to Ambrogioni's. He explains a specific phenomenon (spatial pattern formation) as being related to spontaneous symmetry breaking and the phenomenon of low-frequency modes "softening" around a critical noise scale. While he works within the standard discrete diffusion model framework, to get his main results he uses approximations that are essentially equivalent to working in our 'field-theoretic' regime. We think it would be very interesting to rederive some of his results in terms of our formalism, since we think it provides a natural language for discussing the issues he discusses in that paper.

In any case, we think that his work provides good motivation for ours. Ours focuses more on fleshing out the formalism, while his shows what you can do with it.

Reviewer_VrwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the theoretical perspective of treating images as fields, and what this perspective means for diffusion modelling. The authors discuss what this means for score based denoising, treating the denoiser as a functional. The authors also discuss the concepts of locality, and equivariance, and consider these concepts in real space as well as in wave space. The authors continue to connect to pattern generation in partial differential equations, and show the effects of various derivatives on the behavior of the denoiser. The authors then study linear denoisers that are mostly local and mostly equivariant and calculate their properties. The authors by studying some particular data distributions and the emergence of symmetry breaking and topological defects.

Critical strengths
  • Generally I think this is an interesting perspective to apply field theory to diffusion models, the connection makes a lot of sense.
Critical weaknesses
  • Overall I find the results quite simple and generally obvious. The conversions and integrals are all implicit, and not a lot is derived from the results. It often seems to boil down to something akin to: when a diffusion term is added, we observe diffusion. As an example, in lines 334-336, the authors state that if ksi or H go to zero, then the result goes to that of the previous section, but, this can already be trivially seen from the equation before it is solved. The authors say that they want to provide a "theoretically interesting perspective", which I think requires demonstrating what we can learn from that perspective.
  • The authors involve a lot of physics models, which on the one hand I like (the reviewer is trained as a phycisist), but on the other hand I find the connections often seem superficial, such as the connection to wavespace and the uncertainty principle, as well as field theory. When those concepts were mentioned, I would have hoped to see a stronger connection made in the paper.
Questions for authors
  • The locality in terms of derivatives seems like an odd choice to me. Maybe this is some field theoretic default, but a more natural way to consider locality would be in terms of an auto correlation. Have the authors considered something like that?
  • I'm not sure I agree that locality means giving up translational equivariance. I think a denoiser can be both. Equivariance just means as you translate the input, the output translates too, that is perfectly compatible with locality, which just means that a particular denoiser output depends only on local inputs that are close to the output coordinate. This goes for fields just as well as projections onto a finite distcrete basis (i.e. arrays). Can the authors elaborate? Or am I misunderstanding the claim the authors are making?
3 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your careful reading of our paper and for your helpful comments. We are certain that engaging with them has greatly improved our work, and we hope that the rebuttal below addresses your concerns; if it does not, please let us know, and we will do our best to improve things further!

On predictions and experimental depth

The authors say that they want to provide a "theoretically interesting perspective", which I think requires demonstrating what we can learn from that perspective.

We agree that a major weakness of the submitted version is that there is much more focus on the fields and pattern formation formalism than on that formalism's predictions and related experimental results. While we think that this formalism is worth developing and presenting in some detail---a nontrivial task given some of the mathematics involved!---we agree that focusing mostly on the formalism leaves the reader with obvious questions. Does sampling with real diffusion models really look like spatial pattern formation? If so, when, and what kind of guidance does our theory provide?

We are happy to report that we have greatly improved the connection between our formalism and real diffusion models via extensive new experiments. We will include many of these in the updated PDF (albeit in an unpolished form given the limited rebuttal time). Here are some highlights / key points:

  • Trained CNN-based denoisers are approximately local for small noise scales. Fig. 1 of our paper showed that CNN-based (UNet) denoisers are approximately linear early in training, but become considerably less local later. We wanted to see if these denoisers still look approximately local after they have been trained, since this implies that sampling in this regime looks like spatial pattern formation. We trained UNets on FFHQ32 and CIFAR-10 and fit several simple functions to the learned denoisers. We found that although the learned denoisers are definitely not linear, they are fairly local (looking at a 7x7 window is good enough to explain their behavior). As a side note, they didn't seem to be well-described as the gradient of an energy function.

  • Spatial stability analysis reveals spatial instabilities in trained CNN-based models. One claim of our framework is that linear stability analysis, including dispersion relations, can help characterize which spatial structures are amplified or suppressed during sampling. Motivated by this idea, we examined the Jacobians of denoisers trained on FFHQ32 and CIFAR-10 and the associated dispersion relations. We found positive growth rates (i.e., unstable directions) across a wide range of noise levels, something inconsistent with a simple linear/Gaussian approximation. These unstable directions did not correspond to single Fourier modes, but combinations of them, which suggests sensitivity to nontrivial spatial perturbations. Which spatial structures were most amplified changed as a function of noise scale; at lower noise levels, edges and fine details were amplified. These instabilities suggest pattern-forming behavior (i.e., what gets most amplified determines most of the final pattern), which we examined more closely in the next bullet point.

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

  • Trained CNN-based denoisers support spatial pattern formation at small noise scales. To check whether denoisers at small noise scales (with the aforementioned spatial instabilities) support pattern formation, we considered an interesting twist on sampling: instead of doing sampling as usual, we considered fixed-noise-scale denoisers and performed sampling using Langevin dynamics. This is useful because it allows us to attribute pattern-forming behavior to a denoiser at a fixed noise scale rather than to a more complex interaction between denoisers at different noise scales. We did this for the aforementioned FFHQ32 and CIFAR-10 models (which we trained), as well as for a pretrained ImageNet-64 (EDM2) denoiser (whose quality is a bit better). All models supported the development of high-contrast stripes, and we found that this was related to spatial instability structure. This yields pictures reminiscent of our Fig. 3, but which come from realistic, trained diffusion models.

  • Trained text-to-image models exhibit spatial pattern formation behavior for certain prompts at low noise scales. What about bigger, more 'realistic' diffusion models? Do they look like they do spatial pattern formation at low noise scales? We suspect that this should not always be true; rather, it should depend on whether the relevant data (+ prompt) has natural spatial structure. Motivated by this idea, we looked at two publicly available models: GPIC JiT (1.12B parameters) and FLUX.2 klein base (3.88B parameters). We repeated the fixed-noise Langevin experiment for prompts related to natural spatial structure, like "White clouds in a blue sky." and "A close-up of leopard fur with black spots."; we wanted to see if structure like stripes emerged here too. We indeed found nontrivial spatial pattern formation dynamics, including the formation of spots, stripes, and hexagonal latices, which depended on initialization much like conventional reaction-diffusion (i.e., dynamics can manipulate and amplify existing structure, but it's harder to create totally new structure). In FLUX, approximately uniform initial states with constrained latent-channel means can develop spatial patterns whose characteristic scales depend on noise level, sometimes exhibiting initial amplification of intermediate spatial frequencies followed by coarsening-like behavior. These experiments demonstrate that spatial pattern formation is relevant to diffusion models relevant beyond small CNNs and analytically constructed examples.

  • Trained denoisers can support spatial pattern formation, but the Ginzburg-Landau model appears to be a bad description. Another way we tried getting more quantitative was to fit the Ginzburg-Landau (GL) model, one of the models we discussed in Sec. 6, to trained denoisers at low noise scales. The GL model's distinguishing feature is that it is local and assumes a specific low-order polynomial form for the denoiser's nonlinearity. Unfortunately, we found that the FFHQ32 and CIFAR-10 models we trained are not well described by the GL model---not because they aren't local (they are local according to our experiments above), but because the assumed nonlinearity doesn't fit well. We have moved toward thinking that the examples in Sec. 6 are more 'proofs of principle' rather than anything else, and that while they capture qualitative features of spatial pattern formation in real diffusion models, real diffusion models are unlikely to look exactly like them except in special cases. We are working to do additional fits to say more about real denoisers' nonlinearities. Our analysis so far suggests that the nonlinearities learned by real denoisers are more complex; for example, they can look like the gradients of more heavy-tailed (rather than polynomial) potential functions.

Overall, we are much more confident that:

  1. Denoisers trained on distributions with spatial structure (like natural images) can be local and support pattern formation (e.g., stripes, spots, lattices) at low noise scales. (Examples: FFHQ32, CIFAR-10, ImageNet-64, GPIC JiT, and FLUX.2 klein base)
  2. This pattern formation behavior can be directly linked to spatial instability, as our framework suggests.
AuthorsOpenReview ↗

Rebuttal (part 3)

(continued)

On our notion of locality

The locality in terms of derivatives seems like an odd choice to me. Maybe this is some field theoretic default, but a more natural way to consider locality would be in terms of an auto correlation. Have the authors considered something like that?

You're absolutely right that this is a pretty constrained notion of locality. While it is common in certain areas of theoretical physics (e.g., stuff related to quantum field theory), it is less natural in many diffusion-model-related contexts. It is more common to approximate receptive fields as having a non-maximal (but not necessarily infinitesimal) size; Kamb and Ganguli (https://arxiv.org/abs/2412.20292) do this and Niedoba et al. (https://arxiv.org/abs/2411.19339) do this.

We do not claim that our derivative-based notion is the only appropriate notion of locality for diffusion models. Rather, we think it is a helpful concrete notion of locality given the goal of connecting diffusion model sampling to classical reaction-diffusion-like, spatial pattern formation dynamics. Essentially, a less constrained notion of locality would be better at capturing real denoisers across a wider range of noise scales, but we would be able to say less about their mathematical properties.

We think it's okay to restrict our scope like this; in our new experiments, we connect the low-noise behavior of a variety of trained denoisers to this notion of locality.

On locality and translational equivariance

I'm not sure I agree that locality means giving up translational equivariance. I think a denoiser can be both. Equivariance just means as you translate the input, the output translates too, that is perfectly compatible with locality, which just means that a particular denoiser output depends only on local inputs that are close to the output coordinate. This goes for fields just as well as projections onto a finite distcrete basis (i.e. arrays). Can the authors elaborate? Or am I misunderstanding the claim the authors are making?

To be clear, we are not claiming that locality and translational equivariance are incompatible. They are compatible! We are only claiming that assuming one or the other is helpful in connecting diffusion model sampling to spatial pattern formation dynamics. (For example, in the linear case, one gets nice PDEs that we can solve using the method of characteristics---but in either pixel space or Fourier space.) By "one or the other", we really mean "at least one". We can revise the text to make this clearer.

Other concerns

Overall I find the results quite simple and generally obvious. The conversions and integrals are all implicit, and not a lot is derived from the results. It often seems to boil down to something akin to: when a diffusion term is added, we observe diffusion. As an example, in lines 334-336, the authors state that if ksi or H go to zero, then the result goes to that of the previous section, but, this can already be trivially seen from the equation before it is solved.

We admit that some of the results in Sec. 4 and 5 may be obvious from a certain point of view. What we were going for is connecting the effective 'receptive field' of the denoiser at different noise scales (which in that setting is related to the diffusion coefficient) to which pixels contribute to the final sample, and how they contribute. We think the method of characteristics provides a 'cool' picture of this. The buildup (ultra-local -> + transport -> + diffusion) is mostly to make it easier to understand that final answer.

Still, maybe we should de-emphasize this section in the revised version of the paper. While we like it, it isn't as important as the later results on nonlinear models, and we now have a lot of new experiments related to those.

The authors involve a lot of physics models, which on the one hand I like (the reviewer is trained as a phycisist), but on the other hand I find the connections often seem superficial, such as the connection to wavespace and the uncertainty principle, as well as field theory. When those concepts were mentioned, I would have hoped to see a stronger connection made in the paper.

We have tried to strengthen this connection in our above new experiments---in particular, the connection between spatial instabilities (in real diffusion models) and spatial pattern formation.

Admittedly, some of the citations are more like 'easter eggs' that we thought would be fun to include. The uncertainty principle citation is an example. We're happy to take it out, and we completely acknowledge that the connection isn't that close beyond being related to real space - Fourier space duality.

Reviewer_PPpiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper models images as continuous fields and treats denoisers and scores as functionals. In this view, probability-flow sampling with a local or translation-equivariant denoiser becomes a reaction-diffusion-like PDE. For linear denoisers built from low-order derivatives, the authors solve the sampling dynamics. Ultra-local terms give independent amplification at each pixel, first-derivative terms give transport along characteristics, and second-derivative terms give diffusion, which they treat with a WKB expansion. Translation-equivariant models give the same structure in Fourier space. For toy priors made of a local potential plus a spatial coupling, a first-order expansion of the denoiser yields Allen–Cahn, Swift–Hohenberg, and vector Ginzburg–Landau forms. Simulations then show domain coarsening, selection of a characteristic wavelength, and vortices (Fig. 3). Fig. 1 provides empirical support: early in training, a CNN denoiser on FFHQ 32×32 is well approximated by a 7×7 linear convolution, while DiT and MLP denoisers become nonlinear sooner.

Critical strengths

Under a local or equivariant denoiser, the PF-ODE becomes a PDE whose terms read directly as reaction, transport, and diffusion. Locality in pixel space and equivariance in Fourier space are handled with the same machinery. This gives a shared vocabulary and toolset (characteristics, WKB, linear stability) for interpreting sampling in CNN-based models, and it is the paper's main contribution.

Critical weaknesses

The framework is not shown to explain or predict the behavior of real diffusion models. The linear results amount to a per-pixel ODE and a standard method-of-characteristics solution, and the exactly equivariant linear case reduces to the known Gaussian-model analysis. The nonlinear results use priors that are themselves Ginzburg–Landau field theories, so the pattern-forming PDEs follow almost by construction. The only link to trained models (Fig. 1) holds early in training and weakens as training proceeds (R2R^2 falls from 1.00 to 0.87). No prediction of the framework, such as per-frequency growth rates, transport or diffusion coefficients, or the selected wavelength 2πk02\pi \over k_0, is tested on a trained model, and Fig. 3 is qualitative with no experimental details. This matters because the paper's contribution is a perspective, so its value rests on demonstrated explanatory or predictive payoff.

Questions for authors
  1. The optimal denoiser's Jacobian is symmetric. For Eq. 10, this forces Gi=(ξ/2)∑j∂jHijG_i = (\xi/2)\sum_j \partial_j H_{ij}, so the first-order term is part of the diffusion term rather than an independent advection term. The pure-drift case in Fig. 2 (center) therefore cannot arise from any exact denoiser. Is transport then specific to non-conservative learned denoisers? If so, could you show that it occurs in practice, for example by measuring the odd (antisymmetric) component of the learned filters in Fig. 1d? The same question applies to the Fourier-space terms in Sec. 5. The answer affects the paper's soundness.

  2. Can any prediction be verified on a fully trained denoiser? For example, extract W~(k,t)\tilde W(k,t) or the derivative coefficients from the network and compare the predicted per-frequency growth with actual sampling trajectories.

3 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your careful reading of our paper and for your helpful comments. We are certain that engaging with them has greatly improved our work, and we hope that the rebuttal below addresses your concerns; if it does not, please let us know, and we will do our best to improve things further!

On predictions and experimental depth

The framework is not shown to explain or predict the behavior of real diffusion models. The linear results amount to a per-pixel ODE and a standard method-of-characteristics solution, and the exactly equivariant linear case reduces to the known Gaussian-model analysis. The nonlinear results use priors that are themselves Ginzburg–Landau field theories, so the pattern-forming PDEs follow almost by construction. The only link to trained models (Fig. 1) holds early in training and weakens as training proceeds (R2R^2 falls from 1.00 to 0.87). No prediction of the framework, such as per-frequency growth rates, transport or diffusion coefficients, or the selected wavelength 2πk0\frac{2\pi}{k_0}, is tested on a trained model, and Fig. 3 is qualitative with no experimental details. This matters because the paper's contribution is a perspective, so its value rests on demonstrated explanatory or predictive payoff.

Can any prediction be verified on a fully trained denoiser? For example, extract W~(k,t)\tilde{W}(k, t) or the derivative coefficients from the network and compare the predicted per-frequency growth with actual sampling trajectories.

We agree that a major weakness of the submitted version is that there is much more focus on the fields and pattern formation formalism than on that formalism's predictions and related experimental results. While we think that this formalism is worth developing and presenting in some detail---a nontrivial task given some of the mathematics involved!---we agree that focusing mostly on the formalism leaves the reader with obvious questions. Does sampling with real diffusion models really look like spatial pattern formation? If so, when, and what kind of guidance does our theory provide?

We are happy to report that we have greatly improved the connection between our formalism and real diffusion models via extensive new experiments. We will include many of these in the updated PDF (albeit in an unpolished form given the limited rebuttal time). Here are some highlights / key points:

  • Trained CNN-based denoisers are approximately local for small noise scales. Fig. 1 of our paper showed that CNN-based (UNet) denoisers are approximately linear early in training, but become considerably less local later. We wanted to see if these denoisers still look approximately local after they have been trained, since this implies that sampling in this regime looks like spatial pattern formation. We trained UNets on FFHQ32 and CIFAR-10 and fit several simple functions to the learned denoisers. We found that although the learned denoisers are definitely not linear, they are fairly local (looking at a 7x7 window is good enough to explain their behavior). As a side note, they didn't seem to be well-described as the gradient of an energy function.

  • Spatial stability analysis reveals spatial instabilities in trained CNN-based models. One claim of our framework is that linear stability analysis, including dispersion relations, can help characterize which spatial structures are amplified or suppressed during sampling. Motivated by this idea, we examined the Jacobians of denoisers trained on FFHQ32 and CIFAR-10 and the associated dispersion relations. We found positive growth rates (i.e., unstable directions) across a wide range of noise levels, something inconsistent with a simple linear/Gaussian approximation. These unstable directions did not correspond to single Fourier modes, but combinations of them, which suggests sensitivity to nontrivial spatial perturbations. Which spatial structures were most amplified changed as a function of noise scale; at lower noise levels, edges and fine details were amplified. These instabilities suggest pattern-forming behavior (i.e., what gets most amplified determines most of the final pattern), which we examined more closely in the next bullet point.

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

  • Trained CNN-based denoisers support spatial pattern formation at small noise scales. To check whether denoisers at small noise scales (with the aforementioned spatial instabilities) support pattern formation, we considered an interesting twist on sampling: instead of doing sampling as usual, we considered fixed-noise-scale denoisers and performed sampling using Langevin dynamics. This is useful because it allows us to attribute pattern-forming behavior to a denoiser at a fixed noise scale rather than to a more complex interaction between denoisers at different noise scales. We did this for the aforementioned FFHQ32 and CIFAR-10 models (which we trained), as well as for a pretrained ImageNet-64 (EDM2) denoiser (whose quality is a bit better). All models supported the development of high-contrast stripes, and we found that this was related to spatial instability structure. This yields pictures reminiscent of our Fig. 3, but which come from realistic, trained diffusion models.

  • Trained text-to-image models exhibit spatial pattern formation behavior for certain prompts at low noise scales. What about bigger, more 'realistic' diffusion models? Do they look like they do spatial pattern formation at low noise scales? We suspect that this should not always be true; rather, it should depend on whether the relevant data (+ prompt) has natural spatial structure. Motivated by this idea, we looked at two publicly available models: GPIC JiT (1.12B parameters) and FLUX.2 klein base (3.88B parameters). We repeated the fixed-noise Langevin experiment for prompts related to natural spatial structure, like "White clouds in a blue sky." and "A close-up of leopard fur with black spots."; we wanted to see if structure like stripes emerged here too. We indeed found nontrivial spatial pattern formation dynamics, including the formation of spots, stripes, and hexagonal latices, which depended on initialization much like conventional reaction-diffusion (i.e., dynamics can manipulate and amplify existing structure, but it's harder to create totally new structure). In FLUX, approximately uniform initial states with constrained latent-channel means can develop spatial patterns whose characteristic scales depend on noise level, sometimes exhibiting initial amplification of intermediate spatial frequencies followed by coarsening-like behavior. These experiments demonstrate that spatial pattern formation is relevant to diffusion models relevant beyond small CNNs and analytically constructed examples.

  • Trained denoisers can support spatial pattern formation, but the Ginzburg-Landau model appears to be a bad description. Another way we tried getting more quantitative was to fit the Ginzburg-Landau (GL) model, one of the models we discussed in Sec. 6, to trained denoisers at low noise scales. The GL model's distinguishing feature is that it is local and assumes a specific low-order polynomial form for the denoiser's nonlinearity. Unfortunately, we found that the FFHQ32 and CIFAR-10 models we trained are not well described by the GL model---not because they aren't local (they are local according to our experiments above), but because the assumed nonlinearity doesn't fit well. We have moved toward thinking that the examples in Sec. 6 are more 'proofs of principle' rather than anything else, and that while they capture qualitative features of spatial pattern formation in real diffusion models, real diffusion models are unlikely to look exactly like them except in special cases. We are working to do additional fits to say more about real denoisers' nonlinearities. Our analysis so far suggests that the nonlinearities learned by real denoisers are more complex; for example, they can look like the gradients of more heavy-tailed (rather than polynomial) potential functions.

Overall, we are much more confident that:

  1. Denoisers trained on distributions with spatial structure (like natural images) can be local and support pattern formation (e.g., stripes, spots, lattices) at low noise scales. (Examples: FFHQ32, CIFAR-10, ImageNet-64, GPIC JiT, and FLUX.2 klein base)
  2. This pattern formation behavior can be directly linked to spatial instability, as our framework suggests.
AuthorsOpenReview ↗

Rebuttal (part 3)

(continued)

On non-conservative score/denoiser fields

The optimal denoiser's Jacobian is symmetric. For Eq. 10, this forces Gi=(ξ/2)∑j∂jHijG_i = (\xi/2) \sum_j \partial_j H_{i j}, so the first-order term is part of the diffusion term rather than an independent advection term. The pure-drift case in Fig. 2 (center) therefore cannot arise from any exact denoiser. Is transport then specific to non-conservative learned denoisers? If so, could you show that it occurs in practice, for example by measuring the odd (antisymmetric) component of the learned filters in Fig. 1d? The same question applies to the Fourier-space terms in Sec. 5. The answer affects the paper's soundness.

This is an astute observation. It is indeed true that, because accurate denoisers are closely related to score functions, and because score functions are conservative (i.e., they are the gradient of some other function), the terms that appear should not be independent. It is absolutely true that the pure-drift case we discuss can't emerge from any exact denoiser. Exact transport is specific to non-conservative learner denoisers, but transport still occurs in conservative ones (albeit with some diffusion).

We did realize this and didn't comment on it due to space limitations. We don't think this is a soundness issue for two reasons. First, it's well-known that trained denoisers are not necessarily conservative. There's been some work on this, with some authors even suggesting that what diffusion models learn in practice is something sort of conceptually different from a score function. (See, e.g., https://arxiv.org/abs/2311.09952 ; https://arxiv.org/abs/2509.00336 ) Hence, we can accommodate a larger variety of real denoisers within our framework by not necessarily requiring exactness.

Second, our goal in Sec. 4 and 5 was really just to build intuition for how to thinking about sampling dynamics in linear local models. Even if the pure transport case isn't that relevant in practice, in our opinion, going step by step (starting from a purely local model, then adding transport, then adding diffusion) makes it easier to understand the final method of characteristics result (Eq. 20).

Finally, given all of our new experiments, we may want to de-emphasize that section in the final paper. It's a 'cool' result, but less important than the nonlinear model results, which may require more space.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Comments on the revision and a summary of discussion so far

Thank you to all the reviewers for your time, energy, helpful comments, and suggestions. We are confident that they have greatly improved the paper, and we are excited to see what you think.

In case it is helpful, here is a summary of the discussion so far, along with a brief guide to the updated PDF.

Perceived strengths and weaknesses

All reviewers generally seem to find our field-theoretic framework interesting and our results mathematically correct. The perceived weaknesses mostly center around the significance of our framework. In particular, do real diffusion models behave like the field-theoretic models we study here? And what does our framework tell us (about said real diffusion models) that we did not already know?

The reviewers were pretty unified regarding this being the major issue---fairly, we think!---so we have made it the focus of our time during the rebuttal period.

There were other, more minor issues/questions (e.g., aren't the coefficients of an exact denoiser constrained?), but we think we have convincingly addressed them in our reviewer responses.

Guide to new experiments and updated PDF

We are happy to report that we have conducted an extensive set of new numerical experiments. Fig. 1 of our original paper already shows that simple CNN-based diffusion models behave like linear, local/field-theoretic models early in training. Hence, we have already shown the relevance of local linear models. Our main focus in these new experiments was in showing that our framework is relevant for explaining the behavior of trained diffusion models.

There are two important points to note here. First, although we study linear models in Sec. 4 and Sec. 5 of our paper, our field-theoretic framework does not require linear models. The linear case is mostly interesting because it's a case where sampling dynamics can be solved exactly (and where the solution one obtains can be compared to existing solutions of linear diffusion model sampling). Second, we generally expect the trained denoisers to be nonlinear, even if they are local.

In other words, in our new experiments, we did not expect to find linear models (and we basically didn't). Instead, we were looking for spatial pattern formation dynamics like the kind we discussed in Sec. 6.

Our new experiments provide the content for Appendix F through N of the updated PDF. While the language and presentation of the appendices is still quite rough (we must disclose that the writing of these appendices is partly LLM-generated at present), our goal during this rebuttal period was to do as many experiments as possible to convince you that our framework is relevant.

Appendix F provides a brief overview of all the new experiments. Here's an excerpt from it:

The main text develops the pattern-formation picture with models that can be solved by hand. The remaining appendices ask how much of it survives in trained networks. We begin with large pretrained image models, where nothing about the data was arranged in our favor, and ask whether running one at a fixed noise level produces anything like a pattern-forming system (Appendices G and H). We then go to the opposite extreme, networks trained on fields whose structure is known exactly, including the models of Sec. 6, where the pattern-forming operator can be read back out of the network and compared with the truth (Appendices I, J, and K). Finally we return to convolutional denoisers trained on natural images: early in training, where they are close to the local linear models of Secs. 3-5 (Appendix L), and at the end of training, where we ask how local the learned operator remains and what nonlinearity it has acquired (Appendices M and N).

In case bullet points are helpful, we reiterate that these experiments fall into a few categories:

  • Showing spatial pattern formation behavior in trained diffusion models. (Appendices G and H) We obtain some cool results here, like showing trained denoisers can produce lattices, spots, and patterns of stripes.
  • Showing denoisers can be inferred when the data comes from spatial pattern formation. (Appendices I, J, and K) This is useful in part because it helps us be confident that, if spatial pattern formation structure exists (for example, in real denoisers), we should be able to extract it successfully.
  • A deeper look into CNN-based denoisers early in training. (Appendix L) Recall that we showed these denoisers are local and linear in Fig. 1. Here, we look at other properties of them, like whether it turns out they have a transport term (related to a question of PPpi).
  • Characterizing nonlinear local denoisers. (Appendices M and N) When trained denoisers are local, so that our field-theoretic framework applies, what do the denoisers actually look like? If they're not linear, what kind of linear model fits? For example, we show that simple Ginzburg-Landau models (see Sec. 6) generally don't fit.
AuthorsOpenReview ↗

Final comments

Final comments

We are now much more confident, based on these experiments, that untrained and trained (FFHQ32, CIFAR-10, ImageNet-64, GPIC JiT, and FLUX.2 klein base) denoisers can exhibit spatial pattern formation. This typically happens at low noise scales.

We have also quantitatively analyzed these denoisers in various ways. This includes looking at Jacobians to analyze their linear stability, empirically determining dispersion relations, and trying to fit simple (e.g., Ginzburg-Landau) models.

We think the major interesting prediction of our framework (which is similar in some ways to ideas considered by Luca Ambrogioni in a paper cited by reviewer yzRp) is that it connects spatial instability to spatial pattern formation. In particular, the former can be used to predict the latter.

This is worth exploring further, but as things are now, we think we have put together a worthy contribution. Part of our contribution is developing in detail a nice mathematical framework (which is backed up by detailed appendices; see Appendix A through E), and the other part is now doing a lot of experiments to show that reaction-diffusion-like spatial pattern formation really does happen in diffusion models, along with doing some early quantification of this.

We hope that these experiments are persuasive---or at the very least, extremely interesting! We humbly ask you to consider our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.