acceptodds
Under review as a conference paper at ICLR 2027

Denoising Strain: A Geometric Order Parameter That Reveals Learning Ahead of Eval Metrics

Abstract

Diffusion models are evaluated through denoising loss and sample-quality metrics such as Fréchet Inception Distance (FID). These scores track performance, but do not directly show the generative structure taking shape during learning. We introduce Denoising Strain, a geometric description of how each complete sampler step contracts, amplifies, and transports nearby perturbations. Separating overall scale from directional stretching gives a tensor order parameter: it vanishes when all directions are treated equally and develops structure as preferred directions form. The discovered axes have measurable consequences for generation. On pretrained animal-face and human-face models, leading-axis interventions produce median final-image separations about 28 and 24 times those of equal-norm random controls. A separate continuous-step study uses three intervention doses on 100 held-out trajectories per dataset and finds feature-space dose-response slopes above controls at all 11 measured states. We also track this geometry along uninterrupted MNIST and CIFAR-10 training runs, where initially near-equal stretches differentiate and expanding directions appear. On CIFAR-10, expansion and anisotropy are detected after 10,000–20,000 training images, whereas the checkpoint completing half of the total log-FID improvement arrives after 5 million images—250–500 times later in training exposure. Denoising Strain thus provides an order parameter that links how diffusion models generate to how that ability develops during training.

Primary area:
generative models
Keywords:
diffusion models, generative models, denoising dynamics, order parameters, Jacobian spectra, training dynamics, dynamical systems

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_rtkqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops a metric called "Denoising Strain" which measures how the local geometry changes during denoising using diffusion models. The authors explore how amplifying certain directions during denoising by perturbing the denoising trajectory can significantly change the generated images' semantics, while authors also note how anisotropy develop in diffusion models during training.

Critical strengths
  1. The paper develops a new metric for assessing how the geometry of diffusion models changes during sampling and training.
  2. The authors follow a well-known framework (EDM) in their evaluations.
  3. The authors describe the method and experiments clearly and concisely.
Critical weaknesses

I find the significance/usefulness of Denoising Strain unclear. The paper shows that certain directions in the sampler are more sensitive to perturbations than random directions, and that anisotropy during denoising develops early in training. However, the paper does not adequately establish how early anisotropy relates to meaningful performance gains.

More generally, it's unclear what new thing Denoising Strain tells us that can't already be gleaned from existing diagnostics or simpler Jacobian-based analyses. The paper doesn't demonstrate how the proposed measurements could be used to improve or guide training, predict model quality, or better understand the distribution of generated images. Given the limited theoretical novelty, experimental scope, and lack of meaningful comparisons with related methods, I find the paper doesn't demonstrate significance of contributions.

Major weaknesses

  1. Unclear significance/practical usefulness of Denoising Strain. The paper introduces Denoising Strain to understand how structure develops during training, but does not clearly establish what useful information it provides beyond existing diagnostics. Specifically, it is unclear whether Denoising Strain can predict model performance, guide training, or identify problems that would otherwise be difficult to detect. (lines 34-47, 456-468)
  2. No direct evidence that early anisotropy corresponds to meaningful learning. The authors show that perturbing in strain directions can produce meaningful changes to generated images for pretrained models (Sec 3), and separately show that anisotropy emerges early during training (Sec 4). However, they do not demonstrate that the early-appearing directions already capture meaningful semantic or distributional structure. In my view, the claim that early geometric organization and learning are connected is not justified by these experiments. (lines 270-323, 351-387, fig 3,4,8)
  3. Insufficient comparisons with other ways of measuring learning. The paper doesn't properly investigate whether other measurements (eg denoising loss, Jacobian spectral statistics of the denoiser, or simple measures of generated-image structure) detect signs of learning equally well. Although held-out loss and image morphology are reported for MNIST, there is no systematic comparison of early-detection across metrics. Denoising Strain changing early in training does not necessarily make it a better indicator of future generative performance. (lines 341-349, 370-387, fig 4,9, appendix D.1-D.2)
  4. The authors did not cover related works comprehensively, particularly on Jacobian-based interpretability of diffusion models. The related works section omits several papers that already use denoiser Jacobians to characterise semantic directions or generative behaviour or perform image editing [1] [2]. (lines 444-452)
  5. Additionally to 4, authors do not provide meaningful comparisons against exisitng Jacobian-based interpretability methods. This paper's main sampling-time experiments compare Denoising Strain directions with random perturbation directions, instead of methods proposed in earlier works. It is unclear whether computing Denoising Strain through sampler steps provides benefits over analysing the denoiser model's Jacobian directly, especially given conceptual overlap with earlier work. (lines 174-197, 207-226, table 1, appendix B.1)
  6. Authors provide limited evidence that the directions from Denoising Strain correspond to meaningful semantic structure. Fig 3 has visualisations, but the main quantitative metric (in the main body) is pixel-space separation between generated images, which doesn't measure semantic difference or sample quality. The feature-prediction experiments report only 0.62-1.72% and 0.66-2.17% explained variance in DINO features on AFHQ and FFHQ, respectively. The authors also report significant feature-space responses in fig 7, but these results don't clearly establish that the directions correspond to interpretable semantic changes. (lines 270-323, 790-809, fig 3,6,7, appendix C)
  7. Limited generality of experiments. Training-time results are limited to MNIST and CIFAR-10, with a few trajectories for measuring anisotropy. Sampling-time experiments use pretrained AFHQ/FFHQ models. All experiments use EDM in pixel space, so it is unclear whether the observations generalize to somewhat more modern SiT/JiT/DiT frameworks ([3] [4] [5]), larger-scale generation (ie higher-resolution images, more trajectories, or text-to-image), or other modalities (eg text to img). The paper does compare Euler/Heun, but only on selected MNIST checkpoints. The authors indicate "Diffusion Transformers provide a direct next setting; language models..." in the future works section around line 464, but this is insufficient since the experimental results are significantly lacking in both depth and breadth for an interpretation paper. (lines 207-226, 333-344, 461-465, appendix B.2)
  8. The claim that during training, anisotropy provides information beyond FID is not adequately established. The experiments mostly show that geometry and FID change on different timescales, rather than demonstrating what additional information is provided by anisotropy measurements about the generated distribution/model performance. Specifically, it's unclear what changes in the generated image distribution correspond to the emergence/strengthening/other changes in anisotropy. (lines 370-426, fig 4,9, appendix D.4)
  9. The paper does not adequately establish whether geometric measurements remain useful for evaluating model quality at later stages of training, particularly when the model approaches its best FID. The paper emphasises early training when anisotropy first emerges, rather than whether it remains informative as a model approaches its best FID. The late-training MNIST comparison uses similarity to the final checkpoint's geometry, and this isn't an independent measure of quality, since becoming more similar to the final checkpoint does not necessarily mean becoming a better generative model. There is insufficient analysis of whether Denoising Strain discerns high-performing checkpoints form low-performing ones or whether it explains changes in the models' performance. (lines 370-426, fig 4(d),9, appendix A,D.4, eq 13)
  10. The authors primarily rely on FID for their training-time results. It can be useful to evaluate whether the conclusions hold with distributional metrics computed in different representation spaces (see [6]). Since authors claim that Denoising Strain reveals aspects of generation that FID and other metrics miss, using different latent spaces for distributional metrics is important. (lines 341-344, fig 9, appendix D.1)

Minor weaknesses

  1. Small sample size for the sampling-time intervention experiment (Sec 3). Table 1 uses only ten trajectories per dataset, which is insufficient to establish how consistently the effects occur across generated images. The authors conduct a larger experiment with 100 held-out trajectories, but this uses a different formulation and different perturbation method.
  2. The paper has limited theoretical novelty. The central derivations are largely standard and lack insight, while various properties largely follow from standard facts from linear algebra and dynamical systems. The paper doesn't provide a theoretical relationship between the tensor order parameter and performance/generalisation.
  3. Computational overhead isn't properly justified via utility/efficiency. Computing the leading strain directions requires repeated jvp/vjp, which is time-consuming based on my experience despite optimised PyTorch-native functions. Appendix B reports ~353-366 seconds per trajectory for 9 denoising states. Without a practical application and a comparison against other diagnostics, it's unclear when this additional computation due to Denoising Strain would be worthwhile.
Questions for authors

See weaknesses.

[1] Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models, https://arxiv.org/pdf/2303.11073

[2] Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing, https://arxiv.org/pdf/2409.02374

[3] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers, https://arxiv.org/pdf/2401.08740

[4] Back to Basics: Let Denoising Generative Models Denoise, https://arxiv.org/pdf/2511.13720

[5] Scalable Diffusion Models with Transformers, https://arxiv.org/pdf/2212.09748

[6] Representation Fréchet Loss for Visual Generation, https://arxiv.org/pdf/2604.28190

Reviewer_u4gROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Denoising Strain, and shows that its preferred directions shape the generated image and emerge during training before FID.

Critical strengths

A well-defined geometric quantity. Denoising Strain is measured on the full executed sampler step and separates overall scale, directional anisotropy, and transport.

The directions affect the generated image. Equal-norm perturbations along the leading axis change the finished image much more than random controls.

The geometry is tracked through training.

Critical weaknesses

The experiments use small, limited models, with no exploration of scale and no attempt on modern DiT architectures, so the generality of the conclusions is limited.

The paper only shows that directional differentiation appears. What does that bring? Need to more explore.

Questions for authors

see the weaknesses

Reviewer_oZsSOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Denoising Strain, a geometric framework for analyzing the learning dynamics of diffusion models. By examining the Jacobian of denoising sampler steps, the authors characterize directional contraction, expansion, and anisotropy. Experiments demonstrate that the identified directions meaningfully affect generated images and that geometric organization emerges substantially earlier than improvements in conventional evaluation metrics such as FID.

Critical strengths

The paper provides an interesting geometric perspective on understanding diffusion model training dynamics. The proposed framework is mathematically well-motivated, and the intervention experiments offer empirical evidence connecting the identified geometric directions to meaningful changes in generated images.

Critical weaknesses
  1. The training dynamics experiments are restricted to MNIST and CIFAR-10, with only one continuous training run per dataset. The lack of validation on larger datasets, modern architectures such as DiTs, and multiple independent training runs raises concerns about the generalizability of the findings.

  2. Early geometric changes do not necessarily imply predictive value. Although Denoising Strain detects geometric changes substantially earlier than FID improvements, this does not establish its effectiveness in evaluating or predicting model quality. The paper lacks sufficient evidence that these early geometric signals reliably correlate with subsequent generative performance.

Questions for authors
  1. Can the authors validate Denoising Strain on larger datasets, modern diffusion architectures (e.g., DiTs), and multiple independent training runs to demonstrate its generalizability?

  2. Can the authors demonstrate whether early changes in Denoising Strain can reliably predict subsequent generative performance (e.g., final FID), rather than merely preceding improvements in evaluation metrics?

Reviewer_kdJvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes geometric previewing of diffusion models in training, showing geometric drift happens early on with evaluations on some faces datasets.

Critical strengths
  1. analyze diffusion model previewing during training/inference is important to save computational cost
  2. the jacobian analysis is in the right direction
Critical weaknesses
  1. writing is confusing and prevents understanding. For example in the abstract: "Separating overall scale from directional stretching gives a tensor order parameter: it vanishes when all directions are treated equally and develops structure as preferred directions form. ": this sentence just means that the directional stretching may appears early one when overall scale change not that obvious. I recommend authors write every sentence of this paper by hand, not through AI tools.
  2. This phenomenon is not something new, the training dynamics of diffusion models has been analyzed many times: like "Analyzing and Improving the Training Dynamics of Diffusion Models" CVPR 2024. The Jacobian analysis is also nothing new: "Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing" NeurIPS 2024. It is quite intuitive why this geometric strain happens, during training, the high noise region decides geometry is easier to steer (its Jacobian has higher matrix conditional number, since it is mostly low rank), so the model will just modify geometry easily. Later on, the low noise region changes more, and then the model refine texture,color and other details more. This paper provides no additional insights beyond this.
  3. If the paper makes some more empirical observations or downstream applications, it can be more valuable. But the evaluation is very limited to the two face datasets, and very limited model types. It is also unclear, what downstream applications can be performed with this diffusion feature. Some advice I can give to authors: for example, you can try analyzing the Jacobian of pretrained diffusion model at different noise levels, finding the one having largest impact on geometry, and try to build a network or an algorithm for image editing. Also if this geometry shift is here, can model training be accelerated on multiresolution dataset.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.