acceptodds
Under review as a conference paper at ICLR 2027

On the Spectral Properties of Generative Denoiser Jacobians

Abstract

Generative denoising models, such as diffusion and flow-matching, learn to sample from complex distributions by training a deep neural network denoiser to recover clean data from noise-corrupted samples. While such models are typically compared on the quality of their synthesized samples, these metrics provide limited insight into how the underlying denoiser, which drives generation, differs. In this work, we propose to analyze the spectrum of the denoiser Jacobian as a tool to characterize these differences. Across pre-trained denoising models, we observe that better generative performance is associated with larger Jacobian eigenvalues. Motivated by this, we introduce a regularization scheme that controls the Jacobian spectrum by training the denoiser on perturbed inputs, with perturbations suppressing or amplifying Jacobian responses. On ImageNet, we test whether directly modifying the Jacobian spectral properties leads to improved generations. Our findings suggest that denoisers benefit from both strengthening responses along data-relevant principal eigen-directions and suppressing the noisy, data-irrelevant ones. This establishes the denoiser Jacobian as a useful tool for identifying differences between generative denoising models.

Primary area:
generative models
Keywords:
Generative models, denoising, flow-matching models, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UhSvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between denoiser Jacobian spectra and generative performance. The authors observe that better-performing models tend to exhibit larger principal Jacobian eigenvalues and propose residual-based and stochastic regularization to control spectral properties and improve generation quality.

Critical strengths
  • Meaningful research question and coherent methodology. The paper presents a clear research progression from empirical observations and theoretical analysis to targeted regularization methods and experimental validation.

  • Interesting theoretical insights. The connection between Jacobian spectra and posterior covariance provides a principled motivation. Using denoising residuals to strengthen principal spectral directions without explicit eigendecomposition is particularly interesting.

  • Encouraging experimental results. The proposed regularization improves FID across SiT-S, SiT-B, and UNet. Combining residual and stochastic regularization further improves performance, suggesting complementary benefits.

Critical weaknesses
  • Insufficient validation of stochastic regularization's spectral mechanism. The authors argue that moderate stochastic regularization suppresses weak, data-irrelevant spectral directions while preserving principal components. However, Figure 5 mainly shows that the top-10 eigenvalues remain nearly unchanged. This does not establish that weaker components are suppressed or that they are data-irrelevant. Consequently, the proposed explanation for the complementarity between the two regularizers remains insufficiently verified.

  • Inconsistent improvements across sampling settings and metrics. Stronger spectral amplification does not always improve FID, and the benefits of residual regularization diminish under some CFG settings. FID and sFID also occasionally show conflicting trends, suggesting that the relationship between spectral properties and generation quality is more nuanced.

Questions for authors

Q1. Can the authors directly demonstrate that stochastic regularization suppresses weaker spectral components, for example by comparing broader spectra across all four model variants?

Q2. What evidence indicates that the suppressed directions are data-irrelevant rather than weak but meaningful variations?

Q3. Can the authors clarify the reduced benefits under CFG and the occasional discrepancies between FID and sFID?

Reviewer_dMTEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the Jacobian of the denoiser in diffusion and flow-matching models. The authors first observe that better generative models tend to have larger leading Jacobian eigenvalues, even when their ordinary denoising MSE is similar. Based on this observation, they propose perturbation-based regularization: residual-based perturbations increase the response along principal, data-relevant directions, while random perturbations suppress sensitivity to random directions. Experiments on ImageNet show that the regularizer changes the Jacobian spectrum as intended and improves FID.

Critical strengths

I like the main design: rather than simply encouraging a large Jacobian everywhere, the method strengthens residual-aligned principal directions while weakening random directions. This gives a clear and intuitive picture of a useful denoiser Jacobian: strong responses on relevant directions and weak responses on irrelevant ones. The experimental design is also well structured.

Critical weaknesses
  1. The related-work discussion and novelty positioning should be strengthened. In particular, EigenScore (ICLR 2026) also studies the leading spectrum of posterior covariance induced by diffusion denoisers, and estimates it using a Jacobian-free subspace iteration based only on forward evaluations of the denoiser. It is recommended to cite this paper and distinguish the contribution and difference.

  2. The residual regularizer requires an additional denoiser forward pass, and the limitation section also notes that this considerably slows training. It would be helpful to report the actual training-time, throughput, and/or memory overhead, so that readers can better assess the practical trade-off between the FID gains and the additional cost.

Questions for authors

Please refer to the Weakness

Reviewer_aAscOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the properties of the Jacobian of the denoising model in diffusion models. In the experiments eigenvalues of the Jacobian are calculated, to show that the calculated spectrum can be a proxy of the model performance. In particular models with lower FID have also larger first eigenvalues. Based on this analysis authors extend it by adding Jacobian regularization during training, showing that this approach can improve the final performance.

Critical strengths

Paper presents a very fresh perspective. After reading abstract and first part of the introduction I was really intrigued and excited for the rest

Critical weaknesses
  • The paper spends a lot of space on textbook material. Sec. 3.2, for instance, explains Jacobians, eigendecomposition and power iteration, which could be cut to a citation. Tightening this would sharpen the presentation and free space for the missing quantitative analyses.
  • The evaluation is extremely narrow, focusing only on a single family of rather small image generation models (even without text conditioning). While I understand that for larger experiments with training from scratch the limitation might be the computational budget, analyses could be performed over different architectures. This would significantly strengthen paper’s claims.
  • Connected to the previous dot, the evaluated baseline models are in general rather far from recent SOTA solutions on the same dataset, with FID=59 on ImageNet without any add-ons. At the same time, for higher quality models the gains from Jacobian regularization shrink significantly.
  • “Our intuition is that while the smaller SiT-S synthesizes similar-looking images from all inputs in the neighborhood of xt, better-performing models produce vastly different results as small perturbations around xt are amplified in the Jacobian. Interestingly, looking only at the denoising error is inconclusive, both with and without guidance.” - This is an interesting hypothesis, but it has no confirmation in any experiments nor there is any theoretical confirmation that this is true. The single generation of a dog presented in Figure 3 is not enough for a proof.
  • I am not convinced by the mathematical derivations presented in Section 4. Authors start from Equation 6 which is not a standard denoising objective of the diffusion model, as it has an additional term which changes the expectation over the denoiser
  • Figure 3 does not support the claims made about it. The \lambda maps are visually almost identical (also they have different scales), The generated sampled points don’t appear to be better aligned with data area - they sometimes are in between distributions in all 3 panels. This analysis also would benefit a lot from quantitative results with numbers supporting the claims.
Questions for authors

Given the extended cost of the proposed method (2 passes instead of one), are the evaluations calculated with fair budget (2 times more training steps for the baselines)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.