acceptodds
Under review as a conference paper at ICLR 2027

When Frequencies Interact: A Gradient View of Diffusion Training

Abstract

Diffusion models typically learn low-frequency structure before high-frequency detail. Existing explanations focus on data variance and differences in learning speed across frequencies. We study frequency learning from a gradient perspective and find that different bands can influence one another through shared denoiser parameters. Our analysis shows that orthogonal frequency residuals can produce opposing gradients, with their magnitudes and directions determining how each band is affected. Low-frequency gradient dominance can amplify even mild opposition, allowing a sufficiently small joint update to reduce total error while increasing high-frequency error. Measurements and test updates on a standard denoiser reveal this interaction and its dependence on noise level. Motivated by these findings, we propose SpecDiffusion, whose Frequency-Aware Loss adjusts the relative band weights across diffusion timesteps, optionally complemented by a lightweight frequency-view adapter. Across pixel- and latent-space diffusion models with convolutional and Transformer denoisers, SpecDiffusion consistently improves matched baselines. These results offer a gradient-based view of frequency learning and connect it to a simple training objective.

Primary area:
generative models
Keywords:
Diffusion & Generative Models for Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z9WoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes frequency bands interaction when the diffusion model trained with shared parameters for time step image (or noise) recovery. They then proposes a method named SpecDiffusion based on Frequency-Aware Loss (FAL) after they perform the weighting analysis on contributions from high-frequency error and low frequency error after gradient descents are applied to update the images. Their experiments show that their method reduces FID when compared with previous method (pixel space and latent-space diffusion models.

When I like the idea of comparing the frequency contributions over time reconstruction steps, I found this paper is poorly written and needs a major revision to make their paper easier to read. Also, the experiments did not show any real image reconstructed.

Critical strengths

This paper analyzes frequency bands interaction when the diffusion model trained with shared parameters for time step image (or noise) recovery, and they provide a new method that uses Frequency-Aware Loss (FAL) after they performed the weighting analysis on contributions from high-frequency error and low frequency error after gradient descents are applied to update the images.

Critical weaknesses
  1. The paper overly is poorly written and you need to go to check the appendix in order to understand the main text.
  2. Frequency content in image is highly sensitive, with high values in the low frequency domain and very small values in the high frequency domain, the analysis doesn't sees address this point clearly.
  3. Their experiment did not specifically point out how they choose the two bands (low frequency and high frequency). And no reconstruction clean image is provided.
Questions for authors
  1. Page 3, eqn. (2), line 142-143, the first term at the left hand side of the equation is a multiplication or a function of (ϑ − ηu)?
  2. why use Nyquist normalization? How it benefits your method?
  3. Page 2, line 78, the paper states that "Their residuals are orthogonal in output space", what "their" referred to?
  4. And what is the output space?
  5. The frequency analysis is performed on mean squared gradient errors. I wonder if that would affect the true frequency content of the errors.
Reviewer_5DkiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the gradient interaction between low-frequency and high-frequency denoising objectives under shared parameters in diffusion models, and analyzes when joint updating reduces the total error but may nevertheless increase the high-frequency error. Based on this, the authors propose SpecDiffusion, which includes a Frequency-Aware Loss (FAL) that adjusts frequency-band weights according to the diffusion timestep and an optional Frequency View Adapter (FVA), and evaluate its generation performance on a variety of pixel-space and latent-space diffusion models.

Critical strengths
  1. The paper clearly distinguishes the orthogonality of frequency-band residuals in output space from the gradient conflict that may arise after they are backpropagated through shared parameters, and derives an interpretable local criterion. The authors also provide empirical results that echo the analysis through gradient measurements and test updates along low-frequency directions.

  2. The generation experiments cover multiple pixel-space and latent-space models and use paired settings wherever possible; most results use three independent seeds. Table 2 also shows that FAL alone reduces the average FID under both the L1 and L2 settings on FFHQ, providing preliminary support for loss reweighting itself.

Critical weaknesses

1.The gradient diagnostics are limited to the init_conv.weight block of one pixel-space UNet, while the theoretical analysis is local to a selected parameter block, minibatch, and timestep. Moreover, the adopted weight ratio depends only on the timestep, whereas the descent conditions in Eqs. (8) and (12) also depend on the gradient norm ratio and alignment; the paper does not quantify how often these conditions hold across layers or actual full-model updates.

2.The main generative comparisons mostly use the combined FAL+FVA configuration, while FAL-only evidence is concentrated on the FFHQ ablation and a single-run Lightning-DiT result. Since FAL changes both relative band weights and the overall timestep loss scale, and the fixed cutoff c=0.35 lacks sensitivity analysis, the current experiments do not fully disentangle the contributions of FAL, FVA, overall loss scaling, and the chosen frequency partition.

Questions for authors

1.Can you report the gradient norm ratios, angles, and high-frequency margins across different network layers or over the full model, and indicate how many actual training updates satisfy the conditions in Eq. (8) or Eq. (12)? This would help determine whether the theoretical analysis guides the actual weight schedule.

2.Can you add controlled experiments on representative pixel-space and latent-space models: normalize the FAL weights by their average scale to isolate the effect of relative frequency-band reweighting, and report FAL-only and FVA-only results separately?

3.Are the main FID conclusions stable when the cutoff frequency c varies within a reasonable range? For latent-space models, how do the authors define the relationship between latent feature frequency bands and image frequencies?

1 comment

Thank you again for your thoughtful questions. We look forward to hearing your thoughts on the revised manuscript.

Reviewer_DnbXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Denoising diffusion models learn data distribution by first learn the low-frequency structure and then the high-frequency details. The authors study how these frequency bands interact through the shared paramters of a denoiser. They show that although the low and high frequency residuals are orthogonal in output space, their gradients could conflict in parameter space due to the shared denoiser weights. When low-frequency gradients dominate, a joint update can reduce the total denoisng error while increasing the high-freq error. Then motivated by these findings, the authors propose a frequency-based loss SpecDiffusion, which adjusts the relative band weights at each timestep t. It emphasizes low frequencies at high noise levels, and high frequencies at low noise levels. Experiments across pixel and latent space diffusion show improved generation quality compared to matched baselines.

Critical strengths
  1. I found the author clearly state their problem and motivations, it is easy to understand. The authors provide a concrete characterization of frequency-band interaction through shared denoiser parameters, and based on these findings, they proposed the frequency-based loss FAL.

  2. The proposed loss functino FAL and the adapter FVA are simple, and it works well that it shows consistent FID improvements across many backbones in both pixel and latent space.

Critical weaknesses
  1. In the diffusion literature, there are many existing weighting approaches, however, most results compare SpecDiffusion with the corresponding unmodified backbone, without matched comparisons against established frequency-aware losses or timestep-weighting methods. I think these comparisons are very important for determining whether the proposed approach has meaningful advantages beyond existing weighting methods.

  2. The experiment evidence is insufficient to establish that the performance gains come from the proposed mechanism (mitigating harmful frequency band interference). I find in the experiments, FAL changes both the relative frequency-band weights and the overall timestep-dependent loss scale, but the experiments do not isolate their effects. It is unclear how much of the improvement attributes to frequency-specific weighting rather than timestep reweighting.

  3. While improving FID, the paper provisdes limited evidence that high frequency prediction or generated details improves. It would be better if the authors can show a more direct experimental support (see question 4).

Questions for authors
  1. I want to clarify: how was the cutoff c=0.35 selected? If my understanding is correct, this hyperparameter c determines the low and high frequency partition and may affect both the gradient diagnostics and the loss weighting. Have you evaluated whether the following remain consistent under nearby cutoffs? e.g. observed gradient dominance, negative margins, FIDs.

  2. Can you isolate the contributions of timestep scaling and relative band weighting by comparing FAL with the following using the same backbone w/o FVA?

  1. scalar-only objective wˉ(t)(EL+EH)\bar w(t)(E_L+E_H);
  2. the normalized band-weighted objective ((wL(t)EL+wH(t)EH)/wˉ(t))((w_L(t)E_L+w_H(t)E_H)/\bar w(t))
  1. In the experiment, you didn't compare FAL wiht existing frequency-aware losses or timestep-weighting methods under the same training and evaluation. Can you provide at least one of each as your baselines?

  2. given the frequency-based motivation, can you provide a more direct evaluation of frequency-specific improvements beyond FID and the local gradient diagnostics? e.g. frequency-resolved denoising errors on held-out data, this will help to answer whether FAL improves high-frequency prediction.

2 comments

Thank you again for your thoughtful questions. We look forward to hearing your thoughts on the revised manuscript.

Dear Reviewer DnbX, Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 1 day, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper. Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.