When Frequencies Interact: A Gradient View of Diffusion Training

Diffusion models typically learn low-frequency structure before high-frequency detail. Existing explanations focus on data variance and differences in learning speed across frequencies. We study frequency learning from a gradient perspective and find that different bands can influence one another through shared denoiser parameters. Our analysis shows that orthogonal frequency residuals can produce opposing gradients, with their magnitudes and directions determining how each band is affected. Low-frequency gradient dominance can amplify even mild opposition, allowing a sufficiently small joint update to reduce total error while increasing high-frequency error. Measurements and test updates on a standard denoiser reveal this interaction and its dependence on noise level. Motivated by these findings, we propose SpecDiffusion, whose Frequency-Aware Loss adjusts the relative band weights across diffusion timesteps, optionally complemented by a lightweight frequency-view adapter. Across pixel- and latent-space diffusion models with convolutional and Transformer denoisers, SpecDiffusion consistently improves matched baselines. These results offer a gradient-based view of frequency learning and connect it to a simple training objective.