- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MoSE, a training-time spectral regularization framework for diffusion models. The method introduces four spectral experts, including Fourier and wavelet energy experts for dataset-level statistics, and two per-sample experts for spectral consistency. A timestep-conditioned router is used to dynamically balance the four losses during training. Experiments on several image generation benchmarks and physical system tasks show improvements in generation quality and spectral fidelity.
Critical strengths
- The idea of combining Fourier and wavelet regularization is reasonable, since they capture different aspects of frequency information. The distinction between dataset-level and per-sample spectral constraints is also interesting.
- The results on image generation are promising with consistent improvement in FID on all four datasets. Also, since the method only adds regularization during training, it does not introduce extra inference overhead.
- The evaluation covers both image generation and physical data, which is appreciated. The ablation studies in Appendix C.2 are also relatively comprehensive, including different combinations of experts and routing strategies.
Critical weaknesses
- My main concern is about the formulation of the spectral energy experts. In Eq. 11, the Fourier energy expert minimizes the Mahalanobis distance between the predicted sample's spectral features and the mean statistics of the real data. However, I don't quite understand why minimizing such a sample-wise distance would encourage matching the whole data distribution. For example, if we only optimize this loss, it would simply encourage all generated samples to move toward the mean of the spectral features, rather than matching their actual distribution. This seems different from the claimed dataset-level distribution matching. Besides, in Eq. 14, the last two terms of the wavelet energy expert are computed using the ground-truth sample (x), instead of the predicted sample (x hat). If the dataset statistics are fixed, these two terms would not provide gradients to the denoising network as intended. I'm not sure if this is just a typo or something wrong with the actual formulation, but I think the authors should clarify these issues, as they directly relate to the core methodology of the paper.
- Another concern is the novelty and whether the proposed combination is really necessary. Spectral regularization for diffusion models has already been explored in previous works, especially [1], which also uses Fourier and wavelet losses during diffusion training. MoSE mainly extends this idea by combining more spectral losses and introducing a learned router. However, there is no direct comparison with this closely related method. Although the authors provide ablations on different experts and routing strategies, these are mainly conducted on a vanilla DDPM with a much weaker baseline (e.g., CIFAR-10 FID 9.55), instead of the stronger VP-EDM and LDM settings used for the main results. I feel like the current experiments are not enough to show whether all these additional designs are actually needed to achieve the reported improvements on stronger models. A direct comparison with the previous spectral regularization method, together with some ablations on the stronger backbones, would help establish the actual contribution.
[1] Spectral Regularization for Diffusion Models. 2026.
Questions for authors
- Regarding the timestep-conditioned router, I wonder what the actual learned routing weights look like across different timesteps. Figure 3 only shows the manually designed scheduling priors. It would be interesting to visualize the final learned weights and see how much they differ from these priors.
- The method introduces several FFT, wavelet transform and statistical calculations during training. How much additional training time and GPU memory does MoSE require compared with the original diffusion backbone?
- In Page 6, the authors mention that the heavy-tailed distribution of wavelet coefficients breaks the Gaussian assumption of diffusion models. I'm a bit confused by this statement, since diffusion models do not require the underlying data distribution to be Gaussian. Could the authors clarify what Gaussian assumption is being referred to here?
Thank you for the constructive assessment and for recognizing the usefulness of combining complementary spectral experts in diffusion models. We provide clarification and additional evidence on spectral diversity, learned routing weights and computational overhead.
We would like to clarify that the energy experts are not designed to perform explicit distribution matching. Their purpose is to embed dataset-level statistics into the diffusion models. The Mahalanobis distance penalize deviations of predicted spectral features from the reference means under the covariance-weighted metric. The reference covariance determines the geometry and relative importance of spectral deviations.
We agree that minimizing the Mahalanobis penalty may reduce the spectral diversity. This motivates our combination of different experts to prevent one single expert from dominating the entire generation process. We evaluated and visualized the generated spectral mean and variance on both CIFAR10 dataset and RealPDEBench dataset. The heatmaps show that MoSE recovers both spectral mean and variance, instead of eliminating the diversity of the samples.
On the other hand, we apologize for the confusion caused by the notation in Eq. (14). The two Mahalanobis terms are evaluated on the predicted clean sample x, using the reference statistics μ and Σ. The kurtosis term compares the predicted sample x with its corresponding clean target x~. All three terms depend on the predicted sample and provide gradients to the neural networks.
We agree that Fourier and wavelet regularization have been explored previously, especially by [1]. Our core contribution is not the introduction of these transforms independently, but an organic combination of complementary spectral experts that embeds dataset-level spectral statistics into the diffusion models while preserving individual structure and diversity. We did not compare our method with [1] under the same training setting. However, on CIFAR-10 dataset, our method can improve FID from 1.97 to 1.78, while [1] improve FID from 1.84 to 1.83, which demonstrates that the combination of complementary spectral experts are essential instead of simply adding a regularization term.
We acknowledge the point that current ablation studies are primarily performed using vanilla DDPM rather than the stronger EDM and LDM. Our main experiments demonstrate that MoSE can improve stronger backbones, while the ablations mainly focus on the contribution of individual components on various but less expensive settings.
The learned weights could differ significantly from the given priors, since the given priors act only as a soft constraint, encouraging the Fourier experts at earlier timesteps and wavelet experts at later timesteps. We visualized the learned routing weights compared with the normalized given priors, which support the usefulness of the given priors.
We additionally conduct CIFAR-10 EDM fine-tuning experiment using four A100 40GB GPUs and ~40k optimizer steps. The mean optimizer step time increases from 606.18 ms for vanilla EDM to 717.45 ms for EDM + MoSE, corresponding to 18.4% additional training time per step. Peak GPU memory usage subtly from 2.362 to 2.364 GiB per GPU.
We agree with the reviewer that diffusion models do not require the data distribution to be Gaussian. The purpose of this statement is that wavelet coefficients in natural images and physical fields often exhibit sparse, heavy-tailed statistics that can not be fully characterized by Gaussian distribution (i.e. mean and variance are not enough to capture the properties of such distribution). This motivates including kurtosis term in the wavelet energy expert term, allowing the expert to capture higher-order spectral characteristics.
We will revise the statement accordingly and remove the suggestion that heavy-tailed data violates an assumption required by the diffusion process.
[1] Spectral Regularization for Diffusion Models