acceptodds
Under review as a conference paper at ICLR 2027

MoSE: Mixture of Spectral Experts in Diffusion Models

Abstract

Diffusion models are usually trained with pixel-space denoising objectives, which constrain frequency content only indirectly. As a result, generated samples may appear realistic visually, while failing to match global spectral structure or localized details at multiple scales, leading to over-smoothed and detail-poor outputs. We introduce Mixture-of-Spectral-Experts (MoSE), a training-time regularizer that combines four spectral experts with diffusion models. Fourier and wavelet energy experts match dataset-level spectral statistics, while per-sample Fourier and wavelet experts preserve sample-specific structure. A timestep-conditioned hierarchical router adjusts the expert weights over the denoising process as the noise level changes. Across four image benchmarks, MoSE improves generation quality and achieves strong FID results with the strongest gains on CIFAR-10, ImageNet and CelebA-HQ. For physical-system tasks, it improves both Fourier-space and solution-error metrics on RealPDEBench dataset, demonstrating its generalizability to different datasets and tasks. These results show that introducing and balancing global and local spectral objectives improve the fidelity of generated samples in diffusion models for both image and scientific data generation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Diffusion Models, Generative Models, Fourier Transform, Wavelet Transform, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_zGHdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoSE, a training-time spectral regularization framework for diffusion models. The method introduces four spectral experts, including Fourier and wavelet energy experts for dataset-level statistics, and two per-sample experts for spectral consistency. A timestep-conditioned router is used to dynamically balance the four losses during training. Experiments on several image generation benchmarks and physical system tasks show improvements in generation quality and spectral fidelity.

Critical strengths
  • The idea of combining Fourier and wavelet regularization is reasonable, since they capture different aspects of frequency information. The distinction between dataset-level and per-sample spectral constraints is also interesting.
  • The results on image generation are promising with consistent improvement in FID on all four datasets. Also, since the method only adds regularization during training, it does not introduce extra inference overhead.
  • The evaluation covers both image generation and physical data, which is appreciated. The ablation studies in Appendix C.2 are also relatively comprehensive, including different combinations of experts and routing strategies.
Critical weaknesses
  • My main concern is about the formulation of the spectral energy experts. In Eq. 11, the Fourier energy expert minimizes the Mahalanobis distance between the predicted sample's spectral features and the mean statistics of the real data. However, I don't quite understand why minimizing such a sample-wise distance would encourage matching the whole data distribution. For example, if we only optimize this loss, it would simply encourage all generated samples to move toward the mean of the spectral features, rather than matching their actual distribution. This seems different from the claimed dataset-level distribution matching. Besides, in Eq. 14, the last two terms of the wavelet energy expert are computed using the ground-truth sample (x), instead of the predicted sample (x hat). If the dataset statistics are fixed, these two terms would not provide gradients to the denoising network as intended. I'm not sure if this is just a typo or something wrong with the actual formulation, but I think the authors should clarify these issues, as they directly relate to the core methodology of the paper.
  • Another concern is the novelty and whether the proposed combination is really necessary. Spectral regularization for diffusion models has already been explored in previous works, especially [1], which also uses Fourier and wavelet losses during diffusion training. MoSE mainly extends this idea by combining more spectral losses and introducing a learned router. However, there is no direct comparison with this closely related method. Although the authors provide ablations on different experts and routing strategies, these are mainly conducted on a vanilla DDPM with a much weaker baseline (e.g., CIFAR-10 FID 9.55), instead of the stronger VP-EDM and LDM settings used for the main results. I feel like the current experiments are not enough to show whether all these additional designs are actually needed to achieve the reported improvements on stronger models. A direct comparison with the previous spectral regularization method, together with some ablations on the stronger backbones, would help establish the actual contribution.


[1] Spectral Regularization for Diffusion Models. 2026.

Questions for authors
  • Regarding the timestep-conditioned router, I wonder what the actual learned routing weights look like across different timesteps. Figure 3 only shows the manually designed scheduling priors. It would be interesting to visualize the final learned weights and see how much they differ from these priors.
  • The method introduces several FFT, wavelet transform and statistical calculations during training. How much additional training time and GPU memory does MoSE require compared with the original diffusion backbone?
  • In Page 6, the authors mention that the heavy-tailed distribution of wavelet coefficients breaks the Gaussian assumption of diffusion models. I'm a bit confused by this statement, since diffusion models do not require the underlying data distribution to be Gaussian. Could the authors clarify what Gaussian assumption is being referred to here?
1 comment

Thank you for the constructive assessment and for recognizing the usefulness of combining complementary spectral experts in diffusion models. We provide clarification and additional evidence on spectral diversity, learned routing weights and computational overhead.

  1. Clarification of the Mahalanobis distance and Eq. (14)

We would like to clarify that the energy experts are not designed to perform explicit distribution matching. Their purpose is to embed dataset-level statistics into the diffusion models. The Mahalanobis distance penalize deviations of predicted spectral features from the reference means under the covariance-weighted metric. The reference covariance determines the geometry and relative importance of spectral deviations.

We agree that minimizing the Mahalanobis penalty may reduce the spectral diversity. This motivates our combination of different experts to prevent one single expert from dominating the entire generation process. We evaluated and visualized the generated spectral mean and variance on both CIFAR10 dataset and RealPDEBench dataset. The heatmaps show that MoSE recovers both spectral mean and variance, instead of eliminating the diversity of the samples.

On the other hand, we apologize for the confusion caused by the notation in Eq. (14). The two Mahalanobis terms are evaluated on the predicted clean sample x\mathbf{x}, using the reference statistics μ\mu and Σ\Sigma. The kurtosis term compares the predicted sample x\mathbf{x} with its corresponding clean target x~\tilde{\mathbf{x}}. All three terms depend on the predicted sample and provide gradients to the neural networks.

  1. Comparison with existing spectral regularization

We agree that Fourier and wavelet regularization have been explored previously, especially by [1]. Our core contribution is not the introduction of these transforms independently, but an organic combination of complementary spectral experts that embeds dataset-level spectral statistics into the diffusion models while preserving individual structure and diversity. We did not compare our method with [1] under the same training setting. However, on CIFAR-10 dataset, our method can improve FID from 1.97 to 1.78, while [1] improve FID from 1.84 to 1.83, which demonstrates that the combination of complementary spectral experts are essential instead of simply adding a regularization term.

We acknowledge the point that current ablation studies are primarily performed using vanilla DDPM rather than the stronger EDM and LDM. Our main experiments demonstrate that MoSE can improve stronger backbones, while the ablations mainly focus on the contribution of individual components on various but less expensive settings.

  1. Learned routing weights

The learned weights could differ significantly from the given priors, since the given priors act only as a soft constraint, encouraging the Fourier experts at earlier timesteps and wavelet experts at later timesteps. We visualized the learned routing weights compared with the normalized given priors, which support the usefulness of the given priors.

  1. Computational overhead

We additionally conduct CIFAR-10 EDM fine-tuning experiment using four A100 40GB GPUs and ~40k optimizer steps. The mean optimizer step time increases from 606.18 ms for vanilla EDM to 717.45 ms for EDM + MoSE, corresponding to 18.4% additional training time per step. Peak GPU memory usage subtly from 2.362 to 2.364 GiB per GPU.

  1. Clarification of the Gaussian assumption

We agree with the reviewer that diffusion models do not require the data distribution to be Gaussian. The purpose of this statement is that wavelet coefficients in natural images and physical fields often exhibit sparse, heavy-tailed statistics that can not be fully characterized by Gaussian distribution (i.e. mean and variance are not enough to capture the properties of such distribution). This motivates including kurtosis term in the wavelet energy expert term, allowing the expert to capture higher-order spectral characteristics.

We will revise the statement accordingly and remove the suggestion that heavy-tailed data violates an assumption required by the diffusion process.

[1] Spectral Regularization for Diffusion Models

Reviewer_iK1QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that pixel-space denoising objectives only weakly constrain the spectral statistics of generated samples. To address this issue, the authors propose MoSE, a training-time regularizer that introduces four spectral losses applied to the predicted clean sample. Two energy losses use Mahalanobis distances to compare Fourier and wavelet statistics with dataset-level means and covariances. The other two per-sample losses match the Fourier amplitude and phase, as well as the wavelet detail coefficients, of each prediction to its corresponding target. A small router, conditioned on the timestep, noisy input, and a hand-designed schedule, assigns weights to the four losses. The method is evaluated on CIFAR-10, ImageNet-64, CelebA-HQ, and FFHQ, as well as five tasks from RealPDEBench. The results show lower FID scores than the EDM and LDM-4 baselines, while the physical tasks yield mixed but often favorable error metrics.

Critical strengths
  • The paper distinguished two design axes: global versus localized representations and dataset-level versus sample-specific matching. This leads to a structured family of losses rather than a single frequency-domain penalty. Since the regularizer is applied only during training, the inference cost and model architecture remain unchanged. The approach also applies to pixel-space models, latent models, and physical fields without modification.
  • The ablation study in Appendix C.2 examines individual experts, pairs of experts along each design axis, leave-one-out variants, and several routing schemes. The experiments are repeated across multiple seeds on CIFAR-10 and the Burgers equation. Spectral errors decrease consistently as more experts are combined. This suggests that the four losses provide complementary information.
Critical weaknesses
  • The training objective of router is unspecified. The energy losses do not match distributions. In particular, they pull individual samples toward dataset means, potentially reducing diversity rather than reproducing spectral statistics. Batch-level matching would better align with the stated goal. In addition, the wavelet energy loss applies a Gaussian Mahalanobis distance to heavy-tailed coefficients without explaining how the high-dimensional covariance is estimated or inverted. Note that Propositions 1 and 2 largely restate the definition of Mahalanobis distance and kurtosis without explaining why the losses should help.
  • The experiments do not provide sufficiently controlled baseline comparisons or consistently convincing improvements. As shown in Tables 1 and 2, the reported FID with MoSE gains are relatively modest, while spectral fidelity, the primary motivation of the paper, is not directly evaluated in the main benchmarks. In addition, the closest prior work and simpler spectral regularization methods are not included. Overall, the current experiments do not clearly establish the performance advantage of MoSE over appropriate baselines.
  • Section 3.2 defines transforms on xtx_t, whereas the losses operate on the predicted clean sample. Equation (14) mixes xx and x~\tilde{x} without explanation. The application of MoSE to conditional forecasting is not described. These issues make the results difficult to verify or reproduce.
  • Table 2 contains numbered citations (e.g., [21], [41], [77]) inconsistent with the paper's author-year references.
Questions for authors
  • The router input is 37-dimensional. How the noisy sample xtx_t summarized into the vector?
  • How is the covariance of the full wavelet coefficient vector in Eq. (14) estimated and inverted?
  • Do the energy losses reduce sample diversity?
2 comments

Thank you for the detailed assessment. We address the concerns below and provide additional results of spectral statistics and comparisons.

  1. Clarification of the Mahalanobis distance

We would like to clarify that the energy experts are not designed to perform explicit distribution matching. Their purpose is to embed dataset-level statistics into the diffusion models. The Mahalanobis distance penalize deviations of predicted spectral features from the reference means under the covariance-weighted metric. The reference covariance determines the geometry and relative importance of spectral deviations.

We agree that minimizing the Mahalanobis penalty may reduce the spectral diversity. This motivates our combination of different experts to prevent one single expert from dominating the entire generation process. We evaluated and visualized the generated spectral mean and variance on both CIFAR10 dataset and RealPDEBench dataset. The heatmaps show that MoSE recovers both spectral mean and variance, instead of eliminating the diversity of the samples.

  1. Explanation of Eq. (14)

We apologize for the confusion caused by the notation in Eq. (14). The two Mahalanobis terms are evaluated on the predicted clean sample x\mathbf{x}, using the reference statistics μ\mu and Σ\Sigma. The kurtosis term compares the predicted sample with its corresponding clean target x~\tilde{\mathbf{x}}. All three terms depend on the predicted sample x\mathbf{x} and provide gradients to the neural networks.

  1. Implementation details

We apologize for the insufficient implementation details concerning the router. The router is a lightweight 37 -> 32 -> 5 MLP with SiLU activation. Its 37-dimensional input consists of:

  • Four timestep features: (τ,τ2,sin⁡(πτ),cos⁡(πτ))(\tau,\tau^2,\sin(\pi\tau),\cos(\pi\tau)).
  • 16 features from adaptive average pooling of the channel-averaged, RMS-normalized noisy input xt\mathbf{x}_t to a 4x4 matrix.
  • 16 features from adaptive average pooling of the corresponding channel-average squared normalized input.
  • One log-transformed global RMS feature.
  1. Covariance estimation

We apologize that Eq. (14) does not accurately describe the implementation of the covariance matrix. We apply a two-level Haar transform and construct per-channel, per-orientation, per-level statistical feature vectors consisting of:

  • Log mean subband power.
  • Log contered subband kurtosis.

We estimate separate reference mean vectors and covariance matrices for the log-power and log-kurtosis features from the training data. For these low-dimensional features, the covariance estimate uses shrinkage toward its diagonal (with coefficient 0.05), followed by diagonal regularization (with ϵ=10−4\epsilon=10^{-4}). The precision matrix is computed using the Moore–Penrose pseudoinverse. This avoids calculating or inverting a covariance matrix over the full spatial wavelet coefficients.

  1. Comparison with existing spectral regularization

We agree that Fourier and wavelet regularization have been explored previously, especially by [1]. Our core contribution is not the introduction of these transforms independently, but an organic combination of complementary spectral experts that embeds dataset-level spectral statistics into the diffusion models while preserving individual structure and diversity. We did not compare our method with [1] under the same training setting. However, on CIFAR-10 dataset, our method can improve FID from 1.97 to 1.78, while [1] improve FID from 1.84 to 1.83, which demonstrates that the combination of complementary spectral experts are essential instead of simply adding a regularization term.

  1. Formatting error

The inconsistent numbered references in Table 2 were introduced during conversion to the ICLR template. We apologize for this formatting error and will correct the citations to the author–year style used throughout the paper.

[1] Spectral Regularization for Diffusion Models

I'd like to thank the authors for their response. Although the rebuttal addresses several concerns related to implementation, my major concerns about the novelty of the proposed objective and the empirical justification of the proposed approach remain unresolved. Thus, I'll maintain my original score.

Reviewer_DxZjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether adding spectral losses to diffusion training improves image generation and physical forecasting. MoSE applies Fourier and wavelet losses to each predicted clean sample, comparing it with both dataset-level statistics and its clean target, and learns to weight these losses over the denoising process. MoSE improves FID over the corresponding backbone on four image benchmarks and achieves lower Fourier-space forecasting error on several RealPDEBench tasks. Ablations on CIFAR-10 and Burgers examine the contributions of the individual losses and the learned router.

Critical strengths
  • The image results are consistent across pixel-space and latent diffusion models. FID improves on all four benchmarks, and the gains on CelebA-HQ and FFHQ are accompanied by improvements in both precision and recall.
  • Table 5 gives a concrete reason to combine the two transforms. On CIFAR-10, the Fourier-only variant performs better on Fourier error, while the wavelet-only variant performs better on wavelet error. Combining them improves both metrics even with uniform weights.
Critical weaknesses
  • The main concern is the link between the spectral discrepancies being measured and the image defects the method is intended to address. Figure 1, which motivates the paper, compares single images with different content across methods (e.g., a bird for real data and a bear, a mountain, or a car for the generators). Because the spectrum of a single image depends strongly on its content, the differences shown cannot be attributed to the generative model, and it is unclear which of them correspond to over-smoothed textures or missing detail. Table 5 shows that spectral errors and FID improve together on CIFAR-10, yet the paper provides no visual comparison between MoSE and its backbone. This leaves the central motivation for the image experiments largely untested.
  • The claim about matching dataset-level spectral statistics is broader than the evidence supports. The Mahalanobis terms in the energy losses penalize each prediction’s distance from a reference mean, using the data covariance to weight deviations. Predictions with too little variation can therefore receive a low penalty. The reported errors compare average spectra and subband energies, so they leave open whether MoSE also reproduces the variation across real samples. The higher recall on CelebA-HQ and FFHQ is measured in Inception feature space and does not directly address this.
  • Given earlier work on spectral losses for diffusion, much of the methodological contribution rests on how the losses are weighted over the denoising process. Uniform weighting already accounts for most of the FID gain in the CIFAR-10 ablation, and the remaining difference from the full router is within about one standard deviation. Routing helps more on Burgers, but Figure 3 shows only the prescribed schedule. The paper offers little insight into what weighting the router learns or when it improves on a simpler combination.
Questions for authors
  • Could you show spectra averaged over many samples per method, or compared on matched content, so that the differences in Figure 1 can be attributed to the models rather than to image content? It would also help to see a few baseline and MoSE samples generated from the same initial noise, shown alongside their spectra, to indicate which discrepancies correspond to visible loss of detail.
  • Does MoSE also bring the spread of spectral features across generated samples closer to that of real data? Even a comparison for a few frequency bands would clarify how to interpret the dataset-level matching claim.
  • How do the learned Fourier and wavelet weights compare with the prescribed schedule over the course of denoising, particularly on Burgers, where the benefit of routing is clearer?
2 comments

Thank you for the detailed assessment. We address the concerns below and provide additional results of spectral statistics and spectral discrepancies.

  1. Clarification of Figure 1

We apologize that the Figure 1 was not clear. The Fourier spectra and radial power curves in Figure 1 are computed over the entire datasets, not from the individual images shown in the first row. We agree that dataset-averaged spectra alone cannot identify which spectral discrepancies directly correspond to over-smoothed textures or missing detail. Figure 1 is intended to motivate the importance of spectral statistics, rather than provide a causal evaluation of image quality.

  1. Spectral Evaluation

We agree that minimizing the Mahalanobis penalty may reduce the spectral diversity. This motivates our combination of different experts to prevent one single expert from dominating the entire generation process. We evaluated and visualized the generated spectral mean and variance on both CIFAR10 dataset and RealPDEBench dataset. The heatmaps show that MoSE recovers both spectral mean and variance, instead of eliminating the diversity of the samples.

  1. Learned routing weights

The learned weights could differ significantly from the given priors, since the given priors act only as a soft constraint, encouraging the Fourier experts at earlier timesteps and wavelet experts at later timesteps. We visualized the learned routing weights compared with the normalized given priors, which support the usefulness of the given priors.

Thank you for the response. While these help clarify some points, I believe the paper still requires substantial improvement in its motivation and experimental support. I will maintain my score.

Reviewer_An2pOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mixture of Spectral Experts (MoSE), a training-time regularization framework for diffusion models. It combines four spectral objectives based on Fourier and wavelet transforms, covering both dataset-level statistics and per-sample reconstruction. A timestep-conditioned hierarchical router adaptively weights these objectives during training. Experiments show improvements in FID across multiple image-generation benchmarks and competitive results on physical-system prediction tasks.

Critical strengths

Practical and flexible framework. MoSE combines global Fourier information and localized wavelet information without modifying the underlying diffusion architecture. Its training-time-only design makes it potentially applicable to different diffusion backbones.

Encouraging empirical results. The method consistently improves image-generation FID over its corresponding backbones across four benchmarks. The component and routing ablations also provide evidence that combining spectral objectives is beneficial.

Critical weaknesses

Dataset-level distribution matching is not fully justified. The Fourier energy expert minimizes the Mahalanobis distance between individual generated samples and dataset statistics. However, minimizing this objective can encourage spectral features to concentrate around the dataset mean rather than reproduce the full data distribution. This raises questions about whether the proposed objective actually achieves the claimed distribution-level matching.

The effectiveness of learned routing needs stronger justification. Since the router minimizes a weighted sum of expert losses, it may favor objectives with smaller values rather than those most beneficial to generation quality. Moreover, the improvement over uniform routing in the CIFAR-10 ablation is relatively modest (FID 9.12 to 9.04). The benefit of learned routing over simpler weighting strategies is therefore not entirely convincing.

Questions for authors

Distribution matching: Can the authors clarify why sample-wise Mahalanobis regularization should preserve dataset-level spectral diversity? Additional comparisons of generated and real spectral feature distributions, including covariance statistics, would strengthen this claim.

Wavelet energy implementation: In Eq. (14), the notation involving (x) and (\tilde{x}) is ambiguous. If the Mahalanobis terms are evaluated on the ground-truth sample, they would provide no gradient to the diffusion model. Could the authors clarify the exact inputs and gradient flow of each term?

Routing behavior: Could the authors visualize the learned expert weights across diffusion timesteps and explain how the router avoids favoring easier losses rather than more useful ones?

Computational overhead: Please report the additional computational cost introduced by Fourier and wavelet transforms, including training time per iteration, GPU memory consumption, and the overhead at different image resolutions. Although the spectral experts are described as training-time-only, measured inference latency relative to the original backbone would help verify that deployment efficiency remains unchanged.

1 comment

Thank you for your constructive assessment and for recognizing the empirical improvements of MoSE. We address the concerns below and provide additional results of spectral statistics, learned routing weights, and computational cost.

  1. Clarification of the Mahalanobis distance

We would like to clarify that the energy experts are not designed to perform explicit distribution matching. Their purpose is to embed dataset-level statistics into the diffusion models. The Mahalanobis distance penalize deviations of predicted spectral features from the reference means under the covariance-weighted metric. The reference covariance determines the geometry and relative importance of spectral deviations.

We agree that minimizing the Mahalanobis penalty may reduce the spectral diversity. This motivates our combination of different experts to prevent one single expert from dominating the entire generation process. We evaluated and visualized the generated spectral mean and variance on both CIFAR10 dataset and RealPDEBench dataset. The heatmaps show that MoSE recovers both spectral mean and variance, instead of eliminating the diversity of the samples.

  1. Explanation of Eq. (14)

We apologize for the confusion caused by the notation in Eq. (14). The two Mahalanobis terms are evaluated on the predicted clean sample x\mathbf{x}, using the reference statistics μ\mu and Σ\Sigma. The kurtosis term compares the predicted sample with its corresponding clean target x~\tilde{\mathbf{x}}. All three terms depend on the predicted sample x\mathbf{x} and provide gradients to the neural networks.

  1. Learned routing weights

The learned weights could differ significantly from the given priors, since the given priors act only as a soft constraint, encouraging the Fourier experts at earlier timesteps and wavelet experts at later timesteps. We visualized the learned routing weights compared with the normalized given priors, which support the usefulness of the given priors.

  1. Computational overhead

We additionally conduct CIFAR-10 EDM fine-tuning experiment using four A100 40GB GPUs and ~40k optimizer steps. The mean optimizer step time increases from 606.18 ms for vanilla EDM to 717.45 ms for EDM + MoSE, corresponding to 18.4% additional training time per step. Peak GPU memory usage subtly from 2.362 to 2.364 GiB per GPU.

MoSE is applied only during training. The sampling architecture, backbone, and number of denoising evaluations remain unchanged, so no additional spectral computations are involved at inference.

We will include these new results and visualizations in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.