MoSE: Mixture of Spectral Experts in Diffusion Models

Diffusion models are usually trained with pixel-space denoising objectives, which constrain frequency content only indirectly. As a result, generated samples may appear realistic visually, while failing to match global spectral structure or localized details at multiple scales, leading to over-smoothed and detail-poor outputs. We introduce Mixture-of-Spectral-Experts (MoSE), a training-time regularizer that combines four spectral experts with diffusion models. Fourier and wavelet energy experts match dataset-level spectral statistics, while per-sample Fourier and wavelet experts preserve sample-specific structure. A timestep-conditioned hierarchical router adjusts the expert weights over the denoising process as the noise level changes. Across four image benchmarks, MoSE improves generation quality and achieves strong FID results with the strongest gains on CIFAR-10, ImageNet and CelebA-HQ. For physical-system tasks, it improves both Fourier-space and solution-error metrics on RealPDEBench dataset, demonstrating its generalizability to different datasets and tasks. These results show that introducing and balancing global and local spectral objectives improve the fidelity of generated samples in diffusion models for both image and scientific data generation.