Different Noise Levels, Different Needs: Discovering Diffusion Phases for Efficient Distillation

Diffusion models apply a shared denoiser across noise levels, even though the importance of its components changes throughout denoising. Recent work has explored this heterogeneity through phase-aware student design. However, phase boundaries and student capacity are typically prescribed without directly measuring how the teacher's parameter importance evolves. We propose a diffusion-distillation framework that identifies denoising phases from time-dependent parameter-usage profiles, which quantify the importance of different parameter groups to the denoising objective. These profiles guide both phase partitioning and the allocation of a fixed student-capacity budget across phases and network depth. We evaluate our method using DiT and convolutional U-Net-based denoisers on CIFAR-10, ImageNet, FFHQ, and LSUN Bedrooms, observing nonuniform phase structure across model families and datasets. Notably, allocating student capacity according to this structure improves the quality-throughput trade-off over uniform allocation under matched parameter budgets. These results highlight the teacher's evolving parameter importance as a practical signal for efficient denoiser design.