acceptodds
Withdrawn from ICLR 2027

Different Noise Levels, Different Needs: Discovering Diffusion Phases for Efficient Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models apply a shared denoiser across noise levels, even though the importance of its components changes throughout denoising. Recent work has explored this heterogeneity through phase-aware student design. However, phase boundaries and student capacity are typically prescribed without directly measuring how the teacher's parameter importance evolves. We propose a diffusion-distillation framework that identifies denoising phases from time-dependent parameter-usage profiles, which quantify the importance of different parameter groups to the denoising objective. These profiles guide both phase partitioning and the allocation of a fixed student-capacity budget across phases and network depth. We evaluate our method using DiT and convolutional U-Net-based denoisers on CIFAR-10, ImageNet, FFHQ, and LSUN Bedrooms, observing nonuniform phase structure across model families and datasets. Notably, allocating student capacity according to this structure improves the quality-throughput trade-off over uniform allocation under matched parameter budgets. These results highlight the teacher's evolving parameter importance as a practical signal for efficient denoiser design.

Primary area:
generative models
Keywords:
diffusion models, knowledge distillation, efficient inference, phase specialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2rdyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper optimally partitions the noise trajectory into segments (phases), using a student of an appropriate size to denoise within each phase. It first proposes a method for identifying these phases. At each timestep, it examines how much the loss changes when the features of a specific group within the network are swapped between different samples. Collecting the importance scores of these groups yields a group-importance vector for each timestep, and phase boundaries are determined so that timesteps with similar vectors belong to the same phase. To this end, the method computes a correlation heatmap across all pairs of timesteps and uses dynamic programming to determine an appropriate overall partition. The resulting phases contain group sensitivity information, and the total parameter budget is allocated in proportion to this importance. In the blockwise setting, each phase’s student model is scaled down uniformly, while the layerwise setting additionally accounts for the importance of individual layers.

Critical strengths
  1. Automatically partitioning the groups using dynamic programming and assigning budgets to student models while minimizing the number of manually selected parameters is a reasonable approach.

  2. Although the total number of stored parameters remains the same, fewer parameters are executed at a time, improving execution speed.

Critical weaknesses
  1. There is insufficient evidence showing how well the teacher’s activation-permutation sensitivity predicts the capacity requirements of a compressed or retrained student. A small change in denoising loss when activations are replaced does not directly imply that less student capacity can be allocated to that part of the network. Furthermore, the analysis of the stability of sensitivity scores, phase boundaries, and capacity allocation across calibration samples and permutation seeds is insufficient.

  2. The proposed method does not consistently perform well. For ConvNets, the speed advantage over the original model is marginal, while performance degrades. For DiT, although the method is faster than other approaches, its performance is sometimes worse than that of DyDiT (ImageNet and LSUN) and TinyFusion (FFHQ).

Questions for authors
  1. If the trajectory is divided into only two or three phases, is such a complex partitioning algorithm necessary? How much does performance differ from a simple approach such as arbitrarily dividing the trajectory into three equal parts?

  2. How was the assumption that the teacher’s activation-permutation sensitivity predicts the student’s capacity requirements validated?

  3. For ConvNets in particular, the speed difference compared with the full model appears small. What causes this? When the speed improvement is small, does this not limit the effectiveness of the proposed idea?

Reviewer_T6uxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method, namely Phase-Aware Capacity-Efficient diffusion distillation (PACE), for allocating student models (with given budgets) to different diffusion phases during diffusion distillation. The phase-aware approach identifies phases based on the similarity in their usage profiles, resulting in contiguous phases along the sampling trajectory. Permutation feature importance is adopted to intervene intermediate activations in the denoiser. Based on the identified phases, respective phase-specialised student models are allocated. The proposed method was experimented with Diffusion Transformers and convolutional U-Net-based denoisers on various benchmarks including CIFAR-10, ImageNet, FFHQ, and LSUN Bedrooms.

Critical strengths
  • The paper studies an important and practical research problem

  • The paper is well-written and easy to follow

  • Extensive experiments across various denoising architectures and datasets were conducted.

Critical weaknesses
  • It is not clear the benefits achieved by the proposed phase-aware approach over the uniform one from the experimental results as the gains seem to happen at phases requiring more steps.

  • The proposed phase-aware splitting would be more convincing to be compared with simple settings like uniform splitting and random splitting

  • How are the phases stable under permutations?

  • Some technical details are missing, making evaluations difficult to validate. For instance, it is not clear how student models are trained (are they pre-trained or trained from scratch?)

  • The idea of step-aware diffusion has been explored in [A]. Although focusing on different tasks, the approach in [A] could be applied to allocate students at different steps and hence should be compared with the proposed approach.

  • Missing reference.

[A] Yang et al. "Denoising Diffusion Step-aware Models", ICLR2025

Questions for authors

Please answer questions/issues raised in the weaknesses section.

Reviewer_EupLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Phase-Aware Capacity-Efficient diffusion distillation (PACE), a framework for constructing noise-specialized student denoisers under a fixed aggregate parameter budget. PACE measures the sensitivity of a frozen teacher by permuting intermediate activation groups between examples at the same noise level. The resulting sensitivity profiles guide contiguous phase partitioning and capacity allocation across phases and network depth. Experiments demonstrate effectiveness.

Critical strengths
  1. The paper addresses a problem: different noise levels may require different computational resources, making uniform denoiser compression potentially suboptimal.

  2. The evaluation provides a broader assessment on four datasets.

Critical weaknesses
  1. Although the proposed problem is meaningful, but it is already well established in prior work. Noise-dependent specialization has been explored by eDiff-I and MEME, while timestep heterogeneity and adaptive computation have been studied by Addressing Negative Transfer in Diffusion Models and DyDiT. The novelty of PACE is limited. Authors need to introduce more specific contributions.

  2. The current experiments lack comparisons with equal-length partitions and randomly selected boundaries at the same phase count. It would also be useful to evaluate small perturbations of the PFI-derived boundaries and compare generation quality and throughput across different numbers of phases.

Questions for authors

At a fixed phase count and comparable inference cost, how does PACE compare with equal-length or randomly selected contiguous partitions?

Reviewer_DH3VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACE, a phase-aware diffusion distillation framework that identifies distinct denoising phases based on activation sensitivity and allocates different student model capacities to each phase. Experiments on image generation tasks demonstrate improved quality–efficiency trade-offs compared with several existing methods.

Critical strengths
  1. The motivation of this paper is strong.

  2. The proposed method is conceptually simple and easy to implement.

  3. The experiments show that the method is able to achieve better or comparable FID with higher throughput.

Critical weaknesses
  1. The discovery of denoising phases heavily relies on loss-based activation sensitivity estimates, which can be noisy and unreliable [1,2]. It remains unclear how robust the resulting phase boundaries are to variations in the calibration data and sensitivity estimates.

  2. The paper lacks a quantitative analysis of the computational overhead introduced by the proposed phase-discovery strategy, including activation permutation, repeated forward passes, and dynamic programming. It is therefore difficult to assess whether the efficiency gains during inference justify the additional offline profiling cost.

  3. Several ablation studies appear to be missing. For example, does the paper investigate the effect of the number of phases (K) on generation quality and throughput? It would also be useful to examine the sensitivity of the results to the calibration set size, the choice of profile similarity metric, and the separation weight, to establish whether the proposed phase-discovery strategy is robust to these design choices.

  4. The evaluation remains somewhat limited. Although the paper reports quality–throughput trade-offs, it does not provide a comprehensive evaluation of end-to-end efficiency that accounts for the offline profiling and the training of multiple student models.

  5. After phase discovery, how quickly do the student models assigned to different phases converge during training? Since the proposed method allocates different parameter budgets to different phases, their training requirements may differ substantially. An analysis of convergence speed, training iterations, and total computational cost for each phase-specific student would help determine whether the proposed capacity allocation also leads to practical training efficiency.

References: [1] Table Target Field for Reduced Variance Score Estimation in Diffusion Models. [2] Variance Reduction of Diffusion Model’s Gradients with Taylor Approximation-Based Control Variate.

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.