- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper optimally partitions the noise trajectory into segments (phases), using a student of an appropriate size to denoise within each phase. It first proposes a method for identifying these phases. At each timestep, it examines how much the loss changes when the features of a specific group within the network are swapped between different samples. Collecting the importance scores of these groups yields a group-importance vector for each timestep, and phase boundaries are determined so that timesteps with similar vectors belong to the same phase. To this end, the method computes a correlation heatmap across all pairs of timesteps and uses dynamic programming to determine an appropriate overall partition. The resulting phases contain group sensitivity information, and the total parameter budget is allocated in proportion to this importance. In the blockwise setting, each phase’s student model is scaled down uniformly, while the layerwise setting additionally accounts for the importance of individual layers.
Critical strengths
-
Automatically partitioning the groups using dynamic programming and assigning budgets to student models while minimizing the number of manually selected parameters is a reasonable approach.
-
Although the total number of stored parameters remains the same, fewer parameters are executed at a time, improving execution speed.
Critical weaknesses
-
There is insufficient evidence showing how well the teacher’s activation-permutation sensitivity predicts the capacity requirements of a compressed or retrained student. A small change in denoising loss when activations are replaced does not directly imply that less student capacity can be allocated to that part of the network. Furthermore, the analysis of the stability of sensitivity scores, phase boundaries, and capacity allocation across calibration samples and permutation seeds is insufficient.
-
The proposed method does not consistently perform well. For ConvNets, the speed advantage over the original model is marginal, while performance degrades. For DiT, although the method is faster than other approaches, its performance is sometimes worse than that of DyDiT (ImageNet and LSUN) and TinyFusion (FFHQ).
Questions for authors
-
If the trajectory is divided into only two or three phases, is such a complex partitioning algorithm necessary? How much does performance differ from a simple approach such as arbitrarily dividing the trajectory into three equal parts?
-
How was the assumption that the teacher’s activation-permutation sensitivity predicts the student’s capacity requirements validated?
-
For ConvNets in particular, the speed difference compared with the full model appears small. What causes this? When the speed improvement is small, does this not limit the effectiveness of the proposed idea?