- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Align Before Distilling (ABD), a framework that combines structured pruning with one-step diffusion distillation to obtain compact and efficient diffusion models. The authors observe that directly applying one-step distillation to a pruned diffusion model can lead to severe performance degradation or training failure. To address this issue, ABD introduces a teacher-aligned repair stage between pruning and distillation, where the pruned student is trained to match the teacher's predictions on noisy real-data inputs. The authors evaluate their approach across several diffusion distillation frameworks, including Diff-Instruct, SiD-LSG, and SiDA.
Critical strengths
The proposed method is simple, intuitive, and easy to understand.
Critical weaknesses
- Significant performance degradation after pruning.
Although ABD reduces the model size, the resulting compact models exhibit noticeable degradation in generation quality compared to their dense one-step counterparts. For example, on ImageNet-512, the FID increases from 2.228 for dense SiDA to 3.12 and 4.26 for the approximately 20% and 30% pruned models, respectively. This raises concerns about the quality-efficiency trade-off achieved by the proposed method.
- Limited methodological novelty.
Both the structured pruning and one-step distillation techniques are largely based on existing methods. The proposed repair stage essentially employs a conventional teacher-student regression objective to align the pruned model with the teacher's predictions. Therefore, the overall methodological contribution appears relatively incremental.
Questions for authors
-
In Table 6, why are the experimental comparisons not conducted under matched computational budgets? Could the authors provide compute-matched comparisons to better isolate the effectiveness of the proposed repair stage?
-
Since the auxiliary score network remains dense during distillation, to what extent does ABD reduce training-time memory consumption and computational overhead compared to dense one-step distillation?