acceptodds
Withdrawn from ICLR 2027

Align Before Distilling: Teacher-Aligned Repair for Compact One-Step Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pruning and step distillation are typically applied separately: pruning reduces the network size of diffusion models, while step distillation reduces their sampling cost. When combined, however, pruning can leave the student poorly initialized for distillation, which often causes step distillation to fail. In our EDM2-XS experiment, direct SiDA training from a pruned checkpoint produces unusable samples. We introduce Align Before Distilling (ABD), a teacher-aligned repair stage before step distillation: the compact model matches teacher outputs on noisy real-image latents, then uses the repaired weights to initialize one-step training. We evaluate this procedure with SD1.5/SiD-LSG, EDM2-XS/SiDA, and CIFAR-10 EDM/Diff-Instruct. On ImageNet-512, the original EDM2-XS teacher uses 124.713M parameters and 63 network evaluations at FID 3.53. Our 20% target reduction (20.7571% actual) model reaches FID 3.12 with 98.826M parameters and one evaluation; the 30% target reduction (29.4146% actual) model reaches 4.26 with 88.029M parameters. These compact models trade quality for size relative to dense one-step SiDA at FID 2.228. The SD1.5 model reduces parameters by 23.9448% and reaches FID 18.7, while the 36.807M-parameter CIFAR-10 student reaches FID 5.32 with Diff-Instruct.

Primary area:
generative models
Keywords:
diffusion distillation, one-step generation, structured pruning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zVXsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Align Before Distilling (ABD), a framework that combines structured pruning with one-step diffusion distillation to obtain compact and efficient diffusion models. The authors observe that directly applying one-step distillation to a pruned diffusion model can lead to severe performance degradation or training failure. To address this issue, ABD introduces a teacher-aligned repair stage between pruning and distillation, where the pruned student is trained to match the teacher's predictions on noisy real-data inputs. The authors evaluate their approach across several diffusion distillation frameworks, including Diff-Instruct, SiD-LSG, and SiDA.

Critical strengths

The proposed method is simple, intuitive, and easy to understand.

Critical weaknesses
  • Significant performance degradation after pruning.

Although ABD reduces the model size, the resulting compact models exhibit noticeable degradation in generation quality compared to their dense one-step counterparts. For example, on ImageNet-512, the FID increases from 2.228 for dense SiDA to 3.12 and 4.26 for the approximately 20% and 30% pruned models, respectively. This raises concerns about the quality-efficiency trade-off achieved by the proposed method.

  • Limited methodological novelty.

Both the structured pruning and one-step distillation techniques are largely based on existing methods. The proposed repair stage essentially employs a conventional teacher-student regression objective to align the pruned model with the teacher's predictions. Therefore, the overall methodological contribution appears relatively incremental.

Questions for authors
  1. In Table 6, why are the experimental comparisons not conducted under matched computational budgets? Could the authors provide compute-matched comparisons to better isolate the effectiveness of the proposed repair stage?

  2. Since the auxiliary score network remains dense during distillation, to what extent does ABD reduce training-time memory consumption and computational overhead compared to dense one-step distillation?

Reviewer_NssDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Align Before Distilling (ABD), a method for combining structured pruning with one-step distillation for diffusion models. After pruning a pretrained diffusion model, the compact student regresses teacher predictions on noisy real data at a single high noise level. The repaired weights then initialize an existing distillation method. Experiments cover EDM2/SiDA on ImageNet-512, SD1.5/SiD-LSG, and EDM/Diff-Instruct on CIFAR-10. The resulting generators offer reduced parameter counts with some quality degradation relative to dense one-step counterparts.

Critical strengths

Combining pruning and step distillation is an interesting research problem. To address this problem, the paper proposes a simple procesdure that can be inserted before existing diffusion distillation methods without changing their objectives.

Critical weaknesses
  1. This submission clearly violates the ICLR format requirements as it (1) removes the page header in the ICLR template and (2) does not include the mandatory AI use statement.
  2. The contribution is incremental. The repair loss is a standard component of diffusion model compression. As I understand, the main contribution of ABD is concentrating this recovery at a selected high noise level and using it only as initialization for subsequent distillation with existing diffusion distillation method.
  3. The recovery-efficiency conclusions are not supported by controlled comparisons, as conventional recovery and ABD repair use different number of images and runs for different walk-clock times.
Questions for authors

Could the authors provide a matched-total-compute (number of images and wall-clock time) comparison between (1) ABD, (2) short conventional teacher-regression recovery followed by the same distillation, and (3) direct distillation?

Reviewer_RNErOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Another reviewer has pointed violations of the paper, which imply desk rejection. Therefore, no reviews are made.

Critical strengths

N/A

Critical weaknesses

Another reviewer has pointed violations of the paper, which imply desk rejection. Therefore, no reviews are made.

Questions for authors

Another reviewer has pointed violations of the paper, which imply desk rejection. Therefore, no reviews are made.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.