- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Summary
This paper proposes Score Bridge Distillation (SBD), a method for one-step diffusion distillation based on distribution matching. Motivated by conflicting student-update gradients across noise levels, SBD periodically constructs a teacher-guided bridge target from the current student score estimate through teacher alignment and real-data denoising. The student is then trained to match the resulting intermediate distribution. The authors analyze cross-noise gradient conflicts and evaluate SBD on unconditional image generation, class-conditional ImageNet generation, and text-to-image generation with SDXL, reporting reduced gradient interference and improved generation quality.
Critical strengths
Strengths
- The core idea of adapting the matching target to the student’s current distribution is conceptually simple and easy to follow.
- The method achieves substantial improvements on several benchmarks. The evaluation covers class-conditional generation, unconditional generation, and high-resolution text-to-image generation with SDXL, providing broad empirical support for its effectiveness.
Critical weaknesses
Weaknesses
- Section 6.3 would benefit from deeper analysis of the dataset-dependent ablation results. For example, removing teacher alignment has a much larger impact on FFHQ than on CIFAR-10, yet this difference is not sufficiently explained.
- Although the compute-matched ablation is informative, SBD’s additional runtime and memory overhead remain unquantified. Reporting training GPU-hours and peak GPU memory relative to the baseline would clarify the practical efficiency trade-off.
Questions for authors
Questions
- Table 4 shows that combining teacher alignment and real-data denoising yields the best results. Since real-data denoising alone can, in principle, learn the data score, could the authors explain the additional benefit of teacher alignment?
- The component gains in Table 4 vary substantially across datasets. For example, teacher alignment provides a much larger improvement on FFHQ than on CIFAR-10. Could the authors report comparable component ablations on ImageNet and SDXL to assess whether these findings generalize?
4 comments
W2. Training time and memory cost
Thank you for raising the issue of training cost. We added measurements of training time and peak GPU memory. For the same number of student updates, SBD increases total training time by 8.0%–28.6%, including the full cost of bridge adaptation.
| Dataset | Baseline GPU hours | SBD GPU hours, including bridge adaptation | Time increase | Baseline peak memory (GB/GPU) | SBD peak memory (GB/GPU) |
|---|---|---|---|---|---|
| CIFAR-10 | 50 | 54 | +8.0% | 4.878 | 5.298 |
| FFHQ | 44 | 48 | +9.1% | 11.575 | 12.040 |
| AFHQv2 | 45 | 50 | +11.1% | 10.686 | 11.152 |
| LSUN-Bedroom | 21 | 27 | +28.6% | ≈28.0 | ≈32.1 |
| LSUN-Cat | 21 | 26 | +23.8% | ≈28.0 | ≈31.8 |
Simply training DMD for longer does not reproduce the gains obtained with this additional compute. In the equal-GPU-time controls, DMD is trained with the same compute budget as SBD. On FFHQ and CIFAR-10, this extended DMD training reaches FIDs of 11.03 and 4.27, respectively, compared with 6.14 and 3.29 for SBD. In these two settings, using the additional compute for bridge adaptation is therefore more effective than allocating it to further DMD training. We will add these resource statistics to the revised manuscript and discuss the trade-off between generation quality and training cost together with the equal-GPU-time results.
Q1. Benefit of teacher alignment beyond real-data denoising
Thank you for the question. With sufficient training, real-data denoising can recover the data score. In SBD, however, the bridge is initialized from the current fake-score model in each cycle and adapted for finite steps. The key is therefore to construct an effective denoising target within this limited adaptation budget. Teacher alignment directly transfers the denoising mapping learned by the pretrained teacher, allowing the bridge to reuse this knowledge rather than learning the target mapping solely from sampled real-data supervision. This guidance is especially valuable for more complex distributions and tasks requiring finer visual detail.
The ablation results support this explanation. On CIFAR-10, real-data denoising alone achieves an FID of 3.36, which improves to 3.29 when teacher alignment is added. On FFHQ, ImageNet, and SDXL, teacher alignment improves FID from 10.55, 1.76, and 24.58 to 6.14, 1.20, and 18.52, respectively. The full objective also outperforms teacher alignment alone, indicating that supervision from real data and pretrained knowledge from the teacher provide complementary benefits.
Thank you for your response. My concerns have been fully addressed.
W1 and Q2. Component ablations across datasets
Thank you for the suggestion. We added component ablations on ImageNet and SDXL to examine how the two supervision terms contribute across different tasks.
The different ablation trends can be explained by the difficulty of learning an effective bridge within a limited number of adaptation steps. The real-data term requires the bridge to learn a denoising mapping from noisy–clean training pairs, whereas teacher alignment directly transfers predictions from a pretrained model. The latter allows the bridge to reuse the teacher's denoising knowledge during the short adaptation stage. On low-resolution CIFAR-10, real-data supervision alone is sufficient to produce an effective bridge, so teacher alignment provides a smaller additional gain. On higher-resolution or more complex generation tasks, learning the target mapping from real-data supervision alone within the same limited adaptation budget is more difficult, making teacher alignment more beneficial. The new results support this explanation: on FFHQ, ImageNet, and SDXL, teacher alignment alone outperforms real-data denoising alone.
The full objective achieves the lowest FID on all four tasks, showing that real-data supervision and teacher guidance are complementary even though their relative contributions vary across tasks. We will add these results and analysis to Section 6.3 and revise the current general statement about component necessity to discuss these task-dependent effects.