- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies distributional guarantees for one-step Score Identity Distillation (SiD). The authors bound the total-variation distance between the distribution of student generator output and underlying data distribution. The bound depends on teacher error, the attained distillation objective, surrogate error, and terminal distribution mismatches. The paper also gives a score-surrogate approximation result and a Gaussian example where distillation outperforms sampling with the same biased teacher.
Critical strengths
-
The paper is very well written. The presentation is clear, well organized, and easy to follow. The mathematical development is careful, and I enjoyed reading it.
-
The paper addresses an important problem. Diffusion distillation is an important subject with many recent empirical developments. However, the theoretical aspect is less understood. I value the results the authors established.
-
The framework handles error sources under different measurements, and they may not naturally be combined together.
Critical weaknesses
-
The connection to practical training remains limited. Assumptions 3.3 and 3.4 require small distillation and surrogate errors for the same trained pair. The paper does not show how alternating training achieves both conditions. As the generator changes, the surrogate must track a moving target. This seems to be an essential difficulty in practice.
-
The technical novelty needs clearer explanation. The main error bound uses standard arguments that convert score error into distributional error. Theorem 3.7 also builds on existing score-approximation theory. The paper should compare these results directly with prior work and identify the new assumptions, rates, or proof techniques.
3_ The Gaussian example is useful, but it does not establish how informative the bounds are during practical distillation. A numerical study connecting the error terms to training behavior would strengthen the paper.
- The results in this paper do not lead to new implementation insights.
Questions for authors
-
Under what conditions can alternating training make the errors in Assumptions 3.3 and 3.4 small at the same time?
-
How can the excess denoising risk in Assumption 3.4 be estimated or controlled when the Bayes risk is unknown?
-
Compared to other distillation methods, SiD does not seem to provide state-of-the-art results; the main reason is the joint training of the one-step generator and its score function sarragte. Learning an additional score function is also computationally (and training) expensive. Can the framework proposed in this paper be applied to analyze different distillation methods? I still think the framework somehow does not capture the essential drawback of the design, as I mentioned.
3 comments
Response to Reviewer dcM2 (Part 2/3)
2. Technical novelty and comparison with prior work
Scope. Our goal is the same as in the convergence theory of diffusion samplers: a provable bound on the distance between the generated distribution and the data. To our knowledge, this is the first such bound for score distillation, and it needs several components that existing work does not provide.
- A common comparison process. The teacher error is controlled under the noised data distribution and the SiD loss under the noised student distribution. Comparing both with one teacher-driven reverse process uses each error where it is controlled, with no discretization error. This also partly explains the empirical benefit of SiD against the pretrained teacher model, as observed in the original paper.
- An exact identity for the fused loss, which shows how the surrogate error enters for every coefficient (Section 2 above).
- The student can be better than sampling from its teacher. The teacher's bias is averaged over the diffusion path (Proposition 3.9), which a triangle-inequality argument cannot give.
3. How informative the bounds are during practical distillation
We added controlled experiments with a trained nonlinear generator and a learned score surrogate on a one-dimensional Gaussian target, where the distributional error and the bound can be evaluated independently (new Section 3.5, more details in Appendices E and F).
- Non-vacuity. At =0.5, the bound of Theorem 3.6 is below 1 for all final models. Each error term and both terminal mismatches are evaluated separately (see Table 1 in Appendix E.1).
- Dependence on the errors. Increasing the teacher or surrogate bias increases the mean TV. For positive surrogate biases, mean 3 rises from 0.0246 to 0.2534 while TV rises from 0.0107 to 0.1857 (see Figure 2 in Appendix E.1). Along training, the error terms, the TV and the bound decrease together (table in Section 1 above).
- Original SiD coefficients. We also train at =1 and =1.2. For unbiased and moderately biased teachers, all models have non-vacuous bounds.
- Comparison with Heun and DDIM sampling. With the biased teacher st(x)=-cx/(1+t), c{0.5,1.5}, and T=1000, trained SiD students achieve lower mean TV than both Heun integration of the probability-flow ODE [2] and deterministic DDIM [3], at every tested training coefficient {0.5,1,1.2}. Both samplers use 1,024 steps, and all methods are evaluated at the common cutoff t0=0.1:
| Teacher c | SiD, =0.5 | SiD, =1 | SiD, =1.2 | Heun | DDIM |
|---|---|---|---|---|---|
| 0.5 | 0.3327 | 0.3911 | 0.4747 | 0.6720 | 0.6721 |
| 1.5 | 0.1999 | 0.1867 | 0.1833 | 0.6720 | 0.6739 |
Both samplers are checked against the exact ODE limit, and the same ordering holds at mixing time T=100.
Response to Reviewer dcM2 (Part 3/3)
4. Implementation insights
Guidance for practice (new Remark D.3). Theorem 3.6 shows how each trained component enters the end-to-end TV error, which gives the following guidance.
- Noise range. For a variance-exploding schedule, the terminal noise level should satisfy to reach accuracy . The early stopping time t0 sets the resolution of the guarantee.
- Balancing training accuracy. Our bound suggests prioritizing the dominant error contribution. Improving the generator and surrogate offers diminishing gains in the guarantee once their combined contribution falls below the teacher error. For near 1, the bound is more sensitive to surrogate error, suggesting that the surrogate should be trained more accurately.
5. Controlling the surrogate error when the Bayes risk is unknown
The Bayes risk never needs to be known or estimated.
- It cancels in the quantity itself. By the denoising identity of [6], restated as Lemma A.1, the excess denoising risk in Assumption 3.4 equals the L2 error between the surrogate score and the true student score. The Bayes risk is a constant that does not depend on the surrogate.
- It cancels in the analysis. The proof of Theorem C.1 works with differences of empirical losses, in which the Bayes risk drops out.
- The estimation error is a computational budget. The surrogate is trained on samples generated by the student, which are unlimited, so the sample size n in Theorem C.1 can be as large as computation allows.
6. State of the art, the cost of the extra score network, and other distillation methods
Empirical standing of SiD. We use SiD as a representative of score distillation and make no claim about rankings. For reference, SiD [4] reports one-step FIDs of 1.92 and 1.71 on CIFAR-10 (unconditional and conditional), below the 1.97 and 1.79 of its EDM teacher, and 1.52 on ImageNet 6464. Its adversarial extension SiDA [7] reports 1.50, 1.40 and 1.11 on the same benchmarks, and DMD2 [8], from the same two-network family, reports 1.28 on ImageNet 6464.
Other distillation methods. Theorem 3.6 has two parts. The first is independent of the method: any one-step generator, however trained, satisfies the bound with K(2,3) replaced by the student-teacher score error . Only Lemma 3.5, which converts the SiD loss into a bound on , is specific to SiD. For KL-based methods such as Diff-Instruct, DMD and variational score distillation, equals twice the decrease of the KL divergence between the noised student and teacher distributions when the teacher is the exact score of a distribution, so their objectives control the same quantity (see Lemma D.8 in Appendix D.4). These methods also train a surrogate score, so Theorem 3.7 and Theorem C.1 apply unchanged. Trajectory-based methods such as consistency models avoid the extra network but pay a self-consistency or discretization error instead, and require different tools.
Does the framework capture the cost of the two-network design? It captures the statistical cost. The surrogate error enters the bound on the same scale as the distillation loss, and Theorem 3.7 and Theorem C.1 quantify its cost in network size and in samples for each generator. It does not capture the stability of the alternating optimization, which is intractable to analyze for general large neural networks (see Section 1 above).
[1] Chen, et al. (2023). Sampling is as easy as learning the score: Theory for diffusion models with minimal data assumptions.
[2] Karras, et al. (2022). Elucidating the design space of diffusion-based generative models.
[3] Song, et al. (2021). Denoising diffusion implicit models.
[4] Zhou, et al. (2024). Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation.
[5] Fu, et al. (2024). Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory.
[6] Vincent (2011). A connection between score matching and denoising autoencoders.
[7] Zhou, et al. (2025). Adversarial score identity distillation: Rapidly surpassing the teacher in one step.
[8] Yin, et al. (2024). Improved distribution matching distillation for fast image synthesis.
Response to Reviewer dcM2 (Part 1/3)
Thank you for your encouraging assessment and your precise questions. Please find our detailed responses below. All changes are marked in blue in the revised PDF. Existing theorem and equation numbers are unchanged, and new results are placed in the new Section 3.5 and Appendices C-F.
Summary of our response.
1. Can both error conditions hold for the same trained pair?
The two conditions depend on different networks. Assumption 3.4 (small surrogate error 3) is a requirement on the surrogate: its capacity and its training samples. Assumption 3.3 (small SiD objective 2) is a requirement on the generator: whether it is expressive enough to match the teacher. They do not conflict.
Assumption 3.4 can be met for every generator, including a changing one. We added a finite-sample guarantee for the surrogate. When it is trained by empirical risk minimization on n samples from the generator, the surrogate error 3 is of order n-1/4 up to logarithmic factors, with high probability (see Theorem C.1 in Appendix C). The guarantee depends on the generator only through the bound on its outputs, so the same surrogate architecture and the same number of samples work for every generator encountered during training (see Corollary C.2 in Appendix C).
Assumption 3.3 is governed by the generator's capacity. The attainable SiD objective is controlled by how well the generator approximates a transport map from Gaussian noise to a reference distribution (see Proposition D.9 in Appendix D.4).
Experiment: both errors become small under alternating training. With one surrogate update per generator update, the two error terms, the measured TV and the bound all decrease together (baseline at =0.5, means over 5 replicas; see also Figure 2 in Appendix E.1 and Figure 5 in Appendix F):
The revised Assumption 3.3. The theory is now stated in SiD's own coefficient of the fused loss ℓ(2)−αℓ(1). Correspondingly, Assumption 3.3 is now stated with an absolute value, ∣Sα∣≤ϵ2, because the fused loss is negative for >1. This matches SiD as it is trained: with the default =1.2, the squared score-difference term has the coefficient 1-=-0.2 (Eq. (23) and Algorithm 1 of [4]).
Optimization. Analyzing the optimization dynamics for general large neural networks, let alone a bilevel one, is intractable. We therefore follow the standard separation in the theory of diffusion models, where sampling guarantees take the score error as given [1] and statistical guarantees analyze the empirical risk minimizer rather than the training dynamics [5]. Our theorem is a guarantee for the trained model: whatever the optimization returns, the bound holds with the attained values.