acceptodds
Under review as a conference paper at ICLR 2027

One Step, Provably: Distributional Guarantees for Score Distillation

Abstract

Diffusion models generate high-quality samples but require many sequential network evaluations. Score distillation enables one-step generation from a pretrained teacher diffusion model, yet theoretical guarantees for its distributional accuracy remain limited. We analyze one-step score distillation using an equivalent reverse-time diffusion whose marginals match the student's forward-noised distributions. Applying this framework to Score Identity Distillation (SiD), we establish an explicit population-level total-variation bound between the forward-noised student and data distributions at a positive noise level corresponding to early stopping in the reverse-time analysis. The bound accounts for the attained SiD objective, teacher-score and score-surrogate errors, and terminal distribution mismatches, with no sampling discretization term. For bounded generators, we further construct a single ReLU score-surrogate network with explicit size bounds for any prescribed mean-square approximation accuracy, uniformly over the analyzed noise range. Finally, a Gaussian example shows that SiD can mitigate the effect of teacher-score bias and achieve smaller distributional error than exact probability-flow sampling driven by the same teacher.

Primary area:
learning theory
Keywords:
Diffusion Models, Distillation, Score Identity distillation, Machine Learning Theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dcM2OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies distributional guarantees for one-step Score Identity Distillation (SiD). The authors bound the total-variation distance between the distribution of student generator output and underlying data distribution. The bound depends on teacher error, the attained distillation objective, surrogate error, and terminal distribution mismatches. The paper also gives a score-surrogate approximation result and a Gaussian example where distillation outperforms sampling with the same biased teacher.

Critical strengths
  1. The paper is very well written. The presentation is clear, well organized, and easy to follow. The mathematical development is careful, and I enjoyed reading it.

  2. The paper addresses an important problem. Diffusion distillation is an important subject with many recent empirical developments. However, the theoretical aspect is less understood. I value the results the authors established.

  3. The framework handles error sources under different measurements, and they may not naturally be combined together.

Critical weaknesses
  1. The connection to practical training remains limited. Assumptions 3.3 and 3.4 require small distillation and surrogate errors for the same trained pair. The paper does not show how alternating training achieves both conditions. As the generator changes, the surrogate must track a moving target. This seems to be an essential difficulty in practice.

  2. The technical novelty needs clearer explanation. The main error bound uses standard arguments that convert score error into distributional error. Theorem 3.7 also builds on existing score-approximation theory. The paper should compare these results directly with prior work and identify the new assumptions, rates, or proof techniques.

3_ The Gaussian example is useful, but it does not establish how informative the bounds are during practical distillation. A numerical study connecting the error terms to training behavior would strengthen the paper.

  1. The results in this paper do not lead to new implementation insights.
Questions for authors
  1. Under what conditions can alternating training make the errors in Assumptions 3.3 and 3.4 small at the same time?

  2. How can the excess denoising risk in Assumption 3.4 be estimated or controlled when the Bayes risk is unknown?

  3. Compared to other distillation methods, SiD does not seem to provide state-of-the-art results; the main reason is the joint training of the one-step generator and its score function sarragte. Learning an additional score function is also computationally (and training) expensive. Can the framework proposed in this paper be applied to analyze different distillation methods? I still think the framework somehow does not capture the essential drawback of the design, as I mentioned.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dcM2 (Part 1/3)

Thank you for your encouraging assessment and your precise questions. Please find our detailed responses below. All changes are marked in blue in the revised PDF. Existing theorem and equation numbers are unchanged, and new results are placed in the new Section 3.5 and Appendices C-F.

Summary of our response.

  1. Both error conditions for the same trained pair. The two conditions depend on different networks. We added a finite-sample theorem that also covers a changing generator, a capacity bound, and experiments in which both errors become small under alternating training.
  2. Technical novelty. We give a direct comparison with prior work.
  3. How informative the bounds are in practice. We added experiments with trained networks. The bound is non-vacuous and follows the error decomposition.
  4. Implementation insights. We give guidance on the noise range and on how accurately each component should be trained.
  5. Unknown Bayes risk. It is never needed, because it cancels.
  6. State of the art, cost of the surrogate, other methods. We explain how the framework extends to other methods and what it captures.

1. Can both error conditions hold for the same trained pair?

The two conditions depend on different networks. Assumption 3.4 (small surrogate error 3) is a requirement on the surrogate: its capacity and its training samples. Assumption 3.3 (small SiD objective 2) is a requirement on the generator: whether it is expressive enough to match the teacher. They do not conflict.

Assumption 3.4 can be met for every generator, including a changing one. We added a finite-sample guarantee for the surrogate. When it is trained by empirical risk minimization on n samples from the generator, the surrogate error 3 is of order n-1/4 up to logarithmic factors, with high probability (see Theorem C.1 in Appendix C). The guarantee depends on the generator only through the bound on its outputs, so the same surrogate architecture and the same number of samples work for every generator encountered during training (see Corollary C.2 in Appendix C).

Assumption 3.3 is governed by the generator's capacity. The attainable SiD objective is controlled by how well the generator approximates a transport map from Gaussian noise to a reference distribution (see Proposition D.9 in Appendix D.4).

Experiment: both errors become small under alternating training. With one surrogate update per generator update, the two error terms, the measured TV and the bound all decrease together (baseline at =0.5, means over 5 replicas; see also Figure 2 in Appendix E.1 and Figure 5 in Appendix F):

Generator updates23Measured TVBound
00.60010.08310.33340.5722
1000.12860.05790.16590.2850
3000.00080.05350.05860.1169
10000.00000.03340.00950.0912
50000.00000.02460.01070.0797

The revised Assumption 3.3. The theory is now stated in SiD's own coefficient of the fused loss ℓ(2)−αℓ(1)\ell^{(2)}-\alpha \ell^{(1)}. Correspondingly, Assumption 3.3 is now stated with an absolute value, ∣Sα∣≤ϵ2\lvert\mathcal S_\alpha\rvert\le\epsilon_2, because the fused loss is negative for >1. This matches SiD as it is trained: with the default =1.2, the squared score-difference term has the coefficient 1-=-0.2 (Eq. (23) and Algorithm 1 of [4]).

Optimization. Analyzing the optimization dynamics for general large neural networks, let alone a bilevel one, is intractable. We therefore follow the standard separation in the theory of diffusion models, where sampling guarantees take the score error as given [1] and statistical guarantees analyze the empirical risk minimizer rather than the training dynamics [5]. Our theorem is a guarantee for the trained model: whatever the optimization returns, the bound holds with the attained values.

AuthorsOpenReview ↗

Response to Reviewer dcM2 (Part 2/3)

2. Technical novelty and comparison with prior work

Scope. Our goal is the same as in the convergence theory of diffusion samplers: a provable bound on the distance between the generated distribution and the data. To our knowledge, this is the first such bound for score distillation, and it needs several components that existing work does not provide.

  1. A common comparison process. The teacher error is controlled under the noised data distribution and the SiD loss under the noised student distribution. Comparing both with one teacher-driven reverse process uses each error where it is controlled, with no discretization error. This also partly explains the empirical benefit of SiD against the pretrained teacher model, as observed in the original paper.
  2. An exact identity for the fused loss, which shows how the surrogate error enters for every coefficient (Section 2 above).
  3. The student can be better than sampling from its teacher. The teacher's bias is averaged over the diffusion path (Proposition 3.9), which a triangle-inequality argument cannot give.

3. How informative the bounds are during practical distillation

We added controlled experiments with a trained nonlinear generator and a learned score surrogate on a one-dimensional Gaussian target, where the distributional error and the bound can be evaluated independently (new Section 3.5, more details in Appendices E and F).

  • Non-vacuity. At =0.5, the bound of Theorem 3.6 is below 1 for all final models. Each error term and both terminal mismatches are evaluated separately (see Table 1 in Appendix E.1).
  • Dependence on the errors. Increasing the teacher or surrogate bias increases the mean TV. For positive surrogate biases, mean 3 rises from 0.0246 to 0.2534 while TV rises from 0.0107 to 0.1857 (see Figure 2 in Appendix E.1). Along training, the error terms, the TV and the bound decrease together (table in Section 1 above).
  • Original SiD coefficients. We also train at =1 and =1.2. For unbiased and moderately biased teachers, all models have non-vacuous bounds.
  • Comparison with Heun and DDIM sampling. With the biased teacher st(x)=-cx/(1+t), c{0.5,1.5}, and T=1000, trained SiD students achieve lower mean TV than both Heun integration of the probability-flow ODE [2] and deterministic DDIM [3], at every tested training coefficient {0.5,1,1.2}. Both samplers use 1,024 steps, and all methods are evaluated at the common cutoff t0=0.1:
Teacher cSiD, =0.5SiD, =1SiD, =1.2HeunDDIM
0.50.33270.39110.47470.67200.6721
1.50.19990.18670.18330.67200.6739

Both samplers are checked against the exact ODE limit, and the same ordering holds at mixing time T=100.

AuthorsOpenReview ↗

Response to Reviewer dcM2 (Part 3/3)

4. Implementation insights

Guidance for practice (new Remark D.3). Theorem 3.6 shows how each trained component enters the end-to-end TV error, which gives the following guidance.

  • Noise range. For a variance-exploding schedule, the terminal noise level should satisfy σT2≳m2/ϵ2\sigma_T^2\gtrsim m_2/\epsilon^2 to reach accuracy . The early stopping time t0 sets the resolution of the guarantee.
  • Balancing training accuracy. Our bound suggests prioritizing the dominant error contribution. Improving the generator and surrogate offers diminishing gains in the guarantee once their combined contribution falls below the teacher error. For near 1, the bound is more sensitive to surrogate error, suggesting that the surrogate should be trained more accurately.

5. Controlling the surrogate error when the Bayes risk is unknown

The Bayes risk never needs to be known or estimated.

  • It cancels in the quantity itself. By the denoising identity of [6], restated as Lemma A.1, the excess denoising risk in Assumption 3.4 equals the L2 error between the surrogate score and the true student score. The Bayes risk is a constant that does not depend on the surrogate.
  • It cancels in the analysis. The proof of Theorem C.1 works with differences of empirical losses, in which the Bayes risk drops out.
  • The estimation error is a computational budget. The surrogate is trained on samples generated by the student, which are unlimited, so the sample size n in Theorem C.1 can be as large as computation allows.

6. State of the art, the cost of the extra score network, and other distillation methods

Empirical standing of SiD. We use SiD as a representative of score distillation and make no claim about rankings. For reference, SiD [4] reports one-step FIDs of 1.92 and 1.71 on CIFAR-10 (unconditional and conditional), below the 1.97 and 1.79 of its EDM teacher, and 1.52 on ImageNet 6464. Its adversarial extension SiDA [7] reports 1.50, 1.40 and 1.11 on the same benchmarks, and DMD2 [8], from the same two-network family, reports 1.28 on ImageNet 6464.

Other distillation methods. Theorem 3.6 has two parts. The first is independent of the method: any one-step generator, however trained, satisfies the bound with K(2,3) replaced by the student-teacher score error Mθ,ϕ\mathcal M_{\theta,\phi}. Only Lemma 3.5, which converts the SiD loss into a bound on Mθ,ϕ\mathcal M_{\theta,\phi}, is specific to SiD. For KL-based methods such as Diff-Instruct, DMD and variational score distillation, Mθ,ϕ\mathcal M_{\theta,\phi} equals twice the decrease of the KL divergence between the noised student and teacher distributions when the teacher is the exact score of a distribution, so their objectives control the same quantity (see Lemma D.8 in Appendix D.4). These methods also train a surrogate score, so Theorem 3.7 and Theorem C.1 apply unchanged. Trajectory-based methods such as consistency models avoid the extra network but pay a self-consistency or discretization error instead, and require different tools.

Does the framework capture the cost of the two-network design? It captures the statistical cost. The surrogate error enters the bound on the same scale as the distillation loss, and Theorem 3.7 and Theorem C.1 quantify its cost in network size and in samples for each generator. It does not capture the stability of the alternating optimization, which is intractable to analyze for general large neural networks (see Section 1 above).

[1] Chen, et al. (2023). Sampling is as easy as learning the score: Theory for diffusion models with minimal data assumptions.

[2] Karras, et al. (2022). Elucidating the design space of diffusion-based generative models.

[3] Song, et al. (2021). Denoising diffusion implicit models.

[4] Zhou, et al. (2024). Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation.

[5] Fu, et al. (2024). Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory.

[6] Vincent (2011). A connection between score matching and denoising autoencoders.

[7] Zhou, et al. (2025). Adversarial score identity distillation: Rapidly surpassing the teacher in one step.

[8] Yin, et al. (2024). Improved distribution matching distillation for fast image synthesis.

Reviewer_mdyiOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides a theoretical analysis of one-step score distillation, focusing on Score Identity Distillation (SiD). The authors derive a population-level total-variation bound between the forward-noised student and data distributions, accounting for teacher-score error, the attained distillation objective, surrogate approximation error, and terminal distribution mismatch. They also establish an approximation result for the student score surrogate using ReLU networks and present a Gaussian example in which score distillation can achieve smaller distributional error than sampling directly from a biased teacher.

Critical strengths

The paper addresses an interesting theoretical gap in one-step diffusion distillation. While previous studies have established convergence guarantees for iterative diffusion sampling and consistency models, the distributional accuracy of score-distilled one-step generators is less well understood.

Critical weaknesses

The paper is entirely theoretical, with the comparison against teacher-driven sampling illustrated through a one-dimensional Gaussian example. While this example is informative, a numerical study involving higher-dimensional distributions or a trained score-distillation model would help demonstrate the relevance of the bounds beyond the idealized setting.

Questions for authors

Can the authors provide more intuition about how the total-variation guarantee at a positive noise level translates into the quality of clean samples, particularly as the early-stopping time approaches zero?

1 comment

Thank you for your careful review and for recognizing the theoretical gap addressed by our work. Please find our responses below. All changes are marked in blue in the revised PDF.

Summary of our response.

  1. Numerical validation. We added experiments with trained networks. The bound is non-vacuous and follows the error decomposition.
  2. From the noised guarantee to clean samples. We explain how the guarantee controls the clean output and what happens as the early stopping time t0 approaches zero.

1. Numerical validation with trained models

We agree that numerical validation would strengthen the analytical example. We added controlled experiments with a trained nonlinear generator and a learned score surrogate on a one-dimensional Gaussian target, where the distributional error and the bound can be evaluated independently (new Section 3.5, more details in Appendices E and F).

  • Non-vacuity. At =0.5, the bound of Theorem 3.6 is below 1 for all final models. The baseline has mean TV 0.0107 against a bound of 0.0797 (see Table 1 in Appendix E.1).
  • Dependence on the errors. Increasing the teacher or surrogate bias increases the mean TV (see Figure 2 in Appendix E.1).
  • Original SiD coefficients. We also train at =1 and =1.2. For unbiased and moderately biased teachers, all models have non-vacuous bounds.
  • Comparison with Heun and DDIM sampling. With the biased teacher st(x)=-cx/(1+t), c{0.5,1.5}, and T=1000, trained SiD students achieve lower mean TV than both Heun integration of the probability-flow ODE [2] and deterministic DDIM [3], at every tested training coefficient {0.5,1,1.2}. Both samplers use 1,024 steps, and all methods are evaluated at the common cutoff t0=0.1:
Teacher cSiD, =0.5SiD, =1SiD, =1.2HeunDDIM
0.50.33270.39110.47470.67200.6721
1.50.19990.18670.18330.67200.6739

Both samplers are checked against the exact ODE limit, and the same ordering holds at mixing time T=100.

2. From the noised guarantee to clean samples

The student outputs the clean sample g(Z) directly; t0 is the early stopping time used in the analysis. Equation (3.5) translates the noised-TV guarantee into a statement about this clean output: there is a comparison distribution R,t0 such that

TV(Pθ,Rθ,t0)≤min⁡{1,Δt0,T},W2(Rθ,t0,Pdata)≤2dσt0αt0,\mathrm{TV}\big(P^\theta,R_{\theta,t_0}\big)\le\min\lbrace 1,\Delta_{t_0,T}\rbrace,\qquad W_2\big(R_{\theta,t_0},P^{\mathrm{data}}\big)\le\frac{2\sqrt{d} \sigma_{t_0}}{\alpha_{t_0}},

where t0,T is the bound in Theorem 3.6.

Intuition. The TV term controls how much probability mass must be changed in the clean student distribution to obtain R,t0. The W2 term controls the root-mean-square displacement needed to transport that distribution to the data. The guarantee thus combines a small mass discrepancy with a small spatial displacement. When the data and the student outputs are bounded, we also provide a direct W1 bound between the clean distributions (see Corollary D.5 in Appendix D.3).

As t0 approaches zero. For Brownian noising, the Wasserstein radius is 2dt0 and vanishes as t00. The TV budget must also remain small: reducing t0 extends the score-error integrals toward lower noise levels, where accurate score approximation can be harder. Decreasing t0 therefore sharpens the spatial resolution of the guarantee, but does not by itself improve a fixed generator. In our experiments with fixed trained models, increasing t0 from 0.1 to 5 reduces the noised TV from 0.0107 to 0.0029, while the clean W2 distance of the generator stays at 0.0269 (see Appendix F.4).

Why the Wasserstein component is needed. Under our general assumptions it cannot be removed. For example, 0 and v have W2 distance v but TV distance 1 for every v0, however small the displacement (see Remark D.6). Related diffusion-sampling guarantees likewise separate a TV error from a Wasserstein perturbation of the clean target ([1], Section 3.2, Corollary 3).

[1] Chen, et al. (2023). Sampling is as easy as learning the score: Theory for diffusion models with minimal data assumptions.

[2] Karras, et al. (2022). Elucidating the design space of diffusion-based generative models.

[3] Song, et al. (2021). Denoising diffusion implicit models.

Reviewer_CqQuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies population-level distributional guarantees for one-step generators trained with a SiD-type objective. By comparing the exact reverse diffusions associated with the data and student distributions through a common teacher-driven process, it derives a total-variation bound at a positive noise level. The bound accounts for teacher-score error, the attained distillation objective, surrogate-score error, and terminal distribution mismatches. The paper also provides a constructive ReLU approximation result for the auxiliary score network and an analytical Gaussian example comparing an oracle student with teacher-driven sampling.

Critical strengths
  • The paper reflects substantial effort in developing and documenting the mathematical analysis. On my reading, the central argument of Theorem 3.6 appears internally coherent under the stated assumptions. The comparison through a common teacher-driven process usefully preserves the different measures under which teacher-side and student-side errors are controlled.

  • The explicit approximation result for the auxiliary score network adds substance beyond the main error decomposition. The analytically tractable Gaussian example is also a useful illustration of how score matching and teacher-driven sampling can respond differently to teacher bias, although its implications remain restricted to the oracle setting considered.

Critical weaknesses
  1. The paper provides an analytical Gaussian example, but no numerical experiments or systematic empirical evaluation. Consequently, it remains unclear whether the overall bound is informative or non-vacuous, whether its error decomposition tracks actual distributional discrepancies, and how the surrogate accuracy, noise cutoff, and loss coefficients affect the resulting generator. The Gaussian example assumes an exact surrogate and an oracle minimizer, so it does not address these questions about practical training. Controlled experiments on tractable distributions could directly compare the bound with the actual distributional error, while experiments with a trained generator could examine the relevance of the predicted dependencies. Given the conditional nature of the guarantees, I regard the complete absence of such validation as a major weakness.

Equation (2.6) defines

ℓ(λ)=(1−λ)ℓ(1)+λℓ(2),λ≥0.\ell^{(\lambda)} =(1-\lambda)\ell^{(1)}+\lambda\ell^{(2)}, \qquad \lambda\geq 0.

In the original SiD formulation [2], the fused objective is L(2)−αL(1)L^{(2)}-\alpha L^{(1)}. For finite λ>0\lambda>0, a positive rescaling identifies these objectives only when

α=1−1λ<1.\alpha=1-\frac{1}{\lambda}<1.

However, the original SiD paper uses α=1\alpha=1 and α=1.2\alpha=1.2 in its main experiments. These settings are therefore not covered by the stated reparameterization. This distinction matters for a guarantee based on the attained objective value: under the paper’s exact-surrogate population idealization with time-only weighting, both component losses equal the score discrepancy MM, whereas the original fused objective equals (1−α)M(1-\alpha)M. Its value therefore cannot directly provide the proposed certificate when α≥1\alpha\geq1. This does not invalidate the theorem for the objective actually defined, but it materially restricts its applicability to the practical method it is intended to analyze.

  1. The main theorem assumes bounds on the attained population objective and surrogate error, without establishing that the alternating training procedure can achieve them. There is no optimization convergence guarantee, finite-sample analysis, or characterization of how the attainable student objective depends on the generator’s capacity. Theorem 3.7 concerns the existence of an auxiliary score approximation for a fixed bounded generator; it does not show that training finds this approximation or that the surrogate can track a changing student. The authors acknowledge several of these limitations, but they substantially restrict what the results establish about learned one-step generators.

  2. Prior work already developed the distribution-matching rationale and relevant score-based objective formulations, including VSD [1] and SiD [2]. The present paper adds a conditional quantitative error decomposition, but the main proof combines existing score identities, a completed-square argument, reverse-process KL comparisons, Pinsker’s inequality, and the triangle inequality. The common teacher-process construction is useful, yet the paper does not sufficiently establish how substantial the resulting advance is over an adaptation of existing diffusion stability tools. Moreover, the analysis is not translated into concrete, validated guidance for selecting loss coefficients, allocating surrogate updates, or choosing the noise range. In combination with the absence of experiments, this leaves the significance of the contribution insufficiently demonstrated.

  3. Theorem 3.6 controls the forward-noised distributions at t0>0t_0>0. The clean-output extension in Eq. (3.5) is a mixed Wasserstein–TV statement involving an intermediate distribution, rather than a direct TV bound between the clean student and data distributions. In addition, Assumption A.2 imposes substantive reverse-process and change-of-measure conditions, which are not implied merely by finite score errors or finite second moments. These qualifications are stated in the paper, but their consequences for practical applicability deserve more attention.

[1] Wang et al. ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. NeurIPS, 2023.
https://proceedings.neurips.cc/paper_files/paper/2023/hash/1a87980b9853e84dfb295855b425c262-Abstract-Conference.html

[2] Zhou et al. Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation. ICML, 2024.
https://proceedings.mlr.press/v235/zhou24x.html

Questions for authors
  1. Can the authors provide controlled numerical validation showing when the overall bound is non-vacuous and whether its predicted dependence on teacher error, surrogate error, and the noise cutoff agrees with the observed distributional error?

  2. How does the reparameterization in Eq. (2.6) cover the original SiD settings α=1\alpha=1 and α=1.2\alpha=1.2? If it does not, which SiD variants are covered, and how should the claimed applicability be qualified?

  3. What is the principal technical advance beyond combining existing score identities with standard reverse-diffusion stability arguments? Does the analysis yield concrete guidance for SiD training that can be evaluated?

  4. Which of the assumed population error quantities can be estimated or controlled during actual training, and under what practical conditions can the reverse-process assumptions be verified?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CqQu (Part 1/3)

Thank you for your careful and detailed review. Please find our detailed responses below. All changes are marked in blue in the revised PDF. Existing theorem and equation numbers are unchanged, and new results are placed in the new Section 3.5 and Appendices C-F.

Summary of our response.

  1. Numerical validation. We added experiments with trained networks. The bound is non-vacuous and follows the error decomposition.
  2. SiD coefficients =1 and =1.2. The theory is now stated directly in SiD's own coefficient .
  3. Finite samples, a changing student, generator capacity, optimization. We added a finite-sample theorem, a result for a changing student, and a capacity bound. As in most theoretical papers on diffusion models [1, 5], optimization is out of our scope.
  4. Novelty and practical guidance. As echoed by reviewer dcM2, the theoretical aspect of diffusion distillation is less understood. To our knowledge, this is the first distributional guarantee for score distillation. We clarify its new technical components and give guidance on the noise range and on how accurately each component should be trained.
  5. Early stopping and Assumption A.2. Early stopping at a positive noise level is necessary in literature [1]. We added a single-metric bound for clean outputs, and gave a verifiable condition for Assumption A.2.
  6. Which error terms can be estimated. We state how each term is obtained in practice.

1. Numerical validation of the bound

We added controlled experiments with a trained nonlinear generator and a learned score surrogate on a one-dimensional Gaussian target, where the distributional error and the bound can be evaluated independently (new Section 3.5, more details in Appendices E and F).

  • Non-vacuity. At =0.5, the bound of Theorem 3.6 is below 1 for all final models. Each error term and both terminal mismatches are evaluated separately (see Table 1 in Appendix E.1).
  • Dependence on the errors. Increasing the teacher or surrogate bias increases the mean TV. For positive surrogate biases, mean 3 rises from 0.0246 to 0.2534 while TV rises from 0.0107 to 0.1857 (see Figure 2 in Appendix E.1).
  • Original SiD coefficients. We also train at =1 and =1.2. For unbiased and moderately biased teachers, all models have non-vacuous bounds.
  • Comparison with Heun and DDIM sampling. With the biased teacher st(x)=-cx/(1+t), c{0.5,1.5}, and T=1000, trained SiD students achieve lower mean TV than both Heun integration of the probability-flow ODE [2] and deterministic DDIM [3], at every tested training coefficient {0.5,1,1.2}. Both samplers use 1,024 steps, and all methods are evaluated at the common cutoff t0=0.1:
Teacher cSiD, =0.5SiD, =1SiD, =1.2HeunDDIM
0.50.33270.39110.47470.67200.6721
1.50.19990.18670.18330.67200.6739

Both samplers are checked against the exact ODE limit, and the same ordering holds at mixing time T=100.

AuthorsOpenReview ↗

Response to Reviewer CqQu (Part 2/3)

2. Coverage of the original SiD coefficients =1 and =1.2

We agree with the reviewer, and we have removed the -reparameterization from the paper. The revision analyzes the SiD loss in its original form, ℓ(α)=ℓ(2)−αℓ(1)\ell^{(\alpha)}=\ell^{(2)}-\alpha \ell^{(1)} (now Eq. (2.6)). Assumption 3.3, Lemma 3.5 and Theorem 3.6 are restated in terms of , with the same proof.

The case 1, including =1.2. As the reviewer points out, with an exact surrogate the fused loss equals (1−α)Mθ,ϕ(1-\alpha) \mathcal M_{\theta,\phi}. It is positive for <1 and negative for >1, but in both cases its magnitude measures Mθ,ϕ\mathcal M_{\theta,\phi}. The revised Lemma 3.5 gives, for every 1,

Mθ,ϕ≤Kα(ϵ2,ϵ3):=(∣1−2α∣ϵ3+ϵ3+4∣1−α∣ϵ22∣1−α∣)2,\mathcal M_{\theta,\phi}\le K_\alpha(\epsilon_2,\epsilon_3):=\left(\frac{\lvert 1-2\alpha\rvert\sqrt{\epsilon_3}+\sqrt{\epsilon_3+4\lvert 1-\alpha\rvert \epsilon_2}}{2\lvert 1-\alpha\rvert}\right)^2,

where 3 is the surrogate error, and Theorem 3.6 holds with this K.

Assumption 3.3. Correspondingly, Assumption 3.3 is now stated with an absolute value, ∣Sα∣≤ϵ2\lvert\mathcal S_\alpha\rvert\le\epsilon_2, because the fused loss is negative for >1. This matches SiD as it is trained: with the default =1.2, the squared score-difference term has the coefficient 1-=-0.2 (Eq. (23) and Algorithm 1 of [4]).

The case =1. At =1 the fused value alone is uninformative. Our guarantee, however, evaluates the trained model, not the training procedure. A model trained with =1 can be evaluated at any '1, since SiD computes ℓ(1)\ell^{(1)} and ℓ(2)\ell^{(2)} separately at every step (see Proposition D.1 in Appendix D.1).

3. Optimization, finite-sample analysis, and the generator's capacity

1) Finite-sample analysis (new Theorem C.1)

We added a finite-sample guarantee for the surrogate. When it is trained by empirical risk minimization on n samples from the student, the surrogate error 3 is of order n-1/4 up to logarithmic factors, with high probability. Please see Theorem C.1 in Appendix C.

2) Tracking a changing student (new Corollary C.2)

The guarantee depends on the student only through the bound on its outputs, so the same surrogate architecture and the same number of samples work for every student encountered during training. Please see Corollary C.2 in Appendix C.

3) Dependence on the generator's capacity (new Proposition D.9)

The attainable SiD objective is controlled by how well the generator approximates a transport map from Gaussian noise to a reference distribution. Please see Proposition D.9 in Appendix D.4.

4) Optimization

Analyzing the optimization dynamics for general large neural networks, let alone a bilevel one, is intractable. We therefore follow the standard separation in the theory of diffusion models, where sampling guarantees take the score error as given [1] and statistical guarantees analyze the empirical risk minimizer rather than the training dynamics [5]. Our theorem is a guarantee for the trained model: whatever the optimization returns, the bound holds with the attained values.

AuthorsOpenReview ↗

Response to Reviewer CqQu (Part 3/3)

4. Novelty and practical guidance

Scope. Our goal is the same as in the convergence theory of diffusion samplers: a provable bound on the distance between the generated distribution and the data. To our knowledge, this is the first such bound for score distillation, and it needs several components that existing work does not provide.

  1. A common comparison process. The teacher error is controlled under the noised data distribution and the SiD loss under the noised student distribution. Comparing both with one teacher-driven reverse process uses each error where it is controlled, with no discretization error. This also partly explains the empirical benefit of SiD against the pretrained teacher model, as observed in the original paper.
  2. An exact identity for the fused loss, which shows how the surrogate error enters for every coefficient (Section 2 above).
  3. The student can be better than sampling from its teacher. The teacher's bias is averaged over the diffusion path (Proposition 3.9), which a triangle-inequality argument cannot give.

Guidance for practice (new Remark D.3). Theorem 3.6 shows how each trained component enters the end-to-end TV error, which gives the following guidance.

  • Noise range. For a variance-exploding schedule, the terminal noise level should satisfy σT2≳m2/ϵ2\sigma_T^2\gtrsim m_2/\epsilon^2 to reach accuracy . The early stopping time t0 sets the resolution of the guarantee.
  • Balancing training accuracy. Our bound suggests prioritizing the dominant error contribution. Improving the generator and surrogate offers diminishing gains in the guarantee once their combined contribution falls below the teacher error. For near 1, the bound is more sensitive to surrogate error, suggesting that the surrogate should be trained more accurately.

5. Early stopping, the clean-output statement, and the reverse-process assumption

Why early stopping is needed. A bound in total variation or KL between the clean distributions is impossible without additional assumptions. Consider Pdata=0 and a student P=v. No matter how small v is, the clean distributions have total variation 1 and infinite KL divergence, although the student is accurate. After adding noise at level t0, KL(Pt0θ∥Pt0data)=αt02∥v∥22σt02,\mathrm{KL}\big(P_{t_0}^{\theta} \Vert P_{t_0}^{\mathrm{data}}\big)=\frac{\alpha_{t_0}^2\Vert v\Vert^2}{2\sigma_{t_0}^2}, which is small for an accurate student but diverges as t00. Our mixed Wasserstein-TV statement follows the standard early-stopping approach in diffusion sampling (e.g. [1], Section 3.2, Corollary 3). We added this example as Remark D.6.

A direct bound on the clean distributions. When the data and the student outputs are bounded, as for images, we provide a direct W1 bound between the clean distributions. Please see Corollary D.5 in Appendix D.3.

Assumption A.2. This is the standard condition under which Girsanov's theorem converts a score error into a KL bound, used throughout the convergence theory of diffusion samplers (e.g. [1]). To make it verifiable, we added Lemma D.4: if the data, the student outputs, and the output of the teacher's denoiser are bounded, then Assumption A.2 holds automatically. This covers image data with any clipped denoiser.

6. Which error terms can be estimated or controlled during training

  • The teacher error 1 is fixed before distillation. It is the excess denoising loss of the teacher on data and is not affected by SiD.
  • The SiD objective 2 can be estimated directly. It is an average, over samples generated by the student, of quantities computed from the teacher and the surrogate networks. This is the loss that SiD already evaluates.
  • The surrogate error 3 can also be estimated. It is the denoising loss of the student surrogate on the one-step generator, which can be estimated through finite samples from the one-step generator.
  • The terminal mismatch is explicit and is given by Eq. (3.4).

[1] Chen, et al. (2023). Sampling is as easy as learning the score: Theory for diffusion models with minimal data assumptions.

[2] Karras, et al. (2022). Elucidating the design space of diffusion-based generative models.

[3] Song, et al. (2021). Denoising diffusion implicit models.

[4] Zhou, et al. (2024). Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation.

[5] Fu, et al. (2024). Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.