acceptodds
Under review as a conference paper at ICLR 2027

Stabilizing Multi-Step Diffusion Distillation with Student Perturbation Consistency

Abstract

Multi-step diffusion distillation trains student transitions with local objectives but deploys them as a composition. Reducing per-step prediction error is therefore insufficient: residual errors can be amplified by the local gain of subsequent maps. This tension is clear in multi-step Distribution Matching Distillation (DMD). Exact differentiation through the student rollout contains expensive and potentially ill-conditioned products of transition Jacobians, so practical methods detach intermediate states using stop-gradient or SGTS. Truncation makes optimization feasible but leaves a hidden Jacobian gap: local sensitivity is absent from truncated cross-step credit assignment yet remains active at inference and governs residual-error propagation. We propose Student Perturbation Consistency (SPC), a finite-difference regularizer that supplies this missing sensitivity supervision by matching endpoint predictions for clean and perturbed inputs at each supervised state. First-order analysis shows that SPC penalizes average local sensitivity through the squared Frobenius norm of the endpoint-map Jacobian, without explicitly constructing the Jacobian or backpropagating through the rollout. SPC neither reconstructs discarded credit-assignment terms nor imposes hard contraction; it conditions the learned composition while retaining stop-gradient training. On Qwen-Image, SPC suppresses Jacobian growth and reduces endpoint amplification of controlled perturbations. Across Qwen-Image, Z-Image, and Qwen-Image-Edit, SPC consistently improves the same DMD base objective, benefits PCM as a plug-in regularizer, and matches or exceeds released accelerated models on several benchmarks.

Primary area:
generative models
Keywords:
Diffusion Distillation, Distribution Matching, Few-Step Generation, Jacobian Regularization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DTEGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on addressing the issue of accumulated multi-step inference error caused by the Jacobian matrix in distribution matching distillation (DMD) and related methods. The primary contribution lies in proposing the SPC loss as a Jacobian regularizer to explicitly minimize the Frobenius norm of the Jacobian matrix. The authors conduct experiments on three modern large-scale image generation models(Qwen-Image,etc.), demonstrating the strong practical utility of the proposed method and its advantages over the original DMD method.

Critical strengths
  • The paper provides an elegant way to address the issue of a large Frobenius norm of the Jacobian matrix, thereby mitigating the accumulation of multi-step inference error in the student model. The theoretical proof presented in Sec. 3.3 is clear and easy to follow.
  • The experiments are conducted on modern flow matching models, demonstrating the scalability and engineering application potential of the method. In Sec. 4, the authors compare against multiple baseline methods on Qwen-Image, and validate the effectiveness of the method across Qwen-Image, Z-Image, and Qwen-Image-Edit.
Critical weaknesses
  • Why does SPC lead to increased diversity? Theoretically, the SPC loss decreases the upper bound on the variance of the student's output and encourages the student to be insensitive to variations in noise—that is, producing the same output under different noise inputs. Shouldn't this lead to reduced diversity? However, an increase in diversity compared to DMD is observed across Tables 1, 2, and 3.
  • Does the instability of the DMD method truly originate from the Jacobian matrix? Could it potentially arise from other factors, such as an underfitted fake model or an overly large learning rate? If the training ratio between the fake model and student model is further increased, would the DMD method also avoid the sudden loss spikes shown in the left panel of Figure 3?
Questions for authors

Please refer to "Critical Weaknesses"

3 comments
AuthorsOpenReview ↗

Response to Reviewer DTEG

Thank you for your thoughtful comments and constructive suggestions.

W1: Why does diversity increase? Our claim is that SPC preserves diversity in the evaluated settings. SPC compares predictions from an input and its slightly perturbed version at the same timestep. A reduction in an upper bound on output variance does not imply that the actual variance, or a feature-based diversity metric, must decrease.

Table 7 provides a useful comparison: on PCM, SPC improves HPS-v2.1 from 28.01 to 30.96 while diversity and coverage statistics remain largely unchanged. Recall changes from 0.495 to 0.492, Coverage from 0.831 to 0.836, and Vendi from 275.45 to 275.21. The quality improvement therefore coexists with essentially preserved diversity.

For DMD, the increases may partly reflect reduced artifacts and structural distortions (Fig. 5), which can affect feature-based diversity and coverage measurements. Across both objectives, the results show no systematic diversity loss from SPC.

W2: Could the instability instead arise from the fake model or the learning rate? We tested these alternatives directly with two-step full-path DMD over 2,000 student updates. We increased the fake: student update ratio to 10 and 20, and separately reduced the student learning rate to $ 10^{-5} $.

Full-path configurationObserved training behavior
Default DMDLoss spike at approximately update 750
Fake:student ratio 10Spike persists similar to default
Fake:student ratio 20Spike persists similar to default
Student learning rate $ 10^{-5} $Loss spike at approximately update 1100
Default DMD + SPCStable throughout 2,000 updates

Increasing the fake-model update ratio did not remove the spike, and lowering the learning rate delayed its onset. Adding SPC maintained stable training throughout the same observation window. These controls show that SPC provides a stabilization benefit beyond the tested update-ratio and learning-rate adjustments, consistent with its role in controlling student sensitivity. The original full-path result in Table 1 further demonstrates its practical value: SPC achieves DPG 88.44 without SGTS.

Thank the authors for their response. While I still have a few remaining questions regarding certain details in the paper, overall, this work is worth sharing with the community. I will maintain my current positive score.

Dear Reviewer,

Thank you for your reply and for your positive assessment of our work. We appreciate the time you have put into the review and the discussion.

We would be glad to answer your remaining questions if you could share them, and we will make sure the corresponding details are clarified in the final version of the paper.

Thank you again.

Best regards,

The Authors

Reviewer_TdmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work targets error accumulation in multi-step diffusion distillation. Since student transition models are trained locally and chained at inference, single-step prediction errors get amplified by the sensitivity of subsequent steps. Full backpropagation for DMD is costly and numerically unstable. Although SGTS stop-gradient truncation helps stabilize training, it does not fix the inference-time Jacobian and cannot eliminate error amplification.

The authors introduce SPC, a forward-only regularizer. It applies latent perturbations at the same timestep and encourages consistent final predictions. This loss implicitly penalizes the Frobenius norm of the endpoint Jacobian via finite differences, with no need for explicit Jacobian construction or full backpropagation. By annealing perturbation magnitude, the method controls input sensitivity moderately, rather than over-suppressing it, thus reducing over-smoothing and detail loss.

SPC is plug-and-play. It is compatible with SGTS truncation and can stabilize untruncated full-chain training. Evaluations on Qwen-Image, Z-Image and Qwen-Image-Edit show improvements in DMD and PCM distillation workflows.

Critical strengths
  1. This workd identifies that SGTS stop-gradient only stabilizes training without addressing compound error amplification during inference.

  2. SPC requires merely one extra forward pass, introduces no inference overhead, and plugs seamlessly into existing distillation pipelines. It is supported by thorough ablations against alternative regularization schemes.

Critical weaknesses
  1. SPC optimizes only the average Frobenius norm and offers no worst-case spectral-norm guarantee. Error amplification is mitigated empirically rather than with rigorous theoretical bounds. Hutchinson serves as an estimator instead of a strict upper bound, which may cause misinterpretation in the manuscript.

  2. The core motivation targets compound error amplification in chained multi-step distillation, yet SPC also yields gains for one-step distillation. The underlying mechanism behind this observation remains insufficiently analyzed.

  3. Performance is sensitive to hyperparameters. Grid search is required for new models, and adaptive tuning strategies are absent.

  4. Evaluation relies solely on automatic metrics without pairwise human subjective studies.

Questions for authors
  1. SPC constrains the average Frobenius norm of the Jacobian, while error amplification is governed by the spectral norm. Could the authors discuss under which conditions Frobenius-norm penalty indirectly suppresses the spectral norm, and in what settings these two quantities decouple? It should also be clearly stated in the paper that Hutchinson provides a statistical estimator rather than a rigorous mathematical bound to prevent misunderstanding.

  2. The motivation centers on compound error amplification in chained multi-step pipelines, yet experiments show clear SPC gains even for one-step distillation. Could the authors further dissect the mechanism behind such one-step improvements and add supporting diagnostic experiments?

  3. SPC performance depends on hyperparameters. Is it possible to develop an adaptive strategy that avoids grid search and automatically sets perturbation radius and loss weight, to ease transfer across different models?

  4. All conclusions are drawn from automatic metrics. Could small-scale pairwise human A/B tests be added to verify whether the reported metric gains translate to perceptible visual improvements for humans?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Tdmp

Thank you for your thoughtful comments and constructive suggestions.

Q1: Frobenius vs. spectral norm; Hutchinson. The relevant norm relationship is

$ |J|_2^2\le|J|_F^2=\sum_i\sigma_i^2\le\mathrm{rank}(J),|J|_2^2. $

Writing the stable rank as $ \rho=|J|_F^2/|J|_2^2 $ makes the relationship explicit: at fixed $ \rho $, reducing the Frobenius norm reduces the spectral norm. They can decouple when the smaller singular values shrink while the largest remains unchanged or increases. The exact Frobenius norm supplies an upper bound; Hutchinson supplies an unbiased statistical estimate of its square. This distinction will be stated explicitly alongside the diagnostic.

We directly measured both quantities for the two-step models. The initial noise is matched across models, and each model follows its own sampling trajectory. Each column below reports the median across prompts.

Model state∣J∣F2\lvert J \rvert_F^2σmax⁡(J)\sigma_{\max}(J)Stable rank ρ\rho
DMD, t=1t=1441939.792.46
SPC, t=1t=1159926.362.50
DMD, t=0.75t=0.75806959.353.49
SPC, t=0.75t=0.75150212.882.17

The median paired SPC/DMD ratio for $ \sigma_{\max} $ is 0.67 at the first state and 0.29 at the final-step input, with reductions on 27/32 and 29/32 prompts, respectively. The corresponding paired Frobenius-squared ratios are 0.43 and 0.07. Thus, SPC reduces measured spectral sensitivity alongside Frobenius sensitivity, directly supporting the proposed mechanism in these evaluated states.

Q2: Why SPC helps one-step distillation. SPC regularizes the local sensitivity of the endpoint map at any step count. Multi-step composition adds downstream amplification through transition Jacobians; the local sensitivity addressed by SPC is already present in a single-step student.

To test this directly, we tracked the one-step Jacobian on the same validation set as the paper:

Training updateDMD: $ \lvert J \rvert_F^2 $SPC: $ \lvert J\rvert_F^2 $
25043332234
500102174211
1500209965835
2500289315676

DMD's estimate grows substantially, while SPC levels off near 5,200–5,900 in the latter part of training. At update 2500, SPC's estimate is approximately 20% of DMD's. Together with the one-step quality gains in Table 5, this supports the same local-sensitivity interpretation without requiring multi-step composition.

Q3: Hyperparameters. We selected $ \lambda=1 $, $ \gamma=0.01 $, and a cosine schedule on Qwen-Image and reused this setting for Qwen-Image at 1/2/4 steps, Z-Image, Qwen-Image-Edit, and PCM, with no separate per-model grid search. With standard Gaussian initialization, $ \gamma=0.01 $ gives a small coordinate-wise perturbation scale relative to the unit noise standard deviation; the timestep and cosine schedule further reduce the radius through $ r_t=\gamma s_i t $.

Table 6 also shows gains over the corresponding SGTS baseline on DPG and HPS-v2.1 for all tested weights $ \lambda\in{0.1,1,10} $ and radii $ \gamma\in{0.001,0.01} $. These results support a simple shared default across the tested settings. Automatic selection of the radius and weight is a useful extension.

Q4: Human A/B tests. We completed a blinded study with five generative-modeling experts on 100 matched SPC–DMD image pairs. Across both step counts, SPC receives 191 wins, 175 ties, and 134 losses for overall quality; wins also exceed losses for prompt alignment in each setting. These results provide human-evaluation support for the reported improvements. The full protocol and per-setting counts are provided in our response to K2D3, W4.

Dear Reviewer,

Thank you again for your time and for your thoughtful review. As the discussion period closes in a few hours, we wanted to gently follow up on our response.

We have responded to each of your concerns in our rebuttal, and we hope you find the responses satisfactory. If anything remains unclear, we would be glad to clarify right away. If our response has resolved your concerns, we would be grateful if you could consider updating your review to reflect this.

Thank you very much.

Best regards,

The Authors

Reviewer_K2D3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for distillation of multi-step diffusion models. The paper addresses the so-called "compositional stability" issue, which is the amplification of the residual error at one diffusion step caused by Jacobians. Student Perturbation Consistency (SPC) is proposed as a treatment to the issue. SPC acts as a regulariser that aims to minimise the squared Frobenius-norm of endpoint prediction differences (between a predicted state and its small perturbation). The proposed method was experimented with several generative pipelines such as Qwen-Image, Z-Image, and Qwen-Image-Edit on several well-known benchmarks such as DPG-Bench, GenEval, HPS-v2.1, OneIG-Bench, ImgEdit, and GEdit-EN. Experimental results support the method, proven via improved quantitative performance.

Critical strengths
  • The paper studies an important and practical research problem
  • Knowledge gap analysis is convincing. The proposed solution (SPC), although simple, but is reasonably developed.
  • Extensive experiments across baselines and benchmarks were conducted
Critical weaknesses
  • The presentation of the paper needs improvement. In particular, Fig 1 does not convey much information about the proposed method. Fig 2 plays an important role but unfortunately is not described in the text. The Introduction is not engaging; its flow could also be improved. For instance, a general formulation which shows the compositional stability issue should be presented upfront, followed by an analysis and illustration (Fig 2).

  • The achieved improvements are small (especially on GenEval). This leads to a concern about the stability of the method (testing the method with multiple seeds would help to consolidate the improvements).

  • Does SPC make the same effect on every step? In other words, what if SPC is applied to the final step only (rather than intermediate steps)?

  • The paper stated that no user studies were conducted. However, I believe that a user study would make the method more convincing as quantitative metrics may not fully reflect generation quality.

Questions for authors

Please answer questions/issues raised in the weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer K2D3

Thank you for your thoughtful comments and constructive suggestions.

W1: Presentation. We will make the compositional-error formulation $ e_{i+1}\approx J_i e_i+\delta_i $ the organizing thread of the Introduction, connecting residual error, downstream amplification, and SPC. We will explicitly explain Fig. 2 alongside the SGTS analysis and revise Fig. 1 to communicate the problem and method more directly.

W2: Small improvements and multiple seeds. We agree that the GenEval gains are modest. The DMD students already perform close to the full-step teacher on this benchmark, leaving limited headroom relative to this reference. SPC nevertheless improves GenEval, while providing more pronounced gains on DPG and HPS-v2.1. We evaluated the two- and four-step students with three seeds. The table reports mean ± standard deviation.

SettingDPGHPS-v2.1
DMD, 2-step87.72 ± 0.0128.79 ± 0.12
SPC, 2-step88.23 ± 0.0431.33 ± 0.07
DMD, 4-step87.87 ± 0.0530.54 ± 0.17
SPC, 4-step88.45 ± 0.0531.84 ± 0.18

SPC improves both metrics across all three seeds. Mean gains are +0.51 / +2.54 for two steps and +0.58 / +1.30 for four steps (DPG / HPS-v2.1), exceeding the observed sampling variability. These results support the robustness of the improvements to sampling noise.

W3: SPC at the final step only. Final-step-only SPC can be effective because the final endpoint Jacobian influences how upstream errors affect the output. To measure the contribution of other steps, we compared first-step-only, final-step-only, and all-step SPC while keeping DMD anchor sampling unchanged. In the restricted variants, SPC is applied only when the sampled anchor matches the selected step. Thus, the final step receives SPC with the same probability, $ 1/K $, in both final-only and all-step training; all-step training additionally regularizes the other steps.

Method2-step DPG2-step HPS-v2.14-step DPG4-step HPS-v2.1
DMD87.7628.8887.8630.42
SPC, first step only87.8830.7487.9930.84
SPC, final step only88.1331.1588.0231.03
SPC, all steps88.2431.3588.4231.64

Final-step-only SPC improves both metrics at both step counts. It captures most of the two-step gain, while the additional benefit of all-step SPC is larger at four steps: +0.40 DPG and +0.61 HPS-v2.1 over final-only. First-step-only SPC also improves over DMD. These results show that useful regularization is not confined to the final step and support our all-step design.

W4: Human evaluation. Five experts in generative modeling evaluated 50 matched SPC–DMD image pairs for each of the two- and four-step settings, totaling 100 pairs. Each expert rated every pair for overall quality and prompt alignment in a blinded comparison, with ties allowed. The table reports judgment counts from SPC's perspective, with 250 judgments per criterion in each setting.

SPC vs. DMDQuality: Win / Tie / LossAlignment: Win / Tie / Loss
2-step102/79/6998/79/73
4-step89/96/6583/108/59

Dear Reviewer,

Thank you again for your time and for your thoughtful review. As the discussion period closes in a few hours, we wanted to gently follow up on our response.

We have responded to each of your concerns in our rebuttal, and we hope you find the responses satisfactory. If anything remains unclear, we would be glad to clarify right away. If our response has resolved your concerns, we would be grateful if you could consider updating your review to reflect this.

Thank you very much.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.