Stabilizing Multi-Step Diffusion Distillation with Student Perturbation Consistency

Multi-step diffusion distillation trains student transitions with local objectives but deploys them as a composition. Reducing per-step prediction error is therefore insufficient: residual errors can be amplified by the local gain of subsequent maps. This tension is clear in multi-step Distribution Matching Distillation (DMD). Exact differentiation through the student rollout contains expensive and potentially ill-conditioned products of transition Jacobians, so practical methods detach intermediate states using stop-gradient or SGTS. Truncation makes optimization feasible but leaves a hidden Jacobian gap: local sensitivity is absent from truncated cross-step credit assignment yet remains active at inference and governs residual-error propagation. We propose Student Perturbation Consistency (SPC), a finite-difference regularizer that supplies this missing sensitivity supervision by matching endpoint predictions for clean and perturbed inputs at each supervised state. First-order analysis shows that SPC penalizes average local sensitivity through the squared Frobenius norm of the endpoint-map Jacobian, without explicitly constructing the Jacobian or backpropagating through the rollout. SPC neither reconstructs discarded credit-assignment terms nor imposes hard contraction; it conditions the learned composition while retaining stop-gradient training. On Qwen-Image, SPC suppresses Jacobian growth and reduces endpoint amplification of controlled perturbations. Across Qwen-Image, Z-Image, and Qwen-Image-Edit, SPC consistently improves the same DMD base objective, benefits PCM as a plug-in regularizer, and matches or exceeds released accelerated models on several benchmarks.