- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies few-step distillation for diffusion and flow models. The student predicts a single velocity for each sampling interval and learns from teacher predictions queried within that interval. The authors add two components to this baseline: a training-only Pivot branch that models within-interval velocity variations, and a repulsive term based on weaker teacher predictions. Only the student is used at inference. Experiments cover image generation, image editing, video, 3D, and joint audio-video generation, with additional studies on cross-model and multi-teacher distillation.
Critical strengths
-
The approach is straightforward and does not require an online student-score model or adversarial training. Pivot is removed at inference and therefore adds no inference-time network evaluations.
-
Showing that simple regression can produce competitive four-step results across several large models has practical value. The FLUX GenEval results and the motion activity of Wan-14B are particularly encouraging.
-
The experiments cover a broad range of tasks. The authors also report limitations and mixed results, including cases where cross-architecture supervision hurts performance and the lack of clear statistical evidence for visual improvements in H3. These disclosures help establish the method's scope.
Critical weaknesses
I have three main concerns.
First, the contribution of the repulsive term is not sufficiently clear. Under the paper's stop-gradient implementation, the loss can be rewritten as weighted regression toward an extrapolated teacher target:
Under standard linear CFG without nonlinear rescaling, this can also be interpreted as increasing the effective guidance scale. It is therefore unclear what additional benefit the formulation provides beyond adjusting the teacher target or guidance strength. The connection to existing guidance methods deserves a more explicit discussion.
Second, the experiments do not directly establish improved training stability. Tables 5-8 show better scores for some configurations at particular checkpoints, but do not show whether Pivot reduces fluctuations during training. Figure 3 illustrates fluctuations in the baseline without a corresponding Pivot comparison, and each ablation setting uses only one training run. In addition, the main Wan-14B checkpoint uses zero active Pivot residual, so that result does not provide evidence that Pivot stabilizes training at larger scale.
Finally, some improvements are small, and parts of the comparison need clarification. The Qwen comparison includes PDD-Euler, while the four-NFE Midpoint variant reported in the PDD paper is closer to SMD. The ImgEdit improvement from 4.48 to 4.51 also needs to be considered alongside evaluation variability. SMD has lower image diversity than Pi-Flow and PDD, and does not lead on video semantic scores. These results establish a competitive method, but do not yet demonstrate a consistently better overall trade-off.
Questions for authors
-
How does the repulsive objective differ from extrapolating the teacher target or increasing guidance? After matching the effective guidance strength, is there still an additional benefit?
-
Could the authors report the baseline, Pivot-only, repulsion-only, and full method under one matched training recipe? For the stability claim, learning curves with and without Pivot, together with independent training seeds on a representative model such as Wan-1.3B, would be helpful.
-
Why were the reported PDD configurations selected rather than the stronger Midpoint variants? Also, were SMD checkpoints selected using an independent validation set or the same benchmarks used for final reporting?
-
For small differences such as those in image editing, could the authors provide paired confidence intervals or repeated evaluations to establish whether the improvements are reliable?
3 comments
Part-2
Continued response to Reviewer CPe2.
3. Comparison Protocol
PDD Comparison
In our main Qwen-Image table, we quoted PDD-Euler because Euler queries the teacher once per local target, matching SMD's single-velocity target construction, whereas PDD-Midpoint refines each target through an additional midpoint query. For Wan2.1-14B, we reported PDD-Midpoint because the PDD paper does not report an Euler variant at that scale.
When compared directly against PDD-Midpoint at 4 NFE, SMD still achieves higher scores on Qwen-Image GenEval, DPG-Bench, and OneIG Overall, and matches PDD-Midpoint on Wan2.1-14B VBench Quality and Semantic scores. On Wan2.1-1.3B, SMD trails PDD-Midpoint on VBench Total at 84.31 versus 84.94, but substantially outperforms PDD-Euler on Semantic alignment at 78.97 versus 72.50 (Table 3). Notably, SMD achieves these results using a global batch size of 16 for both Qwen-Image and Wan2.1, compared to 2,048 and 256 in PDD.
| Model | Metric | PDD-Midpoint | SMD |
|---|---|---|---|
| Qwen-Image | OneIG Overall | 0.538 | 0.540 |
| Qwen-Image | DPG | 88.66 | 88.7 |
| Qwen-Image | GenEval | 0.860 | 0.873 |
| Qwen-Image | OneIG Diversity | 0.174 | 0.144 |
| Wan2.1-14B | VBench Quality | 85.69 | 85.72 |
| Wan2.1-14B | VBench Semantic | 80.71 | 80.52 |
Checkpoint Selection
Because evaluating a complete benchmark is time-consuming, we evaluate a small subset of benchmark examples to help select checkpoints. We have clarified this procedure in the appendix.
4. Evaluation Reliability
To account for VLM judge variance in image editing, we re-evaluated the same generated outputs on Qwen-Image-Edit-2511 five times and now report the mean and standard deviation in Table 2 and Table 14. Across repeated evaluations, SMD and Lightning tie on ImgEdit Overall at , while SMD achieves clear gains on specific editing operations such as Removal, rising from to , and Adjustment, rising from to , alongside consistently higher mean scores across all GEdit-Bench dimensions. These results confirm that simple on-policy regression delivers competitive 4-NFE editing performance without auxiliary score networks or adversarial losses.
| Method | ImgEdit Overall | ImgEdit Rem. | ImgEdit Adj. | GEdit SC | GEdit PQ | GEdit Overall |
|---|---|---|---|---|---|---|
| Lightning | 4.48 +/- 0.02 | 4.52 +/- 0.07 | 4.40 +/- 0.07 | 8.95 +/- 0.08 | 7.73 +/- 0.09 | 8.18 +/- 0.09 |
| SMD | 4.48 +/- 0.02 | 4.70 +/- 0.05 | 4.56 +/- 0.08 | 8.98 +/- 0.05 | 7.79 +/- 0.09 | 8.21 +/- 0.07 |
5. Diversity and Semantics
We agree that the diversity trade-offs observed in image generation do not trivially translate to semantic gains in video generation.
Specifically, SMD relies on a simple regression objective and operates under limited computational resources, utilizing LoRA rather than full-parameter fine-tuning and smaller batch sizes compared to methods like PDD and rCM. Nevertheless, SMD remains highly competitive across tasks, demonstrating that simple on-policy distillation can achieve robust performance under lightweight training constraints. We have also updated our anonymous project page with more qualitative comparisons against LightX2V and AnyFlow across text-to-image generation, image editing, and text-to-video generation.
Thanks for clarifying how the repulsive term works. The new CFG comparison and ablations address my main concerns, so I’m raising my score to 3 (Weak Accept).
Part-1
We thank Reviewer CPe2 for the thorough mathematical and empirical critique, and for recognizing the practical value of our simple on-policy regression framework across diverse tasks. We have updated the manuscript with the requested theoretical clarifications, matched four-way ablations, training-progression comparisons, and multi-run evaluation statistics.
1. Repulsion and CFG
The extrapolated-target formulation holds under squared Euclidean distances, where standard linear CFG with target scale wT=5.0, reference scale wref=0, and λ=0.1 reduces to an effective guidance scale of weff=(wT−λwref)/(1−λ)≈5.56. However, SMD optimizes the unsquared L2 distance (Appendix C.2, Eq. 28). For a predicted velocity v with nonzero residuals, the gradient with respect to v is
∇vℓ=∣v−vT∣2v−vT−λ∣v−v~T∣2v−v~T.
Because the positive and negative residuals are normalized separately by their respective norms ∥v−vT∥2 and ∥v−v~T∥2, their relative weighting changes dynamically with the current student prediction rather than reducing to a fixed linear extrapolation of the teacher target. Among student predictions with identical teacher-matching error, those lying closer to the weak reference v~T incur a steeper repulsive gradient, introducing a directional preference motivated by weak-model guidance.
To test whether repulsion provides benefits beyond matching this effective guidance strength, we added a controlled comparison on Wan2.1-1.3B at 4 NFE (Appendix I.3, Table 30 and Figure 30). Holding Pivot and all other training settings fixed, we compare CFG 5.0 with repulsion (λ=0.1, wref=0) against CFG 5.6 without repulsion. We also evaluate extracted video frame grids using GPT-5.6 Sol for joint Realism and Saturation (R&S) on a 0–5 scale. Repulsion at CFG 5.0 outperforms CFG 5.6 across VBench Quality, Semantic, and Total scores while achieving a substantially higher R&S score of 3.182 compared to 2.853, avoiding the color over-saturation caused by simply raising teacher CFG.
2. Component Ablations and Stability
We have added matched four-way ablations covering the baseline, Pivot-only, repulsion-only, and full SMD across Qwen-Image, Wan2.1-1.3B, and Wan2.1-14B at 4 NFE (Appendix I.1, Tables 28–29). Without repulsion, enabling Pivot improves VBench Total from 83.04 to 83.78 on Wan2.1-1.3B and from 83.48 to 84.19 on Wan2.1-14B. When combined with repulsion, adding Pivot raises Qwen-Image OneIG Overall from 0.534 to 0.540, with Text accuracy increasing from 0.921 to 0.944, and improves Wan2.1-1.3B VBench Total from 84.11 to 84.31. On Wan2.1-14B, repulsion alone already reaches a VBench Total of 84.68; adding Pivot further increases Semantic alignment from 80.52 to 80.80 while trading off a slight shift in Quality from 85.72 to 85.63, resulting in a nearly identical Total of 84.66.
Qwen-Image
Wan2.1-1.3B and Wan2.1-14B
To directly examine training stability, we added side-by-side checkpoint trajectories with and without Pivot on Qwen-Image and Wan2.1-14B under matched prompts, initial noise, and training iterations with repulsion disabled (Appendix I.2, Figures 28–29). On Qwen-Image, the baseline without Pivot suffers from intermediate structural and text regressions between iterations 300 and 500—such as distorted stop-sign lettering and a neon night skyline temporarily collapsing into a daylight scene—whereas enabling Pivot maintains consistent geometry, readable text, and stable scene appearance throughout training. Similarly, on Wan2.1-14B, Pivot preserves more consistent scene composition across intermediate checkpoints.