- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on addressing the issue of accumulated multi-step inference error caused by the Jacobian matrix in distribution matching distillation (DMD) and related methods. The primary contribution lies in proposing the SPC loss as a Jacobian regularizer to explicitly minimize the Frobenius norm of the Jacobian matrix. The authors conduct experiments on three modern large-scale image generation models(Qwen-Image,etc.), demonstrating the strong practical utility of the proposed method and its advantages over the original DMD method.
Critical strengths
- The paper provides an elegant way to address the issue of a large Frobenius norm of the Jacobian matrix, thereby mitigating the accumulation of multi-step inference error in the student model. The theoretical proof presented in Sec. 3.3 is clear and easy to follow.
- The experiments are conducted on modern flow matching models, demonstrating the scalability and engineering application potential of the method. In Sec. 4, the authors compare against multiple baseline methods on Qwen-Image, and validate the effectiveness of the method across Qwen-Image, Z-Image, and Qwen-Image-Edit.
Critical weaknesses
- Why does SPC lead to increased diversity? Theoretically, the SPC loss decreases the upper bound on the variance of the student's output and encourages the student to be insensitive to variations in noise—that is, producing the same output under different noise inputs. Shouldn't this lead to reduced diversity? However, an increase in diversity compared to DMD is observed across Tables 1, 2, and 3.
- Does the instability of the DMD method truly originate from the Jacobian matrix? Could it potentially arise from other factors, such as an underfitted fake model or an overly large learning rate? If the training ratio between the fake model and student model is further increased, would the DMD method also avoid the sudden loss spikes shown in the left panel of Figure 3?
Questions for authors
Please refer to "Critical Weaknesses"
3 comments
Thank the authors for their response. While I still have a few remaining questions regarding certain details in the paper, overall, this work is worth sharing with the community. I will maintain my current positive score.
Dear Reviewer,
Thank you for your reply and for your positive assessment of our work. We appreciate the time you have put into the review and the discussion.
We would be glad to answer your remaining questions if you could share them, and we will make sure the corresponding details are clarified in the final version of the paper.
Thank you again.
Best regards,
The Authors
Response to Reviewer DTEG
Thank you for your thoughtful comments and constructive suggestions.
W1: Why does diversity increase? Our claim is that SPC preserves diversity in the evaluated settings. SPC compares predictions from an input and its slightly perturbed version at the same timestep. A reduction in an upper bound on output variance does not imply that the actual variance, or a feature-based diversity metric, must decrease.
Table 7 provides a useful comparison: on PCM, SPC improves HPS-v2.1 from 28.01 to 30.96 while diversity and coverage statistics remain largely unchanged. Recall changes from 0.495 to 0.492, Coverage from 0.831 to 0.836, and Vendi from 275.45 to 275.21. The quality improvement therefore coexists with essentially preserved diversity.
For DMD, the increases may partly reflect reduced artifacts and structural distortions (Fig. 5), which can affect feature-based diversity and coverage measurements. Across both objectives, the results show no systematic diversity loss from SPC.
W2: Could the instability instead arise from the fake model or the learning rate? We tested these alternatives directly with two-step full-path DMD over 2,000 student updates. We increased the fake: student update ratio to 10 and 20, and separately reduced the student learning rate to $ 10^{-5} $.
Increasing the fake-model update ratio did not remove the spike, and lowering the learning rate delayed its onset. Adding SPC maintained stable training throughout the same observation window. These controls show that SPC provides a stabilization benefit beyond the tested update-ratio and learning-rate adjustments, consistent with its role in controlling student sensitivity. The original full-path result in Table 1 further demonstrates its practical value: SPC achieves DPG 88.44 without SGTS.