- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to combine OPD and distillation into a single method that improves the generation quality of the student model while reducing the number of generation steps. The paper conducts experiments on SD 3.5, demonstrating the superiority of the proposed method over training with OPD and distillation sequentially.
Critical strengths
- Combining OPD and DMD into a single distillation method indeed has considerable research value, as it can prevent the gains from the preceding stage from deteriorating when OPD and DMD training are performed sequentially.
Critical weaknesses
- The paper provides relatively weak theoretical justification for the effectiveness of the method. According to the OPDMD loss in Eq. 20, the proposed method can essentially be viewed as a linear combination of the DMD loss and the OPD loss. Moreover, the OPDMD loss is applied only at a single point rather than along the entire ODE sampling trajectory. The paper does not provide a theoretical justification for the effectiveness of this design, but merely describes how the loss is constructed. I would like the authors to theoretically establish the convergence of the OPDMD loss, or at least explain why this design was chosen over other alternatives.
Questions for authors
See “Critical Weaknesses.”
4 comments
3. Single-Point Formulation vs. Full-Trajectory Supervision and Gradient Path Although Eq. 21 is written for a single step , supervision covers all discrete rollout steps in expectation. At each training iteration, we uniformly sample from the rollout nodes (Algorithm 1, Line 13), optimizing the expected objective across the trajectory:
Furthermore, because the single-transition Jacobian is analytically absorbed into the detached target (Algorithm 1, Lines 26 and 30), the detached MSE implements a gradient equivalent to the single-transition surrogate while actual backpropagation proceeds only through the single student prediction . Compared with unrolling gradients through all rollout steps (full-trajectory BPTT), this design avoids multi-step activation memory overhead and avoids backpropagating through long chains of multi-step Jacobians.
Thank you again for your thoughtful review. We hope our theoretical clarifications and empirical comparisons address your concerns, and we welcome any additional questions or suggestions you may have.
Thank the authors for their reply. My concerns have largely been addressed, so I decide to raise my score to wa.
Thank you very much for taking the time to read our response and for raising your score! We sincerely appreciate your constructive feedback and insightful questions, which have greatly helped us strengthen both the theoretical clarity and empirical evaluation of our work. We will make sure to incorporate all the new comparisons, ablations, and clarifications discussed above into the revised manuscript.
Thank you for recognizing the research value of combining OPD and DMD into a single distillation framework. Below, we explain (1) why our design was chosen over naive linear combination alternatives, (2) the design rationale and idealized balance condition of the objective, and (3) how single-node sampling relates to full-trajectory supervision and actual gradient computation.
1. Why This Design Over Alternatives While the target velocity v~k=sg[vT,k+2λhkgk] in Eq. 20 combines two signals, OPDMD differs from a naive linear combination of standard OPD and DMD losses (LOPD+λLDMD) in two ways:
We empirically compare OPDMD against two-stage sequential pipelines, the one-stage Joint OPD + DMD baseline, and Ours (const. weight) (which replaces hk with the mean span hˉ=1/K) at 4 NFE:
OPDMD outperforms both
Joint OPD + DMD(+0.2008 on OCR, +0.0901 on GenEval) and the constant-weight variant (reducing FD from 180.10 to 139.97 and improving OCR from 0.8358 to 0.8835). Together with the λ ablation in Fig. 5, these results support the benefit of our transition-coupled design under the evaluated settings.2. Design Rationale and Idealized Pointwise Balance Condition Our design is motivated by the complementary failure modes of OPD and DMD under coarse sampling: matching only the teacher's instantaneous velocity vT,k over coarse steps hk accumulates discretization error and produces blurry outputs (Fig. 3), whereas optimizing distribution matching alone does not constrain the local transport path and causes style/contrast drift (Fig. 3).
Span-Scaled Correction: In Eq. 20, vT,k provides a local teacher anchor at xk, while 2λhkgk adjusts the target velocity using the correction evaluated at zk, scaled by the transition Jacobian ∂vk∂zk=−hkI.
Idealized Pointwise Balance Condition: Treating vk as an unconstrained vector at a given state, the local surrogate gradient in Eq. 19 vanishes when
vk∗=vT,k+2λhkgk.
In the idealized limit where vϕ=vT, the correction vanishes (gk=0) and the target reduces to the local teacher velocity v~k=vT,k. When the successor correction is nonzero (gk=0), this pointwise balance condition shifts the target velocity away from vT,k by 2λhkgk. This pointwise balance condition clarifies how the objective combines local teacher anchoring and successor-state correction at a fixed state.