- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper shows a theoretically grounded way to distill the knowledge from a pre-trained / post-trained ARM into a DLM, via on-policy rollouts from the DLM. Since common DLMs are initialized from ARMs, this is a practical research questions to bridge the gap between DLMs and ARMs.
The method is simple, and also quite easy to understand, and can be applied as a post-training method to DLMs. However, most of my concerns are on the experimental validation.
Critical strengths
-
The paper understands and presents an active gap in ARM to DLM conversion, i.e., OPDLM, a recent DLM that converts an ARM into a DLM. OPDLM, uses on-policy distillation to convert an ARM into a DLM, this paper recognizes the conditioning mismatch and proposed a trajectory based KL between the teacher ARM and student DLM to post-train DLMs. The results are strong.
-
The paper derives a simple loss for the proposed post-training approach. This part is clear and easy to understand.
Critical weaknesses
-
My primary concern with this paper is that the results are limited to SDAR. Several other Qwen-based DLMs exist, such as Fast-dLLM and Dream. Extending the proposed method to these models would be important to demonstrate its broader applicability and generalizability.
-
Figure 4 could be simplified by presenting the results in terms of total variation (TV) distance. This would provide a more formal characterization of the phenomenon the authors aim to illustrate and communicate their argument more clearly.
-
The paper should discuss the limitations of the proposed approach, particularly the sparsity of the Path-KL objective, which is formulated at the trajectory level. Furthermore, the approach appears closely related to using an ARM as a reward model, with its negative log-likelihood (NLL) serving as a reward signal for trajectories generated by the DLM. The authors should include this baseline and clarify how their method differs from such a formulation, both conceptually and empirically.
-
The paper is heavily motivated by the limitations of a recent work, OPDLM, which is also included as a baseline. OPDLM performs on-policy distillation from an ARM teacher to a DLM student. As the authors correctly identify, a key limitation of OPDLM is the mismatch between the ARM's causal distribution and the DLM's marginal distributions. However, the paper does not investigate whether applying Path-KL to OPDLM could recover the performance gap relative to other baselines. I believe this is an important missing experiment, particularly given the paper's motivation, and would substantially strengthen its overall narrative.
Questions for authors
-
Can the authors formalize the difference in an RL setup where the NLL of an ARM is used as a reward for trajectories rolled out from the DLM? A comparison both empirically and theoretically will be useful.
-
Can the authors use Path-KL to make OPDLM and other Qwen-based DLMs better?