- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies on-policy distillation (OPD) for block dLLMs when the student uses a larger block size than the teacher, and the effect on training stability and accuracy. It identifies two problems and a fix for each, plus one optimization. (i) Context misalignment: with a larger student block, the teacher cannot condition on the student's intermediate state when that state straddles the teacher's block boundary, so its supervision is biased toward over- or under-confidence. Fix: distill only at denoising steps where the teacher's and student's visible contexts coincide, and drop the rest. (ii) Overconfidence collapse, which is applicable to any OPD setting, especially for weak students: positions where the teacher is more confident than the student push confidence up faster than low-support positions pull it down, producing a surge and later degradation. Fix: at those high-support positions, interpolate the teacher's logits toward the student's. The optimization: replay each rollout batch for N updates without importance weights, up to an empirically found limit (N=10), roughly halving GPU-hours.
On SDAR-1.7B/4B students with a TraDo-8B (b=4) teacher, Un-OPD improves math accuracy over both the base model and TOPD (4B-b8: base 56.7, TOPD 57.5, Un-OPD 59.7; 1.7B-b8: base 42.5, TOPD 42.0, Un-OPD 44.1). On the 1.7B student, TOPD ends below the base model. Unlike Un-OPD, the figure shows its accuracy declining over training.
Critical strengths
Strengths
- The context-misalignment observation is specific to block-diffusion OPD and appears new; the fix is simple, and the ablation shows it improves accuracy.
- The ability to distill into weaker students with block sizes larger than the teacher's is practically useful, since large-block students enable parallel decoding and small-block teachers are the strongest available. The 1.7B-b8 case, where TOPD falls below the base model while Un-OPD improves, is a clear demonstration, and the support-rebalance ablation isolates the fix.
- Halving training cost (152 -> 72 GPU-h at <= 0.3 pt) is a useful contribution too.
Critical weaknesses
- The lift is marginal, though consistent across datasets. Gains over TOPD are modest.
- It looks like Un-OPD uses rollout reuse and TOPD doesn't. Rollout reuse can apply to TOPD too. For the stronger student, we see TOPD accuracy increase before dropping. TOPD sees more prompt rollouts than Un-OPD. It will be useful to also have a TOPD version that uses rollout reuse, so they see a similar number of rollouts.
- Authors show the overconfidence mechanism but not after the fix. Fig. 2 plots top-1 confidence with the I+/I- fractions before support rebalance, but it does not show the same curves with support rebalance enabled.
Questions for authors
- What fraction of denoising steps survive the alignment filter at b=8 and b=16?
2 comments
W3: Authors show the overconfidence mechanism but not after the fix. Fig. 2 plots top-1 confidence with the I+/I- fractions before support rebalance, but it does not show the same curves with support rebalance enabled.
We appreciate the reviewer's valuable feedback. We have updated some relevant results in Appendix C.6. In Figure 2, we illustrate the two-stage evolution of confidence levels during the OPD process of BDLM and analyze its impact on training stability. To address this, we introduced the Support Rebalance strategy. As shown in the visualization results in Appendix C.6, this strategy ensures that the mean top-1 confidence increases more gradually during training and eventually stabilizes within a steady range, aligning with our expectations. Notably, however, simply dampening updates for positions with high support ratios does not inherently guarantee performance gains. As revealed by the hyperparameter ablation study in Table 3, a calibrated adjustment of update intensity is more effective than an excessive reduction of supervisory signals at high-support positions.
Q1: What fraction of denoising steps survive the alignment filter at b=8 and b=16?
Thank you for this insightful question. We tracked the fraction of decisions that survive alignment filtering: it is approximately for a student block size of , and for a block size of . As the block size increases, the decoding order becomes increasingly non-autoregressive, which naturally reduces the proportion of steps aligned with the teacher's context. While this reduction could potentially affect the efficacy of OPD, the retained ratio remains within a reasonable and effective range.
We hope these responses address your concerns. Please let us know if you have any further questions.
We sincerely thank the reviewer for the thoughtful comments and suggestions. Our responses are as follows.
We thank the reviewer for this insightful observation. We agree that gains over
TOPDare not uniformly large across all settings. Rather than claim broad accuracy gains, we clarify whereUn-OPDis most effective and how its efficiency contribution makes OPD more practical for block diffusion language models.Un-OPDis particularly effective when the teacher and student use different block sizes, the setting where their visible contexts can become misaligned. On MATH-500 (Table 1 of manuscript), it improves overTOPDby 4.9 forSDAR-4B-b8(67.1 to 72.0) and 4.0 forSDAR-1.7B-b8(50.9 to 54.9). The result is especially meaningful for the capacity-constrained 1.7B student:TOPDfalls below Base (50.9 vs. 51.9), whileUn-OPDprevents this degradation and raises accuracy above Base. This supports the role of context-aligned step selection and support rebalancing in stabilizing distillation when the student is smaller and uses a larger block size.AMC-2023. We believe this reflects limits in both the training data and current block diffusion models' reasoning capacity on more difficult problems. Improving reasoning on such tasks remains an open challenge for the Block dLLM community.Un-OPDalso introduces rollout reuse toOPDfor block diffusion language models. To our knowledge, this is the first use of rollout reuse in this setting. For the4Bstudent, it reduces training cost from 152 to 72 GPU-hours, a reduction of about 53, while largely preserving accuracy.Taken together,
Un-OPDoffers its strongest accuracy and stability benefits in asymmetric block-size settings, while rollout reuse provides a substantial efficiency gain. Therefore its contribution include both a more robust distillation procedure and a practical way to reduce the training cost of OPD for block diffusion models. This provides insights for community research and practical value.We thank the reviewer for this constructive suggestion. We fully agree that rollout reuse is a general mechanism that can also be applied to TOPD. Following your advice, we implemented a TOPD + Rollout-Reuse baseline so that both methods are trained under the exact same rollout budget and reuse setting. The results on MATH-500 are presented in Table 1:
Table 1. MATH-500 results comparing Un-OPD with TOPD equipped with rollout reuse. Subscripts show gains over Base.
From the results, we conclude that applying rollout reuse to TOPD yields virtually identical performance to standard TOPD. This indicates that rollout reuse primarily serves as an efficiency-enhancing technique without altering the performance ceiling of the underlying distillation objective.
When evaluated under the exact same rollout reuse setting, Un-OPD still outperforms
TOPD + Rollout-Reuseby a substantial margin: +3.1 on Dynamic and +1.3 on Static. This confirms that the accuracy gains of Un-OPD do not stem from seeing reused rollouts, but rather from our core algorithmic designs—namely, context-aligned step selection and support-based rebalancing.