acceptodds
Under review as a conference paper at ICLR 2027

Towards Unbiased On-Policy Distillation for Block Diffusion Language Models

Abstract

On-policy distillation (OPD) has emerged as an effective post-training paradigm for language models, with recent efforts extending it to block diffusion language models (BDLMs). However, existing studies focus almost exclusively on small block sizes, leaving distillation into student models with larger blocks underexplored. In this work, we investigate this regime and reveal two critical optimization biases that induce severe training instability. First, mismatched block boundaries between teacher and student cause context misalignment, providing distorted supervisory signals that misguide student decoding. Second, even under aligned contexts, an intrinsic optimization bias in OPD, where the student tends to rapidly absorb high-support signals while lagging on low-support updates, drives a premature confidence surge that traps weaker students in catastrophic overconfidence collapse. To resolve these problems, we propose Un-OPD, an unbiased on-policy distillation framework with two novelties for stabilizing BDLM training. First, Un-OPD introduces a boundary-aware step filtering strategy that eliminates context-misaligned decoding steps. Second, Un-OPD proposes moderating optimization intensity at high-support positions via a support-rebalanced confidence calibration, thereby bypassing overconfidence collapse. Beyond stability, we also introduce a rollout reuse mechanism to reduce rollout generation overhead. Extensive experiments on math reasoning and code generation benchmarks show that Un-OPD consistently stabilizes training and delivers superior performance while reducing wall-clock training time by approximately half.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On policy distillation, Block Diffusion Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_63apOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies on-policy distillation (OPD) for block dLLMs when the student uses a larger block size than the teacher, and the effect on training stability and accuracy. It identifies two problems and a fix for each, plus one optimization. (i) Context misalignment: with a larger student block, the teacher cannot condition on the student's intermediate state when that state straddles the teacher's block boundary, so its supervision is biased toward over- or under-confidence. Fix: distill only at denoising steps where the teacher's and student's visible contexts coincide, and drop the rest. (ii) Overconfidence collapse, which is applicable to any OPD setting, especially for weak students: positions where the teacher is more confident than the student push confidence up faster than low-support positions pull it down, producing a surge and later degradation. Fix: at those high-support positions, interpolate the teacher's logits toward the student's. The optimization: replay each rollout batch for N updates without importance weights, up to an empirically found limit (N=10), roughly halving GPU-hours.

On SDAR-1.7B/4B students with a TraDo-8B (b=4) teacher, Un-OPD improves math accuracy over both the base model and TOPD (4B-b8: base 56.7, TOPD 57.5, Un-OPD 59.7; 1.7B-b8: base 42.5, TOPD 42.0, Un-OPD 44.1). On the 1.7B student, TOPD ends below the base model. Unlike Un-OPD, the figure shows its accuracy declining over training.

Critical strengths

Strengths

  1. The context-misalignment observation is specific to block-diffusion OPD and appears new; the fix is simple, and the ablation shows it improves accuracy.
  2. The ability to distill into weaker students with block sizes larger than the teacher's is practically useful, since large-block students enable parallel decoding and small-block teachers are the strongest available. The 1.7B-b8 case, where TOPD falls below the base model while Un-OPD improves, is a clear demonstration, and the support-rebalance ablation isolates the fix.
  3. Halving training cost (152 -> 72 GPU-h at <= 0.3 pt) is a useful contribution too.
Critical weaknesses
  1. The lift is marginal, though consistent across datasets. Gains over TOPD are modest.
  2. It looks like Un-OPD uses rollout reuse and TOPD doesn't. Rollout reuse can apply to TOPD too. For the stronger student, we see TOPD accuracy increase before dropping. TOPD sees more prompt rollouts than Un-OPD. It will be useful to also have a TOPD version that uses rollout reuse, so they see a similar number of rollouts.
  3. Authors show the overconfidence mechanism but not after the fix. Fig. 2 plots top-1 confidence with the I+/I- fractions before support rebalance, but it does not show the same curves with support rebalance enabled.
Questions for authors
  1. What fraction of denoising steps survive the alignment filter at b=8 and b=16?
2 comments

We sincerely thank the reviewer for the thoughtful comments and suggestions. Our responses are as follows.

W1: The lift is marginal, though consistent across datasets. Gains over TOPD are modest.

We thank the reviewer for this insightful observation. We agree that gains over TOPD are not uniformly large across all settings. Rather than claim broad accuracy gains, we clarify where Un-OPD is most effective and how its efficiency contribution makes OPD more practical for block diffusion language models.

  • Asymmetric block sizes: Un-OPD is particularly effective when the teacher and student use different block sizes, the setting where their visible contexts can become misaligned. On MATH-500 (Table 1 of manuscript), it improves over TOPD by 4.94.9\\% for SDAR-4B-b8 (67.167.1\\% to 72.072.0\\%) and 4.04.0\\% for SDAR-1.7B-b8 (50.950.9\\% to 54.954.9\\%). The result is especially meaningful for the capacity-constrained 1.7B student: TOPD falls below Base (50.950.9\\% vs. 51.951.9\\%), while Un-OPD prevents this degradation and raises accuracy above Base. This supports the role of context-aligned step selection and support rebalancing in stabilizing distillation when the student is smaller and uses a larger block size.
  • Harder benchmarks: The gains are smaller on harder math bench like AMC-2023. We believe this reflects limits in both the training data and current block diffusion models' reasoning capacity on more difficult problems. Improving reasoning on such tasks remains an open challenge for the Block dLLM community.
  • Efficiency and practical value: When teacher and student use the same block size, accuracy improvements are modest, with support rebalancing providing most of the gain. Un-OPD also introduces rollout reuse to OPD for block diffusion language models. To our knowledge, this is the first use of rollout reuse in this setting. For the 4B student, it reduces training cost from 152152 to 7272 GPU-hours, a reduction of about 5353\\%, while largely preserving accuracy.

Taken together, Un-OPDoffers its strongest accuracy and stability benefits in asymmetric block-size settings, while rollout reuse provides a substantial efficiency gain. Therefore its contribution include both a more robust distillation procedure and a practical way to reduce the training cost of OPD for block diffusion models. This provides insights for community research and practical value.

W2: It looks like Un-OPD uses rollout reuse and TOPD doesn't. Rollout reuse can apply to TOPD too. It will be useful to also have a TOPD version that uses rollout reuse, so they see a similar number of rollouts.

We thank the reviewer for this constructive suggestion. We fully agree that rollout reuse is a general mechanism that can also be applied to TOPD. Following your advice, we implemented a TOPD + Rollout-Reuse baseline so that both methods are trained under the exact same rollout budget and reuse setting. The results on MATH-500 are presented in Table 1:

Table 1. MATH-500 results comparing Un-OPD with TOPD equipped with rollout reuse. Subscripts show gains over Base.

Teacher-StudentMethodMATH-500 (Dynamic)MATH-500 (Static)
TraDo-8B-b4 / SDAR4B-b8Base66.366.368.968.9
TOPD67.1+0.867.1_{+0.8}72.9+4.072.9_{+4.0}
TOPD+Rollout-Reuse67.0+0.767.0_{+0.7}73.0+4.173.0_{+4.1}
UnOPD70.1+3.870.1_{+3.8}74.3+5.474.3_{+5.4}

From the results, we conclude that applying rollout reuse to TOPD yields virtually identical performance to standard TOPD. This indicates that rollout reuse primarily serves as an efficiency-enhancing technique without altering the performance ceiling of the underlying distillation objective.

When evaluated under the exact same rollout reuse setting, Un-OPD still outperforms TOPD + Rollout-Reuse by a substantial margin: +3.1+3.1\\% on Dynamic and +1.3+1.3\\% on Static. This confirms that the accuracy gains of Un-OPD do not stem from seeing reused rollouts, but rather from our core algorithmic designs—namely, context-aligned step selection and support-based rebalancing.

W3: Authors show the overconfidence mechanism but not after the fix. Fig. 2 plots top-1 confidence with the I+/I- fractions before support rebalance, but it does not show the same curves with support rebalance enabled.

We appreciate the reviewer's valuable feedback. We have updated some relevant results in Appendix C.6. In Figure 2, we illustrate the two-stage evolution of confidence levels during the OPD process of BDLM and analyze its impact on training stability. To address this, we introduced the Support Rebalance strategy. As shown in the visualization results in Appendix C.6, this strategy ensures that the mean top-1 confidence increases more gradually during training and eventually stabilizes within a steady range, aligning with our expectations. Notably, however, simply dampening updates for positions with high support ratios does not inherently guarantee performance gains. As revealed by the hyperparameter ablation study in Table 3, a calibrated adjustment of update intensity is more effective than an excessive reduction of supervisory signals at high-support positions.

Q1: What fraction of denoising steps survive the alignment filter at b=8 and b=16?

Thank you for this insightful question. We tracked the fraction of decisions that survive alignment filtering: it is approximately 1515\\% for a student block size of 88, and 1111\\% for a block size of 1616. As the block size increases, the decoding order becomes increasingly non-autoregressive, which naturally reduces the proportion of steps aligned with the teacher's context. While this reduction could potentially affect the efficacy of OPD, the retained ratio remains within a reasonable and effective range.


We hope these responses address your concerns. Please let us know if you have any further questions.

Reviewer_DJ7iOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Un-OPD addresses unstable distillation between block diffusion models with different block sizes. It combines context alignment, support-based reweighting, and rollout reuse. Experiments on SDAR-1.7B/4B show improved mathematics and code performance, while rollout reuse reduces reported GPU hours by approximately 53%.

Critical strengths
  1. The paper identifies a useful problem: different block sizes can give the teacher and student different visible contexts. Context alignment improves 4B MATH500 accuracy from 67.1% to 69.5%.
  2. The method improves both student sizes and offers practical savings. For the 4B-b8 student, mathematics/code averages improve from 57.5/58.7 to 59.7/63.4 over TOPD. Rollout reuse roughly halves training cost with little accuracy change.
Critical weaknesses

The ablations show that the components help, but do not fully isolate the benefit of the selection rules. In Eq. (6), the detached target effectively scales the reverse-KL gradient by λ\lambda at high-support positions. A comparison with uniform downweighting would help establish whether the support criterion matters beyond reducing update strength. Similarly, the alignment comparison does not control for the amount of retained supervision. These gaps limit the explanation of the gains, rather than the evidence that the method works.

Questions for authors
  1. How does support-based weighting compare with uniform downweighting at matched overall update strength?
  2. What checkpoint-selection rule is used for Tables 1–3 and Figure 3?
  3. What fraction of decisions survives alignment filtering, and does alignment still help when the amount of retained supervision is matched?
1 comment

We sincerely thank the reviewer for the thoughtful comments and suggestions. Our responses are as follows.

W1&Q1: In Eq. (6), the detached target effectively scales the reverse-KL gradient by at high-support positions. A comparison with uniform downweighting would help establish whether the support criterion matters beyond reducing update strength.

We appreciate the reviewer's constructive feedback regarding our experimental analysis. To better isolate the benefit of the support criterion, we conducted an additional uniform downweighting experiment (shown in Table 1), where we uniformly scale down the token update gradients of both components by a factor of 0.50.5. The results are summarized below:

Table 1. MATH-500 results comparing uniform downweighting with support-based rebalancing. Subscripts show gains over Base.

Teacher-StudentMethodMATH-500 (Dynamic)MATH-500 (Static)
TraDo-8B-b4 / SDAR4B-b8Base66.366.368.968.9
context alignment step69.5+3.269.5_{+3.2}73.3+4.473.3_{+4.4}
context alignment step + uniform downweighting (0.50.5)69.2+2.969.2_{+2.9}73.5+4.673.5_{+4.6}
context alignment step + support rebalance70.3+4.070.3_{+4.0}74.3+5.474.3_{+5.4}

Relative to context-alignment-only, uniform downweighting changes performance by −0.3-0.3\\% under dynamic decoding (69.269.2\\% vs. 69.569.5\\%) and +0.2+0.2\\% under static decoding (73.573.5\\% vs. 73.373.3\\%), so it provides no consistent gain. Support-based rebalancing further improves over context-alignment-only, reaching 70.370.3\\% and 74.374.3\\%, which supports the value of the support criterion.

Q2: What checkpoint-selection rule is used for Tables 1–3 and Figure 3?

For a fair comparison, we follow the protocol of prior works (TraceRL [A] and TOPD [B]), which select the best-performing checkpoint on the MATH-500 test set for evaluation. Furthermore, to ensure the robustness of our results, we average the performance across the top-33 checkpoints.

W1&Q3: What fraction of decisions survives alignment filtering, and does alignment still help when the amount of retained supervision is matched?

Thank you for this insightful comment. We tracked the fraction of decisions that survive alignment filtering: it is approximately 1515\\% for a student block size of 88, and 1111\\% for a block size of 1616. As the block size increases, the decoding order becomes increasingly non-autoregressive, which naturally reduces the proportion of steps aligned with the teacher's context. While this reduction could potentially affect the efficacy of OPD, the retained ratio remains within a reasonable and effective range. In fact, recent studies [C, D] have also demonstrated that optimizing over all tokens does not necessarily yield the best OPD performance, and selective token optimization is often superior.

To further verify whether alignment still helps when the amount of retained supervision is matched, we analyzed the setting with a student block size of 88 (where approximately 1515\\% of tokens satisfy the context alignment condition). For a fair comparison, we created a baseline based on TOPD that randomly samples 1515\\% of the tokens for gradient updates. The results are shown in Table 2:

Table 2. MATH-500 results comparing context-aligned step selection with TOPD under a matched supervision budget (∼15\sim15\\% retained).

Teacher-StudentMethodMATH-500 (Dynamic)MATH-500 (Static)
TraDo-8B-b4 / SDAR4B-b8Base66.366.368.968.9
TOPD (100100\\%)67.1+0.867.1_{+0.8}72.9+4.072.9_{+4.0}
TOPD (1515\\%)67.3+1.067.3_{+1.0}72.8+3.972.8_{+3.9}
context alignment step (∼15\sim15\\%)69.5+3.269.5_{+3.2}73.3+4.473.3_{+4.4}

As shown in the table, even when TOPD computes the loss over the exact same fraction of tokens (1515\\%), it still significantly lags behind our method. This demonstrates that our performance gains do not stem merely from reducing the number of tokens participating in OPD, but rather from accurately identifying and extracting the critical tokens that facilitate effective capability distillation across models with different block sizes.

We hope these responses can well address your concerns. Please let us know if you have any further questions.


References

[A] Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models, ICLR2026

[B] Trace-Based On-Policy Distillation for Masked Diffusion Language Models, arXiv 2607.16872

[C] TIP: Token Importance in On-Policy Distillation, arXiv 2604.14084

[D] Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation, arXiv 2605.26844

Reviewer_VGdAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an on-policy distillation method for block diffusion language models, focusing on improving the distillation from small-block teachers to large-block students. The authors identify two optimization biases: visible-context misalignment caused by mismatched teacher–student block boundaries, and overconfidence arising from asymmetric learning dynamics under reverse-KL optimization. The proposed method, Un-OPD, selects decoding steps with aligned contexts and interpolates teacher and student logits at positions with high teacher support to moderate over-confidence. Experiments on math and coding show that the method improves overall student performance after distillation, with more benefits for weaker students with larger blocks.

Critical strengths
  1. The paper shows good performance on different models. With SDAR-4B and a block size of 8, Un-OPD improves average math and code accuracy over TOPD by 2+ and 4+ percentage.
  2. Trajectory reuse reduces the reported training cost by approximately half while largely preserving accuracy. Ablation studies further support the respective contributions of context alignment and support-based confidence calibration to training stability and performance.
  3. The paper provides good ablation studies and analysis for the design of Un-OPD. This is helpful for understanding the proposed method.
Critical weaknesses
  1. The experiments use a fixed TraDo-8B teacher and SDAR-family students. It would be helpful to include additional cross-architecture distillation experiments with a more diverse set of teachers and students.
  2. While the appendix includes a block size of 16, the main experiments focus on {4, 8, 16}. It would be useful to evaluate larger block sizes and clarify the method’s limitations as the teacher–student block-size gap increases.
  3. How does Un-OPD handle cases where the student’s block size is not divisible by the teacher’s? Does context-aligned step selection remain effective in this setting?
Questions for authors

Please see the weaknesses.

2 comments

We sincerely thank the reviewer for the thoughtful comments and suggestions. Our responses are as follows.

W1: The experiments use a fixed TraDo-8B teacher and SDAR-family students. It would be helpful to include additional cross-architecture distillation experiments with a more diverse set of teachers and students.

We thank the reviewer for pointing out the limited teacher-student coverage in our original experiments. Here we have conducted additional distillation experiments with two new teachers: DiRL-8B-b4 [A] and TraDo-4B-b4. Specifically, we paired the DiRL-8B-b4 teacher with SDAR-4B students at block sizes 8 and 16 (Table 1), and paired the TraDo-4B-b4 teacher with both SDAR-1.7B and SDAR-4B students at block sizes 8 and 16 (Table 2).

As shown in Tables 1 and 2, Un-OPD consistently outperforms both the corresponding Base models and TOPD on MATH-500 across all evaluated teacher-student pairs under both dynamic and static decoding:

  • With DiRL-8B-b4 as the teacher (Table 1): Un-OPD improves over the Base models by 1.51.5\\% to 2.22.2\\% under dynamic decoding and 2.52.5\\% to 3.93.9\\% under static decoding, consistently surpassing TOPD by 1.61.6\\% to 1.81.8\\% (dynamic) and 1.21.2\\% to 1.61.6\\% (static).
  • With TraDo-4B-b4 as the teacher (Table 2): Un-OPD achieves substantial gains over the Base models ranging from 1.91.9\\% to 4.14.1\\% under dynamic decoding and from 3.13.1\\% to 5.65.6\\% under static decoding. Notably, under dynamic decoding, TOPD even suffers from performance degradation relative to the Base model, whereas Un-OPD effectively prevents this degradation and delivers robust improvements.

Table 1. MATH-500 results using DiRL-8B-b4 as the teacher. Subscripts show gains over Base.

Teacher-StudentMethodMATH-500 (Dynamic)MATH-500 (Static)
DiRL-8B-b4 / SDAR4B-b8Base66.366.368.968.9
TOPD66.7+0.466.7_{+0.4}71.2+2.371.2_{+2.3}
UnOPD68.5+2.268.5_{+2.2}72.8+3.972.8_{+3.9}
DiRL-8B-b4 / SDAR4B-b16Base63.463.467.867.8
TOPD63.3−0.163.3_{-0.1}69.1+1.369.1_{+1.3}
UnOPD64.9+1.564.9_{+1.5}70.3+2.570.3_{+2.5}

Table 2. MATH-500 results using TraDo-4B-b4 as the teacher.

Teacher-StudentMethodMATH-500 (Dynamic)MATH-500 (Static)
TraDo-4B-b4 / SDAR1.7B-b8Base51.951.956.956.9
TOPD51.8−0.151.8_{-0.1}59.8+2.959.8_{+2.9}
UnOPD54.2+2.354.2_{+2.3}61.3+4.461.3_{+4.4}
TraDo-4B-b4 / SDAR1.7B-b16Base49.749.756.356.3
TOPD48.2−1.548.2_{-1.5}57.6+1.357.6_{+1.3}
UnOPD51.6+1.951.6_{+1.9}59.4+3.159.4_{+3.1}
TraDo-4B-b4 / SDAR4B-b8Base66.366.368.968.9
TOPD66.7+0.466.7_{+0.4}72.8+3.972.8_{+3.9}
UnOPD70.4+4.170.4_{+4.1}74.5+5.674.5_{+5.6}
TraDo-4B-b4 / SDAR4B-b16Base63.463.467.867.8
TOPD63.7+0.363.7_{+0.3}70.5+2.770.5_{+2.7}
UnOPD66.3+2.966.3_{+2.9}73.3+5.573.3_{+5.5}

Although Un-OPD consistently outperforms TOPD with DiRL-8B-b4 as the teacher, the overall gains are relatively smaller compared to using TraDo-4B or TraDo-8B. We hypothesize that this stems from a distribution mismatch: DiRL was post-trained on a different data distribution than the one used in our distillation setup, which may limit how effectively its capabilities transfer to the student. This hypothesis aligns well with recent findings [B, C], which show that teacher guidance transfers less effectively in OPD when the teacher’s behavior is misaligned with the student’s reasoning patterns or visited states.

W2: While the appendix includes a block size of 16, the main experiments focus on {4, 8, 16}. It would be useful to evaluate larger block sizes and clarify the method's limitations as the teacher-student block-size gap increases.

We appreciate the reviewer's constructive suggestion. Here we have extended our evaluation to a student block size of 3232 (SDAR4B-b32) using TraDo-4B-b4 as the teacher. The results across student block sizes {8,16,32}\{8,16,32\} are summarized in Table 3:

Table 3. MATH-500 results across increasing student block sizes {8,16,32}\{8,16,32\} using TraDo-4B-b4 as the teacher. Subscripts show gains over Base.

Teacher-StudentMethodMATH-500 (Dynamic)MATH-500 (Static)
TraDo-4B-b4 / SDAR4B-b8Base66.366.368.968.9
TOPD66.7+0.466.7_{+0.4}72.8+3.972.8_{+3.9}
UnOPD70.4+4.170.4_{+4.1}74.5+5.674.5_{+5.6}
TraDo-4B-b4 / SDAR4B-b16Base63.463.467.867.8
TOPD63.7+0.363.7_{+0.3}70.5+2.770.5_{+2.7}
UnOPD66.3+2.966.3_{+2.9}73.3+5.573.3_{+5.5}
TraDo-4B-b4 / SDAR4B-b32Base60.960.965.065.0
TOPD61.0+0.161.0_{+0.1}67.8+2.867.8_{+2.8}
UnOPD63.1+2.263.1_{+2.2}69.9+4.969.9_{+4.9}

First, even at a block size of 3232, Un-OPD improves the student model's performance under both dynamic and static decoding, demonstrating its effectiveness across the evaluated block configurations.

Second, we observe that the performance margin over the Base model narrows as the block size increases. We hypothesize that this is because a larger block size introduces greater randomness in the decoding order, which reduces the proportion of contextually aligned tokens between the teacher and student. This characteristic hinders the effective transfer of knowledge from the teacher to the student.

Overall, we argue that how to extract richer and effective supervision signals under extreme block-size gaps represents an important direction for future research.

W3: How does Un-OPD handle cases where the student's block size is not divisible by the teacher's? Does context-aligned step selection remain effective in this setting?

Thank you for this insightful question. By design, Un-OPD does not require the student block size to be divisible by the teacher block size. This is because context alignment is determined at the token level based on the decoded prefix available when each individual student token is generated, rather than relying on rigid block boundaries.

For instance, consider a teacher block size of 44 and a student block size of 77. For each token in the student block, we can identify whether it is generated before or after the teacher's corresponding 44-token block is fully decoded, and select the appropriate context-aligned teacher step. Thus, our step selection mechanism remains applicable to indivisible block sizes.

Intuitively, this setup can be understood through a sliding-window perspective: we can virtually prepend 11 already-decoded token from the left context of the current 77-token student block so that the window spans 88 tokens. This expanded window roughly functions as a decoding block of size 88, reducing the alignment process to a formulation similar to that used for a student block size of 88. Thus, context-aligned step selection remains applicable by design, with the same prefix-matching criterion and a comparable supervision density to the divisible setting (b=8b=8).


We hope these responses address your concerns. Please let us know if you have any further questions.


References

[A] DiRL: An Efficient Post-Training Framework for Diffusion Language Models, arxiv 2512.22234

[B] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, arxiv 2604.13016

[C] The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes, arxiv 2605.11182

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.