acceptodds
Under review as a conference paper at ICLR 2027

d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models

Abstract

Large language models (LLMs) typically generate text autoregressively (AR), predicting one token at a time. Block diffusion language models (dLLMs) instead generate blocks sequentially while denoising multiple tokens in parallel within each block, offering a promising way to accelerate generation. Rather than training such models from scratch, recent work adapts strong pretrained AR models into block dLLMs through distillation. On-policy distillation (OPD) has been widely used for LLM training because it supervises the student on states generated by its current policy, rather than only on fixed offline trajectories. By training on the states the student actually visits, it reduces the mismatch between training and generation and can provide more relevant supervision as the student evolves. Recent work has extended this idea to AR-to-block-diffusion conversion. However, this setting introduces a fundamental mismatch in supervision: the block-diffusion student and the causal AR teacher condition on different information at the same training state. The student predicts from the entire partially denoised block, including visible future context, whereas the standard AR teacher target is defined only from the causal prefix. As a result, the teacher distribution used for distillation is not fully aligned with the information available to the student. We therefore introduce d-OPD, a future-aware on-policy distillation method that corrects the AR teacher distribution to better align with the student-visible state by incorporating visible future information within each block, providing supervision that better matches the information used by the student. Across Qwen3 models from 0.6B to 8B, d-OPD improves the six-benchmark average by up to points over OPDLM and reduces training time by -.

Primary area:
foundation or frontier models, including LLMs
Keywords:
block diffusion, diffusion language models, knowledge distillation, on-policy distillation, large language models, future-aware correction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_ExaZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes d-OPD as a future-aware on-policy distillation approach that aims to transform an AR teacher model into a block diffusion student model. The core observation is that the existing method OPDLM suffers from the target-conditioning mismatch, where the student receives supervision that comes from the AR teacher without any awareness of the subsequent denoised tokens. Upon detailed derivation, the proposed method reveals such a mismatch and proposes an empirical fix that restricts the correction to the joint of the teacher and student top-k candidates on the verifier-correct rollouts. Experiments have been performed on Qwen3 family models with various sizes, showing that the proposed approach improves over OPDLM in terms of accuracy, with moderate training-time speedup.

Critical strengths
  1. The paper is well motivated by a key observation on the target-conditioning mismatch between the AR teacher and the block diffusion student.

  2. Experiments show that the method improves benchmark scores while also obtaining decent training-time speedup.

Critical weaknesses
  1. The paper requires a practical surrogate to the theoretically motivated student state-conditioned posterior. The exact target from Eq.7-10 needs to be empirically transformed using only the completed suffix from a single student rollout. Therefore, the resulting target is not necessarily an unbiased estimate of the exact theoretically derived target.

  2. The ablation study does not entirely isolate the effect of the verifier-correct rollout and the future-aware correction. A more controlled study that isolates the effect of these two components would strengthen the paper further.

Questions for authors

Q1. The exact posterior conditions on the student-visible state, while the practical correction scores the completed suffix. Could the authors compare the proposed approach against a variant that uses only the visible future tokens, marginalizing or approximating the likelihood of intervening masked positions?

Q2. Could the authors better isolate the contribution of future-likelihood reweighting from correctness gating, in the ablation study?

Reviewer_jHS4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In on-policy distillation from an AR teacher to a block diffusion student, the teacher target uses only the causal prefix, while the student can see unmasked tokens to the right within the block. d-OPD reweights the teacher distribution by how likely the completed future is under each candidate token. On Qwen3 models from 0.6B to 8B, it improves the six-benchmark average over OPDLM by 2.6–4.0 points and reaches OPDLM’s best performance 1.35–1.58× faster.

Critical strengths
  1. The problem is clearly stated, and the method follows directly from Bayes’ rule (Eq. 7–10). When the completion is drawn from the teacher posterior, the corrected target is an unbiased estimate of the exact target, so the forward-KL gradient is also unbiased. Stating this property would make the method easier to justify.

  2. The method is simple and can be added to existing OPD training with a small reported overhead. The gains are consistent across four model scales and three block sizes, and the enumerable diagnostic in Section 3.3 directly supports the motivation.

Critical weaknesses

Missing code tasks. The evaluation covers only knowledge and math benchmarks. The training mixture inherited from OPDLM contains about 21.6K code prompts, and OPDLM reports HumanEval, MBPP, LiveCodeBench and IFEval. Code is also where OPDLM is relatively weak. It is also unclear whether code prompts receive the future-aware correction. I would ask the authors to report results on code benchmarks and IFEval, and to state the fraction of rollouts and positions that receive the correction in each data domain.

Questions for authors
  1. The teacher scoring cost grows with block size. When the block size increases, more masked positions per block need correction. The reported 2.63% overhead is for the default setting. Could the authors report the scoring cost (e.g., scored tokens, FLOPs or GPU hours) and the per-iteration overhead for N = 8 and N = 16?

  2. How does d-OPD perform on code tasks such as HumanEval, MBPP and LiveCodeBench? Does the future-aware correction help more or less there than on math?

Reviewer_9HhAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to mitigate the distribution mismatch between the student (block dLLM) and teacher (autoregressive) models under on-policy distillation (OPD). The authors claim that in block dLLM, when the student predicts a masked token, it can utilize "future context," whereas the autoregressive (AR) teacher makes predictions conditioned solely on the causal prefix. Consequently, the conditioning information available to the teacher and the student for a given student state does not align. To address this, the authors propose d-OPD. d-OPD reweights the standard causal distribution of the AR teacher based on the future information visible to the student. Experiments using Qwen3 (0.6B to 8B models) show that d-OPD improves the average score across six benchmarks by up to 4.0 points compared to OPDLM (a primary baseline), while also reducing the wall-clock training time required to reach equivalent performance to OPDLM by a factor of 1.35 to 1.58.

Critical strengths
  1. Clear problem setting. The paper clearly defines the problem that arises when distilling the AR teacher's behavior to the block dLLM student.
  2. Organized theoretical analyses. The paper carefully analyzes the "target-conditioning mismatch" and establishes how to approximate the future-conditioned target distribution.
  3. Experimental results. The paper evaluates d-OPD on various benchmarks across several model sizes, showing that d-OPD outperforms the primary baseline, OPDLM (an OPD method for block dLLM WITHOUT future conditioning for the AR teacher).
Critical weaknesses
  1. Insufficient theoretical guarantee. The paper does not explicitly provide proofs or evidence to justify the use of the approximation methods in d-OPD.
  2. Concern on reproducibility. The paper lacks information on the "correctness gate" and "task verifier" in d-OPD and the experimental details of the ablation study.
  3. Concerns about the quality of writing. The paper never refers to Figures 1 and 2 in its main text. Figure 2 is too abstract to understand the proposed method.
Questions for authors
  1. Please cite Figures 1 and 2 in your paper.
  2. Please explicitly provide proofs or evidence to justify the use of the following approximation method and technique:
  • Avoiding the direct calculation of the "exact future-conditioned teacher target distribution" by instead using the completed prefix and completed future from the same on-policy rollout as a single approximation.
  • Implementing a "top-kk candidate restriction" to reduce computational complexity.
  1. What are "correctness gate" and "task verifier" in d-OPD?
  2. In Table 1, the results of all methods and all model sizes on AIME2025 are smaller than I expected. Even vanilla models have low accuracy. Do you have any insights into this? I guess Qwen3-8B typically achieves ~70% accuracy on AIME2025.
  3. Please provide the experimental details of the ablation study (Table 3).
Reviewer_ZZ8NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses a target-conditioning mismatch in on-policy distillation from a causal AR teacher to a block diffusion student. The student predicts a masked token from all visible tokens in the block, including those to its right, but the AR teacher targets ri(v)=pT(v∣q,z,x^<i)r_i(v)=p_T(v \mid q, z, \hat{x}_{<i}) conditions only on the left prefix. The paper derives the correct target as the teacher's block-posterior marginal, which factors into the causal prior times a future-compatibility likelihood. Since that likelihood is intractable, d-OPD estimates it from one completed rollout: candidates vv are reweighted by exp⁡(Δsi(v))\exp(\Delta s_i(v)), the teacher log-likelihood of the completed suffix C^i\hat{C}_i given vv relative to the rollout token x^i\hat{x}_i. Only the teacher and student top-kk tokens are scored, and only rollouts that pass a verifier are reweighted. On Qwen3 0.6B–8B, d-OPD improves the six-benchmark average over OPDLM by 2.6–4.0 points and reaches OPDLM's best accuracy 1.351.35–1.58×1.58\times faster, at about 2.6% extra cost per iteration.

Critical strengths
  1. A well-defined, valuable research question solved in a mathematically sound way: the paper derives the correct teacher target from the block posterior and approximates it with a cheap change that touches only the target.
  2. Consistent gains over all baselines in every setting, with faster convergence and negligible overhead.
Critical weaknesses
  1. Figure 2 does not help explain the method. Ideally the figure would complement the text and help readers understand d-OPD; at present a reader has to understand the method from the text before the figure makes sense. Unclear points: (1) where the completed prefix A^i\hat{A}_i comes from; (2) how the candidates v1,v2,v3v_1, v_2, v_3 are obtained; (3) which tokens receive gradients from the distillation loss; (4) how the orange token vv in A^i\hat{A}_i relates to v1,v2,v3v_1, v_2, v_3 and x^i\hat{x}_i.
Questions for authors
  1. Future scores si(v)s_i(v) are computed only for a small set of high-probability candidates. What are the teacher's sampling settings here (e.g., is any temperature applied to rir_i when choosing candidates or scoring)? Is d-OPD performance sensitive to these settings, and to the rollout temperature and kk? A study of downstream accuracy would be great; Table 3 reports only probability coverage and overhead.
  2. How does d-OPD change the trade-off between accuracy and decoding speed (tokens per step, throughput) compared with OPDLM and the AR teacher?

Following the weaknesses above, I suggest revising Figure 2 so it complements the text; currently it only makes sense after reading the method. Unclear points: (1) where the completed prefix A^i\hat{A}_i comes from; (2) how the candidates v1,v2,v3v_1, v_2, v_3 are obtained; (3) which tokens receive gradients from the distillation loss; (4) how the orange token vv in A^i\hat{A}_i relates to v1,v2,v3v_1, v_2, v_3 and x^i\hat{x}_i.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.