acceptodds
Under review as a conference paper at ICLR 2027

Different Factorizations, Same Path: On-Policy Distillation from Autoregressive to Diffusion Language Models

Abstract

Diffusion language models (DLMs) have drawn increasing attention for their parallel decoding capability, but still lag behind strong autoregressive language models (ARLMs) in generation quality. This motivates the use of on-policy distillation to transfer the stronger generation capability of ARLMs to DLMs. Yet, such cross-architecture distillation is challenging because ARLMs and DLMs employ fundamentally different conditioning structures for token prediction. To bridge this mismatch, we augment the ARLM response distribution with a prescribed forward masking process to construct a teacher distribution over the same path space as the DLM decoding process. We then introduce Path KL to align the student and teacher path distributions, and prove that it upper-bounds the reverse KL between their final response distributions, enabling teacher evaluation of complete-response to guide the student denoising trajectories. Experiments demonstrate consistent improvement across model scales with average benchmark gains of up to 3.15 points, which become more pronounced under highly parallel decoding.

Primary area:
reinforcement learning
Keywords:
diffusion language models, on policy distillation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_riTCOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows a theoretically grounded way to distill the knowledge from a pre-trained / post-trained ARM into a DLM, via on-policy rollouts from the DLM. Since common DLMs are initialized from ARMs, this is a practical research questions to bridge the gap between DLMs and ARMs.

The method is simple, and also quite easy to understand, and can be applied as a post-training method to DLMs. However, most of my concerns are on the experimental validation.

Critical strengths
  1. The paper understands and presents an active gap in ARM to DLM conversion, i.e., OPDLM, a recent DLM that converts an ARM into a DLM. OPDLM, uses on-policy distillation to convert an ARM into a DLM, this paper recognizes the conditioning mismatch and proposed a trajectory based KL between the teacher ARM and student DLM to post-train DLMs. The results are strong.

  2. The paper derives a simple loss for the proposed post-training approach. This part is clear and easy to understand.

Critical weaknesses
  1. My primary concern with this paper is that the results are limited to SDAR. Several other Qwen-based DLMs exist, such as Fast-dLLM and Dream. Extending the proposed method to these models would be important to demonstrate its broader applicability and generalizability.

  2. Figure 4 could be simplified by presenting the results in terms of total variation (TV) distance. This would provide a more formal characterization of the phenomenon the authors aim to illustrate and communicate their argument more clearly.

  3. The paper should discuss the limitations of the proposed approach, particularly the sparsity of the Path-KL objective, which is formulated at the trajectory level. Furthermore, the approach appears closely related to using an ARM as a reward model, with its negative log-likelihood (NLL) serving as a reward signal for trajectories generated by the DLM. The authors should include this baseline and clarify how their method differs from such a formulation, both conceptually and empirically.

  4. The paper is heavily motivated by the limitations of a recent work, OPDLM, which is also included as a baseline. OPDLM performs on-policy distillation from an ARM teacher to a DLM student. As the authors correctly identify, a key limitation of OPDLM is the mismatch between the ARM's causal distribution and the DLM's marginal distributions. However, the paper does not investigate whether applying Path-KL to OPDLM could recover the performance gap relative to other baselines. I believe this is an important missing experiment, particularly given the paper's motivation, and would substantially strengthen its overall narrative.

Questions for authors
  1. Can the authors formalize the difference in an RL setup where the NLL of an ARM is used as a reward for trajectories rolled out from the DLM? A comparison both empirically and theoretically will be useful.

  2. Can the authors use Path-KL to make OPDLM and other Qwen-based DLMs better?

Reviewer_YWWROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce Path KL as a method for dLLMs’ post-training, which distills from an AR-teacher model. This work attempts to close the gap between their distribution difference in distillation, by introducing a forward noising schedule to construct a teacher distribution over trajectories.

Critical strengths
  • The practical gains on the teacher (Qwen) -student (SDAR models) pair are consistent across benchmarks and meaningful.
  • The idea of adding the forward noising mask to construct a normalized teacher distribution over the trajectories while keeping the marginals fixed is another direction than matching the decoding order between AR distribution and DLM distribution.
Critical weaknesses
  • Although OPD is appealing for the token-level guidance, this work as an extension of OPD uses trajectory level signals without enough judgements on the consequences (Alg. 1).
  • The missing details on training budget (other than the training steps mentioned in the text) makes it hard to compare against other post-training methods at a matched setting. For example, OPDLM uses very little data to train.
Questions for authors
  • Could authors elaborate on the design choice and its consequences? For example, does it affect the convergence speed? How does the implementation control the variance?
  • Could authors specify the training budget? Does Path KL improve the accuracy-training-compute trade-off compared to the token-level-OPD or other RL-based post-training?
  • Table 3 is an ablation on the training denoising steps. It is claimed to show the within-block parallelism performance, but by the table itself it is not shown. Varying across the inference budget could help support the claim.
Reviewer_pQaUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation from autoregressive language models to masked diffusion language models. To address their different conditioning structures, the authors augment the teacher’s response distribution with an auxiliary forward masking process and define Path KL over the shared trajectory space. They show that this objective upper-bounds the response-level reverse KL and optimize it using student-generated trajectories. Experiments on three SDAR student sizes show consistent improvements on mathematical reasoning and code generation, including under more parallel decoding.

Critical strengths

The design is coherent and directly addresses the conditioning mismatch in cross-architecture distillation. The autoregressive teacher predicts tokens from causal prefixes, whereas the diffusion student uses partially masked bidirectional contexts, making direct token-distribution matching difficult to interpret probabilistically. The auxiliary masking process extends the teacher’s response distribution to the student’s trajectory space, allowing evaluations of completed responses to supervise the denoising process. Each component corresponds to a clearly identified problem.

The theoretical development clearly explains the relationship between trajectory-level and response-level objectives. The paper decomposes Path KL into response-level reverse KL and conditional trajectory KL, identifying both the source of the upper bound and its gap. The authors also establish normalization of the augmented teacher distribution, preservation of its response marginal, and the constant auxiliary term under linear masking. These derivations make the probabilistic interpretation and assumptions relatively easy to check.

The empirical improvements are consistent, and the controlled comparisons strengthen the evidence for effectiveness. All three student sizes improve over their initial checkpoints on all four benchmarks, with SDAR-4B gaining an average of 3.15 percentage points. Improvements also persist under more parallel decoding. The authors additionally reproduce two OPDLM variants starting from the same SDAR-4B checkpoint, providing more informative evidence about the distillation objective than comparisons with published results alone.

Critical weaknesses

The paper does not sufficiently establish that the implemented sampling process matches the trajectory probabilities used in the objective. The formulation assumes that reveal positions are determined before token sampling, but the implementation is described only as confidence-based decoding. If candidate tokens are sampled before reveal positions are selected using their probabilities, the selection event affects the transition probability, which cannot generally be computed as a product of the selected token probabilities alone. The authors also retain only the top-16 candidates for log-probability computation without explaining whether rollouts use the same truncation, whether probabilities are renormalized, or how tokens outside the candidate set are handled. If sampling and probability evaluation correspond to different distributions, the update cannot directly be interpreted as an unbiased gradient estimator of the stated Path KL objective.

The independent contribution of the path-space construction is not sufficiently distinguished from existing reverse-KL policy-gradient distillation. The upper bound follows from the chain rule for KL divergence, and under linear masking with fixed response length and denoising steps, the auxiliary masking term is constant and does not affect optimization. The resulting objective can therefore be understood as increasing the teacher likelihood of completed responses while encouraging student trajectory entropy. The paper does not sufficiently explain what this formulation adds beyond existing policy-gradient distillation, nor does it isolate the contributions of the teacher-likelihood and entropy terms through ablations. The experiments support the effectiveness of the objective, but do not establish which aspect of the path-space formulation produces the gains.

The main OPDLM comparison uses different training starting points and therefore does not isolate the effect of the distillation method. Original OPDLM converts an autoregressive model into a diffusion model, whereas this paper further distills an already trained SDAR checkpoint. Matching parameter counts does not control for differences in initialization and prior training. The reproduction from the same SDAR-4B checkpoint partially addresses this concern, but the paper does not sufficiently document whether the teacher, training data, rollout counts, training budget, and tuning budget are matched. This controlled comparison should serve as the primary evidence, with its conditions fully specified, rather than relying mainly on larger differences across training settings.

Missing uncertainty estimates and implementation details make the stability of some gains difficult to assess. The HumanEval improvements for the 1.7B and 8B students correspond to approximately two and one additional solved problems, respectively, but the paper reports no repeated runs, confidence intervals, or paired per-problem analysis. Although hardware, learning rate, and training steps are provided, batch size, rollouts per prompt, sampling temperature, EOS handling, and evaluation configurations are insufficiently specified. Since training uses GSM8K and MATH, the authors should also clarify the training splits and deduplication against evaluation data. These details directly affect the credibility of small improvements and the reproducibility of the main conclusions.

Questions for authors
  1. Can the authors provide rollout pseudocode specifying the order of confidence computation, reveal-position selection, and token sampling? Is top-16 truncation applied consistently to sampling and probability evaluation, and are the probabilities renormalized?

  2. When the auxiliary masking term is constant, what distinguishes Path KL from policy-gradient distillation using teacher response likelihood and student trajectory entropy regularization? Can the authors ablate the teacher-likelihood and entropy terms separately to identify the sources of improvement?

  3. Does the controlled comparison between OPDLM-RKL and Path KL use the same teacher, data, rollout counts, and training budget? Can the authors provide the complete configurations and make this comparison the primary experimental evidence?

  4. Can the authors report repeated runs or paired confidence intervals for the main results, together with the missing sampling, evaluation, and data-deduplication details, to establish whether the smaller gains are stable?

Reviewer_4MjdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Path KL, an on-policy distillation method for transferring knowledge from autoregressive language models (ARLMs) to diffusion language models (DLMs). To address the mismatch between their token-level conditioning distributions, the authors augment the AR teacher's response distribution with a forward masking process, allowing the student and teacher to be compared in a shared trajectory space. They derive an upper bound on response-level KL divergence and optimize the resulting objective using student-generated trajectories. Experiments on SDAR models of different sizes show consistent improvements on mathematical reasoning and code generation benchmarks.

Critical strengths
  1. The paper is well written and easy to follow. The motivation for moving beyond token-level distillation is convincing, and the path-space formulation offers a clean way to address the different conditioning structures of ARLMs and DLMs. The theoretical development is also clearly presented.

  2. The method improves performance across all three student sizes and four benchmarks, with average gains of 2.19–3.15 percentage points. The authors also provide useful comparisons against alternative distillation objectives and show that the improvements persist under more parallel decoding settings. Overall, the experiments provide reasonable evidence that the proposed training approach is effective within the evaluated setting.

Critical weaknesses
  1. The experiments are restricted to math and code benchmarks, using a single AR teacher and students from the same SDAR model family. While the improvements are fairly consistent, the evaluation does not establish whether the method generalizes to other DLM architectures or more general language generation tasks. A broader evaluation would make the empirical contribution more convincing.

  2. The benefit of the path-space formulation is not fully isolated. The comparisons against Token KL and Block KL are helpful, but it is still unclear how much of the improvement comes specifically from the proposed formulation, as opposed to on-policy optimization with sequence-level teacher supervision. A comparison against a simpler sequence-level distillation objective under the same rollout and training budget would help clarify this. It would also be useful to report the additional training cost, particularly given the need for student rollouts and AR teacher evaluations.

  3. Some assumptions behind the theoretical formulation need clarification. The derivation assumes that the token positions to be revealed are selected before sampling, such that transition probabilities can be factorized over the revealed positions. However, confidence-based decoding implementations commonly determine which positions to reveal based on the confidence of sampled candidate tokens. In this case, the assumed factorization does not generally hold. Since the paper uses confidence-based rollouts, it is unclear whether the implementation follows the assumptions used in the derivation. This is worth clarifying, particularly as the theoretical justification is a central part of the contribution.

Questions for authors
  1. Could the authors clarify how the reveal positions are selected during training rollouts, and whether the trajectory log-probabilities account for the confidence-based selection procedure?

  2. How would Path KL compare against a simpler on-policy distillation objective based on the AR teacher's sequence-level likelihood, under matched training budgets? This would help better isolate the value of the proposed formulation.

  3. Are there results on other DLM architectures or more general generation tasks? It would also be helpful to understand the method's sensitivity to teacher choice and training compute.

I find the paper well motivated and clearly presented, with promising empirical results. The improvements are consistent, and the ablations provide useful support for the proposed approach. My main reservations are the limited experimental coverage and the lack of a more direct comparison isolating the benefit of Path KL itself. I would also appreciate clarification on the correspondence between the theoretical assumptions and the actual decoding procedure. Overall, I view this as an interesting and potentially useful contribution, although the evidence for its broader applicability and the specific advantage of the proposed formulation could be stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.