acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Residual Learning for Diffusion Language Models

Abstract

Residual context is emerging as a key ingredient for unlocking the speed–quality Pareto frontier of diffusion language models (DLMs) by carrying forward predictive distributions across denoising steps. Yet turning this seemingly simple mechanism of computation recirculation into an effective and efficient learning paradigm is far from simple: there is no clear recipe for how residual context should be learned. We introduce ORL, a framework for on-policy residual learning in DLMs. Algorithmically, we identify that residual learning is inherently on-policy. ORL learns directly from these model-induced residual trajectories on the fly and introduces an entropy-contraction objective that encourages sharpening predictions as the model revisits its own intermediate predictions. System-wise, exact residual supervision requires capturing, retaining, and repeatedly processing many trajectory-dependent intermediate states. ORL co-designs the learning algorithm with asynchronous residual capture and transport, specialized block-diffusion attention, and packed compute-balanced training, making on-policy residual learning practical at scale. On a 4B DLM converted from Qwen3-4B, ORL improves tokens per forward (TPF) by up to 6.2× and end-to-end throughput by approximately 5× at comparable or better generation quality. On a single H100, ORL-trained models achieve a stronger speed–quality Pareto frontier than autoregressive models accelerated with EAGLE3 and DFlashV1 speculative decoding, under both strict and relaxed verification. On LLaDA-2.1-mini, 36 H100-hours and 4.2M self-generated tokens, about 10× less post-training data than prior LLaDA parallel-decoding recipes, yield up to 1.5× higher TPF and up to 3× higher throughput at matched quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Model, On-policy optimization, Residual Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8dCQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ORL, a method to train diffusion language models on the intermediate states they produce during generation. It helps the model make decisions in fewer steps and includes system changes that make the training practical. Experiments on three model families show faster decoding with generally similar accuracy.

Critical strengths
  1. ORL trains on fresh, verified trajectories from the current model, so it learns from the states it actually encounters during generation.
  2. The authors release their training and evaluation code. This makes the results easier to reproduce.
Critical weaknesses
  1. Quality drops on some additional tasks, even though decoding gets faster.
  2. The reported post-training cost leaves out the earlier AR-to-DLM conversion for QwenDLM.
Questions for authors

How much compute did the AR-to-DLM conversion and offline initialization require before ORL training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8dCQ (Part 1/2)

We thank you for the feedback. We addressed the questions below.

W1. Quality drop on some additional benchmarks

We would like to clarify the following two points.

(1) The observed accuracy drops are within evaluation variance.

We additionally analyze the standard deviations of the benchmark results. As shown below, all observed accuracy drops are within one standard deviation, while the TPF improvements are consistently substantial.

ModelBenchmarkΔAcc / σΔTPF / σ
LLaDAAIME-24−1.7 / 12.9 (0.13σ)+2.46 / 0.54 (4.6σ)
LLaDAGPQA-D−3.4 / 5.0 (0.68σ)+1.16 / 0.15 (7.9σ)
LLaDAIFEval−1.8 / 2.3 (0.79σ)+0.21 / 0.06 (3.6σ)
DiffusionGemmaMMLU-Pro−0.5 / 0.5 (0.94σ)+4.03 / 0.07 (56.4σ)
DiffusionGemmaGPQA-D−0.4 / 4.8 (0.08σ)+3.77 / 0.46 (8.3σ)
DiffusionGemmaLCB-v6−2.1 / 5.3 (0.39σ)+4.13 / 0.68 (6.1σ)

These results suggest that the small accuracy fluctuations are not statistically conclusive, whereas the decoding efficiency improvements are much larger relative to their measured variability.

(2) ORL achieves higher decoding efficiency at matched or better accuracy.

Importantly, the original results in Table 2 use the aligned decoding configurations for each model. Since ORL improves refinement efficiency, we can adopt stricter decoding criteria to recover generation quality while maintaining a substantial speed advantage.

We conducted additional evaluations with adjusted decoding settings to match or exceed the baseline accuracy:

ModelBenchmarkOriginal ΔAccAligned ΔAccTPF speedup
LLaDAAIME-24−1.70.01.26×
LLaDAGPQA-D−3.40.01.16×
LLaDAIFEval−1.8+0.11.10×
DiffusionGemmaLCB-v6−2.10.01.12×
DiffusionGemmaMMLU-Pro−0.5+0.21.10×
DiffusionGemmaGPQA-D−0.4+0.51.13×

These results demonstrate that ORL consistently maintains or improves generation quality while achieving 1.10–1.26× higher TPF. The small accuracy differences in Table 2 therefore do not indicate an inherent quality degradation, but reflect particular operating points on the speed–quality trade-off.

We will include the variance analysis and these additional accuracy-aligned evaluations in the revised manuscript to provide a more comprehensive characterization of ORL's speed–quality trade-off.

AuthorsOpenReview ↗

Response to Reviewer 8dCQ (Part 2/2)

W2 & Q1. Training cost for converting Qwen AR model to QwenDLM

Thank you for raising this question. AR-to-DLM conversion is an increasingly common approach to developing diffusion language models. ORL is a general framework that can be applied either to AR-converted DLMs or directly to existing pretrained DLMs, without requiring a conversion stage. We have made the distinction between AR-to-DLM conversion and ORL post-training, including their respective training costs, more explicit in our updated manuscript.

Importantly, AR-to-DLM conversion is not a prerequisite for ORL. As demonstrated in Table 1, we apply ORL directly to existing DLMs, including LLaDA-2.1-mini and DiffusionGemma, achieving consistent improvements without any conversion overhead. This demonstrates the generalizability of ORL across different model families and initialization strategies.

For QwenDLM, we initialize from Qwen3-4B through an SDAR-style [1] offline SFT conversion stage, followed by ORL post-training. The AR-to-DLM conversion uses approximately 10 billion training tokens, a relatively modest data budget compared with prior SDAR-series conversion pipelines (50B tokens in SDAR). The subsequent ORL post-training requires only 60 H100 GPU-hours.

We will clarify the distinction between conversion and post-training and report their respective costs separately in the revised manuscript.

Reference

[1] Cheng, Shuang, et al. "Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear Reviewer 8dCQ,

Thank you again for your valuable comments and constructive feedback. As the discussion phase is nearing its end, we would like to kindly follow up and ask whether our responses have addressed your concerns, at least in part.

If there are any remaining questions or points that need clarification, we would be happy to discuss them further. We sincerely hope that the additional explanations and evidence we provided can be considered in your final assessment.

Thank you again for your time.

Best regards,

My concerns have been fully addressed, and I would like to keep my score at a positive 3.

Reviewer_GMuXOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper proposes On-Policy Residual Learning (ORL) for diffusion language models. ORL collects fresh model-generated trajectories, retains verified responses, and trains intermediate states using their captured residual inputs. It combines final-output supervision with an adjacent entropy-contraction objective (AEC) to encourage faster refinement. The paper also introduces system optimizations for state capture, transport, packed training, and attention execution. Experiments on three model families report improvements in parallel decoding efficiency and measured speed–quality tradeoffs.

Critical strengths
  1. Addresses the important problem of learning efficient cross-step refinement in residual-conditioned DLMs using on-policy learning.
  2. Combines training objectives with practical systems optimizations, supported by detailed system ablations
Critical weaknesses
  1. The central claim about on-policy learning is overstated. The paper states, “We identify residual refinement as an inherently on-policy learning problem.” However, the experiments appear to show only that on-policy data collection improves post-training performance under the evaluated setup. This is not equivalent to establishing that residual refinement is inherently on-policy.

  2. The writing and experimental presentation need substantial improvement. This paper is difficult for humans to read. Important experiments are insufficiently explained. For example, for Figures 1 and 2, neither the main text nor the captions clearly describes how the experiments were conducted, making the results difficult to interpret. Some material in the related-work section would be more appropriately presented as analysis of the proposed method. The introduction is also overly wordy, as are several figure captions, yet essential experimental details remain missing. A clearer structure and more precise descriptions would substantially improve readability. These problem leading it difficult to read this paper.

  3. Insufficient ablations. AEC can also be applied to non-residual DLMs, but the experiments do not establish whether it specifically improves residual utilization or merely provides general confidence shaping. A full ablation with and without both AEC and the residual pathway is needed. Similarly, removing entropy-gated RKL conflates low-entropy CE suppression with KL regularization. A baseline applying CE only at high-entropy positions and zero loss elsewhere would isolate RKL’s contribution.

  4. The policy-gradient diagnostic does not substantiate the claimed mechanism. Appendix B.2 interprets Figure 8’s policy-gradient covariance as explaining entropy reduction during ORL training. However, ORL uses verified CE/AEC with AdamW, and Appendix I.3 explicitly acknowledges that the policy-gradient specialization need not apply to these updates. Moreover, rollout-average entropy changes also reflect state-distribution shifts. This diagnostic therefore does not establish that ORL improves residual refinement rather than merely sharpening prediction

Questions for authors

NAN

6 comments
AuthorsOpenReview ↗

Response to Reviewer GMuX (Part 1/4)

We thank you for the feedback. We addressed the questions below.

W1. Claim on On-Policy Learning

We want to kindly point out that our key contribution is on-policy residual learning, rather than simply on-policy data collection.

(1) On-policy learning framework. At every optimization step, ORL generates fresh residual-conditioned trajectories using the current policy, trains on the exact intermediate states encountered during rollout, and updates the model before generating the next batch. This follows the standard on-policy learning paradigm adopted in modern LLM reinforcement learning, such as DeepSeek-R1 [1], while explicitly optimizing residual refinement.

(2) Theoretical motivation. Residual predictions are fed back into subsequent denoising steps, directly influencing future states. As the model evolves, both its residual predictions and the induced state distribution change, making offline or stale residual supervision subject to training–inference distribution mismatch.

(3) Empirical evidence. Figure 2(a) controls rollout-policy staleness (N = 1, 25, 75) while keeping other training settings identical. On-policy learning achieves 89.8% accuracy / 13.7 TPF, compared with 77.1% / 11.7 TPF (N = 25) and 78.1% / 9.9 TPF (N = 75).

Together, these results highlight that on-policy learning is a fundamental design principle of ORL, supported by both the residual feedback mechanism and controlled empirical evidence, rather than merely a strategy for collecting fresh training data.

Reference

[1] Guo, D., et al. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." Nature, 645, 633–638 (2025). https://arxiv.org/abs/2501.12948

AuthorsOpenReview ↗

Response to Reviewer GMuX (Part 2/4)

W2. Writing and Experimental Presentation

We would like to clarify that the experimental setups are already documented in the manuscript and appendices, with explicit references from the main text. Furthermore, our anonymous repository provides complete training and evaluation configurations (configs/paper/experiments.yaml and docs/reproduction/SETTINGS.md), as described in the Reproducibility Statement.

Specifically:

  • Figure 1(b): The introduction and caption specify the evaluation benchmarks (GSM8K/HumanEval) and single-H100 setting. Section 4.1 further describes throughput measurements and verification thresholds, with detailed configurations and comparisons in Appendix G.
  • Figure 2(a): Section 4.1 describes the controlled staleness experiment (N = 1, 25, 75), using identical initialization, verifier, and optimization objectives. Appendix H.4 provides the full evaluation protocol and per-seed results across five independent training seeds.
  • Figure 2(b): We compare QwenDLM trained on GSM8K without AEC (baseline) against full ORL (ours), keeping all other settings identical. The normalized entropy of unresolved positions is measured across denoising steps, showing that AEC reduces the required steps from 41 to 22.
  • Figure 2(c): We compare the training-step time and memory overhead of a naive implementation against our optimized training system on QwenDLM/GSM8K. Appendix B.1 analyzes exact-state capture overhead, while Appendix H.1 provides detailed system measurements.

These experimental details are already included, and we will work to make them more accessible. We have made the experimental setups more explicit in the figure captions. We will further streamline the introduction, and reorganize the related-work and motivation sections to improve readability.

AuthorsOpenReview ↗

Response to Reviewer GMuX (Part 3/4)

W3. Additional Ablations

Thank you for the suggestions. We would like to clarify that Table 3(a) already includes ablations independently removing AEC and the residual pathway, demonstrating the importance of both components. To further address the reviewer's concerns, we conducted two additional ablation studies.

(1) Interaction between AEC and residual conditioning.

As mentioned above, our original ablations already demonstrate that removing either AEC or residual conditioning degrades the speed–quality trade-off. We further extend these experiments to a complete 2×2 ablation to clarify their interaction.

ResidualAECGSM8K Acc. / TPFMATH-500 Acc. / TPF
YesYes94.5 / 6.2590.0 / 5.64
YesNo92.6 / 4.4892.6 / 4.38
NoNo92.0 / 3.6875.0 / 3.56
NoYes81.4 / 5.2563.0 / 4.80

Without residual conditioning, AEC excessively sharpens predictions without sufficient information to support accurate refinement. Although this increases TPF (3.68 → 5.25 on GSM8K), it substantially reduces accuracy (92.0% → 81.4%). In contrast, residual conditioning allows the model to build upon previous predictive distributions, enabling AEC to accelerate refinement without the same severe quality degradation.

These results demonstrate that AEC and residual conditioning are complementary: AEC encourages faster refinement, while residual conditioning provides the information needed to make this refinement reliable. Their combination achieves a substantially better speed–quality trade-off than either component alone.

(2) Separating entropy gating from reverse KL.

We agree that the original ablation removes both entropy gating and reverse KL, making it difficult to isolate their individual contributions. We therefore conduct an additional ablation retaining entropy-gated CE while assigning zero loss to low-entropy positions, without RKL.

ConfigurationGSM8K Acc. / TPFMATH-500 Acc. / TPF
Full ORL (entropy-gated CE + RKL)94.5 / 6.2590.0 / 5.64
Entropy-gated CE without RKL91.1 / 4.5178.8 / 3.91
Ungated CE (without entropy-gated RKL)87.8 / 3.2976.8 / 3.46

Entropy gating alone improves performance over ungated CE, while RKL provides further substantial gains, particularly on MATH-500 (+11.2 accuracy points and +1.73 TPF). This confirms that RKL contributes beyond simply suppressing CE on low-entropy predictions, helping stabilize the residual learning process.

Together, these additional ablations clarify the complementary roles of AEC, residual conditioning, entropy gating, and RKL. We have incorporated these results into the revised manuscript.

Reviewer_cshEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript introduces ORL, an on-policy residual learning framework for diffusion language models (DLMs). The residual in question is the probability distribution over the output tokens at step kk: a weighted sum over that distribution of the input token embeddings is processed by a lightweight residual adapter before being added to the actual discrete input token embedding.

The authors argue that residual refinement is an inherently on-policy learning problem: "residuals must be optimized on the evolving state distribution they induce, with the residual pathway updated slowly enough that the loop stays stable." They then present a general recipe aiming to do that, then design rollout/training systems making this practical using surprisingly modest post-training resources (e.g., 60 H100-hours).

Benchmark results and inference throughput are favorable to ORL across three model families.

Critical strengths

This is a recipe for efficiently training residual context diffusion models, with rollout/training system questions thought out. Assuming that the code will be release, this contribution has the potential to be influential. The results promised with modest post-training resources (60 H100-hours) makes me want to try it myself.

Critical weaknesses

While the code is available to reviewers (I haven't tested it), it is not clear from the submission whether the code/models will be open sourced, and if yes under what license. This is the topic of Q1 below.

The Ethics Statement makes no mention of special monitorability ("neuralese") concerns applying to residual context diffusion models, nor does the manuscript check if the "reassuring" results from "How Transparent is DiffusionGemma?" https://arxiv.org/pdf/2606.20560 also apply to ORL. This is the topic of Q2 below.

A positive outcome on Q1 and Q2 would likely have me raise my score to 4. Q3 is optional, just my curiosity.

Questions for authors

Q1. Will this be open-sourced and, if yes, under what license?

My critical strength reported above is made less relevant by the code not being made available to the community.

Q2. Does truncating ri,v(k)r_{i,v}^{(k)} to keeping only the top tokens cause a large drop in performances?

From page 9 of the DiffusionGemma:

Remark on interpretability. Recent interpretability analyses demonstrate that while the self-conditioning signal ztz_t introduces a continuous latent space injection into the sequence, these intermediate self-conditioning vectors map robustly to an interpretable token bottleneck, preserving the model’s algorithmic transparency (Asaria et al., 2026; Engels et al., 2026).

This last "Engels et al, 2026" is "How Transparent is DiffusionGemma?" https://arxiv.org/pdf/2606.20560 : I encourage the authors to browse it, with a special focus on the eval-only test consisting of limiting the residual to only code the top tokens. If, like DiffusionGemma, ORL's performances do not degrade much, then you may report the same reassurance for cheap. If not, this is actually more interesting, and may warrant some additional warnings to the community. Both outcomes would satisfy me, as long as they're duly reported.

Note that, as discussed in the "Performance degradation from top-k truncation largely is a sampler artifact" section from https://www.lesswrong.com/posts/QBuJ3suRZxrrxSTtv/does-diffusiongemma-do-latent-reasoning , more diffusion steps should be used to perform this test.

As for the Ethics Statement, I make no specific request. However, if the authors wish to edit it now that they've been made aware of these concerns, they have my support in doing so.

Q3. Can something be done with rejected rollouts?

Rejected rollouts contribute no gradient in ORL. Do you see residual+diffusion as causing fundamental issus why it is hard to leverage them? If you have thought about those issues, it could be good to give some words on them.

6 comments
AuthorsOpenReview ↗

Response to Reviewer cshE (Part 1/3)

We thank you for the feedback. We addressed the questions below.

Q1: Will this be open-sourced and, if yes, under what license?

We fully agree that open-sourcing our implementation is essential for reproducibility and broader adoption.

We confirm that we will publicly release our reproducible training code, evaluation code, and trained model checkpoints under the MIT license. Beyond the initial release, we are committed to actively maintaining and updating the repository, addressing community feedback, and supporting researchers in reproducing our experiments and building upon our work.

As stated in our Reproducibility Statement, we have already provided an anonymous repository containing our implementation, training configurations, evaluation settings, and detailed reproduction instructions.

AuthorsOpenReview ↗

Response to Reviewer cshE (Part 2/3)

Q2. Does restricting the residual to its top tokens cause a large performance drop?

Thank you for pointing us to Engels et al. (2026). We expect ORL to preserve useful residual information within a compact token-level representation, since its residuals are derived from token probability distributions and trained through verified supervision and progressive refinement. Indeed, our new experiments support this hypothesis: retaining only the top-4 tokens causes no statistically significant accuracy degradation, and even top-2 truncation preserves performance under conservative sampling. Moreover, ORL does not systematically increase truncation sensitivity compared with the pre-ORL models. We provide the detailed results below.

(1) Results with the original decoding settings.

We truncate the residual distribution by preserving its top-k token probabilities and uniformly redistributing the remaining probability mass. We also evaluate a probability-threshold variant (p ≥ 0.03), which retains all tokens with probability at least 0.03 rather than a fixed number of tokens.

The table reports accuracy changes in percentage points (GSM8K / MATH-500 / HumanEval) relative to the untruncated model.

TruncationQwenDLM-ORLDiffusionGemma-ORL
Top-8−0.8 / −1.4 / −2.4−0.5 / +1.6 / 0.0
Top-4−1.0 / −0.4 / −4.9−0.2 / −2.2 / +1.2
p ≥ 0.03−1.4 / +0.2 / −5.5+0.2 / −1.0 / −1.8
Top-2−1.3 / −5.4 / −3.7−0.8 / −3.4 / −0.2
Top-1−1.6 / −18.4 / −11.0−4.3 / −14.7 / −16.3

Untruncated accuracy: 92.4 / 92.2 / 89.6 for QwenDLM-ORL and 95.8 / 91.6 / 92.1 for DiffusionGemma-ORL. Bold indicates statistically significant differences based on paired 95% bootstrap confidence intervals.

Top-4 truncation causes no statistically significant degradation for either model, while top-1 truncation can substantially reduce accuracy.

(2) Results with conservative sampling.

Following the LessWrong analysis, some performance degradation under residual truncation may arise from overly aggressive decoding rather than information loss. To distinguish these effects, we repeat the experiments using more conservative sampling configurations:

  • DiffusionGemma: We increase the maximum denoising steps from 48 to 96, adjust the temperature from 1.0 to 0.5, and set the entropy bound to 0.02.
  • QwenDLM: We reduce the entropy budget from 0.4 to 0.1 (0.6 to 0.15 for HumanEval), and further to 0.02 for MATH-500, allowing more refinement before accepting tokens.

The following table reports accuracy changes relative to the corresponding untruncated models under the same conservative sampler.

TruncationQwenDLM-ORLDiffusionGemma-ORL
Top-2MATH-500: −1.6; HumanEval: −0.6MATH-500: −1.0; HumanEval: +0.2
Top-1MATH-500: −7.8; HumanEval: +4.3MATH-500: −3.3; HumanEval: −1.6

Most of the top-1 degradation disappears under conservative sampling, and top-2 truncation causes no statistically significant accuracy loss. However, top-1 truncation still reduces MATH-500 accuracy by approximately 8 and 3 points, respectively, indicating that information beyond the most likely token remains useful for challenging mathematical reasoning.

(3) Does ORL increase reliance on residual information?

We additionally compare the sensitivity to top-1 truncation before and after ORL training, under conservative sampling.

ModelGSM8KMATH-500HumanEval
QwenDLM (base)−32.9−72.8−76.8
QwenDLM + ORL−0.7−7.8+4.3
DiffusionGemma (base)−0.5−1.1−3.9
DiffusionGemma + ORL−1.2−3.3−1.6

For QwenDLM, ORL substantially reduces sensitivity to top-1 truncation. For DiffusionGemma, ORL shows largely comparable sensitivity to the base model, with no substantial additional degradation.

We have included these additional experiments in the revised manuscript and update the Ethics Statement to explicitly acknowledge the transparency concerns of continuous residual representations, including the remaining top-1 sensitivity on challenging mathematical reasoning tasks. While the observed top-k robustness is encouraging, we recognize that it does not by itself establish full monitorability.

AuthorsOpenReview ↗

Response to Reviewer cshE (Part 3/3)

Q3. Can rejected rollouts be utilized?

We agree that rejected rollouts can also provide useful learning signals, and there is no fundamental limitation preventing their use in ORL.

We add new experiments incorporating negative rollouts, using our newly trained Qwen3.5-DLM-9B on LiveCodeBench. The results are summarized below:

Training MethodAccuracy (%) ↑TPF ↑
SFT baseline48.43.27
ORL (positive only)58.94.77
ORL (positive + negative)61.03.75

Incorporating negative rollouts further improves accuracy (+2.1 points), while reducing TPF (4.77 → 3.75) compared with positive-only ORL. Nevertheless, both accuracy and TPF remain higher than the SFT baseline.

These preliminary results demonstrate that negative rollouts can be effectively incorporated into ORL, thus showing ORL’s generalization, but introduce an additional accuracy–decoding efficiency trade-off. We believe understanding and optimizing this trade-off is an interesting, complementary direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.