- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ORL, a method to train diffusion language models on the intermediate states they produce during generation. It helps the model make decisions in fewer steps and includes system changes that make the training practical. Experiments on three model families show faster decoding with generally similar accuracy.
Critical strengths
- ORL trains on fresh, verified trajectories from the current model, so it learns from the states it actually encounters during generation.
- The authors release their training and evaluation code. This makes the results easier to reproduce.
Critical weaknesses
- Quality drops on some additional tasks, even though decoding gets faster.
- The reported post-training cost leaves out the earlier AR-to-DLM conversion for QwenDLM.
Questions for authors
How much compute did the AR-to-DLM conversion and offline initialization require before ORL training?
4 comments
Response to Reviewer 8dCQ (Part 2/2)
W2 & Q1. Training cost for converting Qwen AR model to QwenDLM
Thank you for raising this question. AR-to-DLM conversion is an increasingly common approach to developing diffusion language models. ORL is a general framework that can be applied either to AR-converted DLMs or directly to existing pretrained DLMs, without requiring a conversion stage. We have made the distinction between AR-to-DLM conversion and ORL post-training, including their respective training costs, more explicit in our updated manuscript.
Importantly, AR-to-DLM conversion is not a prerequisite for ORL. As demonstrated in Table 1, we apply ORL directly to existing DLMs, including LLaDA-2.1-mini and DiffusionGemma, achieving consistent improvements without any conversion overhead. This demonstrates the generalizability of ORL across different model families and initialization strategies.
For QwenDLM, we initialize from Qwen3-4B through an SDAR-style [1] offline SFT conversion stage, followed by ORL post-training. The AR-to-DLM conversion uses approximately 10 billion training tokens, a relatively modest data budget compared with prior SDAR-series conversion pipelines (50B tokens in SDAR). The subsequent ORL post-training requires only 60 H100 GPU-hours.
We will clarify the distinction between conversion and post-training and report their respective costs separately in the revised manuscript.
Reference
[1] Cheng, Shuang, et al. "Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation." Findings of the Association for Computational Linguistics: ACL 2026. 2026.
A Gentle Follow-up on Our Responses
Dear Reviewer 8dCQ,
Thank you again for your valuable comments and constructive feedback. As the discussion phase is nearing its end, we would like to kindly follow up and ask whether our responses have addressed your concerns, at least in part.
If there are any remaining questions or points that need clarification, we would be happy to discuss them further. We sincerely hope that the additional explanations and evidence we provided can be considered in your final assessment.
Thank you again for your time.
Best regards,
My concerns have been fully addressed, and I would like to keep my score at a positive 3.
Response to Reviewer 8dCQ (Part 1/2)
We thank you for the feedback. We addressed the questions below.
W1. Quality drop on some additional benchmarks
We would like to clarify the following two points.
(1) The observed accuracy drops are within evaluation variance.
We additionally analyze the standard deviations of the benchmark results. As shown below, all observed accuracy drops are within one standard deviation, while the TPF improvements are consistently substantial.
These results suggest that the small accuracy fluctuations are not statistically conclusive, whereas the decoding efficiency improvements are much larger relative to their measured variability.
(2) ORL achieves higher decoding efficiency at matched or better accuracy.
Importantly, the original results in Table 2 use the aligned decoding configurations for each model. Since ORL improves refinement efficiency, we can adopt stricter decoding criteria to recover generation quality while maintaining a substantial speed advantage.
We conducted additional evaluations with adjusted decoding settings to match or exceed the baseline accuracy:
These results demonstrate that ORL consistently maintains or improves generation quality while achieving 1.10–1.26× higher TPF. The small accuracy differences in Table 2 therefore do not indicate an inherent quality degradation, but reflect particular operating points on the speed–quality trade-off.
We will include the variance analysis and these additional accuracy-aligned evaluations in the revised manuscript to provide a more comprehensive characterization of ORL's speed–quality trade-off.