- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the low-budget conversion of pretrained autoregressive (AR) language models into diffusion language models (dLLMs), focusing on the trade-off between generation quality and knowledge retention. Under matched training budgets, the authors compare in-place conversion with a frozen context-tower architecture across both MoE and dense backbones. Experiments show that in-place conversion substantially degrades long-form generation, whereas the frozen-tower configuration preserves considerably more of the parent model’s performance.
Critical strengths
The paper addresses an important and practical problem of converting pretrained AR models into diffusion language models under limited training budgets, offering valuable insights into efficient diffusion language modeling.
Critical weaknesses
- The frozen context-tower architecture is largely adopted from Nemotron-Labs-TwoTower (Reda et al., 2026). The main contribution lies in empirical comparison and analysis rather than a new conversion method.
- Although the main experiments match the training budget and the trainable parameter count, the two configurations differ in several aspects, including architecture, alignment loss, and diffusion corruption procedure. Therefore, it is unclear whether the performance improvements can be attributed solely to freezing the context representations.
- The practical inference efficiency of the proposed method appears to be rather limited. Despite supporting parallel decoding, the model achieves only 0.134–0.168× the throughput of the autoregressive baseline at batch size 1.
Questions for authors
- Could the authors provide a more controlled ablation that changes only the context architecture while keeping the alignment loss, corruption procedure, and block-size schedule fixed? This would help identify the primary source of the performance gap.
- Given that the frozen and trainable context towers achieve similar HumanEval scores, is the main generation improvement attributable to separating context computation from denoising rather than freezing the context representations?
- Since the core two-tower architecture follows Reda et al. (2026), could the authors clarify the methodological contributions beyond applying the existing architecture to a lower-budget setting and conducting additional empirical analyses?
- The frozen-tower model is substantially slower than the AR baseline despite its parallel decoding capability. What architectural or decoding changes could make the approach competitive in end-to-end inference throughput?