- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the conversion of pretrained autoregressive models into masked diffusion language models. It proposes Repr-Align, which trains the diffusion student with the masked denoising loss plus a layer-wise cosine loss that aligns the student’s hidden states on masked inputs to those of the frozen AR teacher on clean inputs. Experiments on Qwen3 models (0.6B to 4B) show better HumanEval pass@10 than training without alignment, along with results on freezing parameter blocks and training on a small data subset.
Critical strengths
- The method is simple and easy to adopt. It adds no adapters or architectural changes, and the only extra cost is a forward pass of the frozen teacher.
- Under the authors’ setup, alignment gives consistent improvements over the unaligned baseline at both 0.6B and 1.7B, with clearly faster progress early in training.
Critical weaknesses
-
The source of the gains is not separated from distillation. The teacher sees the clean sequence while the student sees the masked one, so at masked positions the alignment target contains information about tokens the student cannot see. This is close to feature-level distillation from the AR teacher, not only preservation of representation geometry. The only baseline is the authors’ own denoising recipe without alignment. Could the authors compare with logit-level distillation from the AR teacher at masked positions [1] (or maybe with existing AR-to-DLM conversion methods [2][3], but at least [1]) under the same data and steps? This would show whether aligning hidden states adds value beyond standard distillation.
-
The evaluation is narrow relative to the claims. All experiments use code benchmarks with 128 generated tokens, while the paper concludes that linguistic representations are universal across generation orders and that alignment could become a standard technique for training DLMs. Could the authors add evaluations on general language, math, or reasoning tasks?
[1] https://arxiv.org/abs/1503.02531
[2] https://arxiv.org/abs/2410.17891
[3] https://arxiv.org/abs/2512.14067
Questions for authors
- How does Repr-Align compare with logit-level distillation from the AR teacher under the same data and training steps?
- Does the improvement hold on non-code tasks?