- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Loop-Distill, a training-only self-distillation framework for improving masked-token representations in autoregressive-to-block-diffusion language models. The authors identify a shallow-layer information-flow bottleneck characterized by abrupt initial representation updates, premature stabilization, and unreliable intermediate predictions. Loop-Distill constructs an online stop-gradient teacher that repeatedly applies selected shallow layers and transfers the resulting representations into a single-pass student through loss-gated, masked-token trajectory distillation. Experiments on two 30B-A3B MoE backbones across 13 benchmarks and two block sizes report average accuracy improvements of 2.05–2.21 percentage points and approximately 11% higher decoding throughput.
Critical strengths
- Clear motivation. This paper identifies a shallow-layer information bottleneck that happens during AR-to-DLM conversion.
- Practical training-only design. This paper combines an online shared-parameter teacher, masked-position cosine distillation, and a task-loss gate that is compatible with the original deployment architecture.
- **Consistent performance gain. ** The evaluations demonstrate positive average improvements across both backbones and block sizes, including gains on coding and reasoning tasks.
Critical weaknesses
- While the authors identify premature stabilization of shallow masked-token representations, it remains unclear whether this reflects insufficient contextual information acquisition rather than benign representational convergence. Although Table 7 and Appendix B show changes in hidden-state dynamics, attention allocation, and expert routing after Loop-Distill, these observations do not directly establish improved acquisition of task-relevant contextual information or demonstrate that alleviating the proposed bottleneck is responsible for the performance gains.
- The methodological novelty relative to LoopMDM [1] and ELT [2] is not sufficiently distinguished, particularly regarding the underlying principles of selective shallow-layer looping and distilling additional recurrent computation into a shorter computation path.
[1] S. Lee et al., “Looped Diffusion Language Models”.
[2] S. Goyal et al., “ELT: Elastic Looped Transformers for Visual Generation”.
Questions for authors
- Could the authors clarify how the proposed shallow-layer local refinement differs conceptually from selective layer looping in LoopMDM, and how the online trajectory distillation mechanism differs from intra-loop self-distillation in ELT?
- Does the looped teacher improve masked reconstruction because it incorporates additional contextual information, or could a non-looped teacher using ordinary later-layer representations provide similar supervision?
- Could the authors provide a more detailed analysis of the heterogeneous performance gains across different backbones, block sizes, and benchmarks, including the negative results reported in Table 1?
- The proposed information-flow bottleneck is primarily analyzed under conventional block-diffusion decoding, where committed tokens are not subsequently revised. Recent diffusion language models, such as LLaDA2.1 [1] and RemeDi [2], support token editing or remasking to correct earlier predictions. Does the same shallow-layer bottleneck persist under these more flexible decoding mechanisms? If so, is Loop-Distill complementary to such inference-time refinement strategies?
[1] T. Bie et al., “LLaDA2.1: Speeding Up Text Diffusion via Token Editing”.
[2] Z. Huang et al., “Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models”.