acceptodds
Under review as a conference paper at ICLR 2027

Loop-Distill: Refining Information Flow in Diffusion Large Language Models

Abstract

Diffusion language models (DLMs) reduce decoding latency by denoising multiple token positions in parallel. A practical way to obtain this is to convert a pretrained autoregressive (AR) large language model into a DLM with the block diffusion paradigm. However, this conversion also changes the role of a forward pass. Instead of predicting one next token from a fully observed prefix, the model must form and refine representations for multiple masked positions under partial context. We show that this mismatch gives rise to an information flow bottleneck: masked token states stabilize prematurely in the shallow layers, yet the predictions made by these layers are often overconfident and rarely consistent with the final output. To address this, we propose _Loop-Distill_, which uses local shallow-layer loop refinement to let masked states absorb known context more fully and distills this deeper shallow refinement into the original single-pass model. During training, we exploit DLM decoding as an intrinsic refinement loop by constructing an online stop-gradient teacher from the student’s current weights and looping selected shallow layers to refine masked states. This compresses useful loop computation into the deployed model without changing its inference-time architecture. Across 13 benchmarks, Loop-Distill improves average accuracy on GLM-4.7-Flash and Qwen-30B-A3B-2507 by 2.05 and 2.21 points, respectively. It also improves decoding throughput by 11.0% on average by enabling more effective masked state refinement and thus fewer full-model denoising passes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_d7ApOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Loop-Distill, a training-only self-distillation framework for improving masked-token representations in autoregressive-to-block-diffusion language models. The authors identify a shallow-layer information-flow bottleneck characterized by abrupt initial representation updates, premature stabilization, and unreliable intermediate predictions. Loop-Distill constructs an online stop-gradient teacher that repeatedly applies selected shallow layers and transfers the resulting representations into a single-pass student through loss-gated, masked-token trajectory distillation. Experiments on two 30B-A3B MoE backbones across 13 benchmarks and two block sizes report average accuracy improvements of 2.05–2.21 percentage points and approximately 11% higher decoding throughput.

Critical strengths
  • Clear motivation. This paper identifies a shallow-layer information bottleneck that happens during AR-to-DLM conversion.
  • Practical training-only design. This paper combines an online shared-parameter teacher, masked-position cosine distillation, and a task-loss gate that is compatible with the original deployment architecture.
  • **Consistent performance gain. ** The evaluations demonstrate positive average improvements across both backbones and block sizes, including gains on coding and reasoning tasks.
Critical weaknesses
  • While the authors identify premature stabilization of shallow masked-token representations, it remains unclear whether this reflects insufficient contextual information acquisition rather than benign representational convergence. Although Table 7 and Appendix B show changes in hidden-state dynamics, attention allocation, and expert routing after Loop-Distill, these observations do not directly establish improved acquisition of task-relevant contextual information or demonstrate that alleviating the proposed bottleneck is responsible for the performance gains.
  • The methodological novelty relative to LoopMDM [1] and ELT [2] is not sufficiently distinguished, particularly regarding the underlying principles of selective shallow-layer looping and distilling additional recurrent computation into a shorter computation path.

[1] S. Lee et al., “Looped Diffusion Language Models”.

[2] S. Goyal et al., “ELT: Elastic Looped Transformers for Visual Generation”.

Questions for authors
  • Could the authors clarify how the proposed shallow-layer local refinement differs conceptually from selective layer looping in LoopMDM, and how the online trajectory distillation mechanism differs from intra-loop self-distillation in ELT?
  • Does the looped teacher improve masked reconstruction because it incorporates additional contextual information, or could a non-looped teacher using ordinary later-layer representations provide similar supervision?
  • Could the authors provide a more detailed analysis of the heterogeneous performance gains across different backbones, block sizes, and benchmarks, including the negative results reported in Table 1?
  • The proposed information-flow bottleneck is primarily analyzed under conventional block-diffusion decoding, where committed tokens are not subsequently revised. Recent diffusion language models, such as LLaDA2.1 [1] and RemeDi [2], support token editing or remasking to correct earlier predictions. Does the same shallow-layer bottleneck persist under these more flexible decoding mechanisms? If so, is Loop-Distill complementary to such inference-time refinement strategies?

[1] T. Bie et al., “LLaDA2.1: Speeding Up Text Diffusion via Token Editing”.

[2] Z. Huang et al., “Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models”.

Reviewer_7vvgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new method, LoopDistill, for AR-to-(Block) Diffusion conversion. Its inspiration roots from the observation that shallow layers have a hard time refining masked tokens. The authors propose to reuse part of the shallow layers to build an online teacher, and then distill the "hidden-state refinement" knowledge into a single-pass student model. The authors validate its idea with two models on multiple benchmarks, and show that with LoopDistill, the student model shows improvements.

Critical strengths

Motivation

  • The motivation mainly comes from Figure 1, where demonstrates information flow bottleneck. Shallow layers show (especially the first layer) stronger update than later layers. This observation inspires authors to reuse this strong signal and distill it into a student model for more efficient decoding.

Method

  • The core technical part of the method is simple and straight-forward.
Critical weaknesses

Motivation

  • The premature stabilization is examined by ∣∣hl+1−hl∣∣||h^{l+1} - h^{l} ||, and then authors believe that shallow layers do not fully get the context. However, it is unclear why a small update ∣∣hl+1−hl∣∣||h^{l+1} - h^{l} || means that the information flow is bottlenecked. Perhaps more evidence is needed for this validation.

Results

  • Although LoopDistill improves the performance, it also brings 10% wall-clock overhead during the training (Appendix E). It might be fair if a comparison between matched-compute training with vanilla SFT and LoopDistill is included.
Questions for authors
  • In practice, how does δ=0.15\delta = 0.15 is decided. Does that also involve extra budgets for deciding a decent δ=0.15\delta = 0.15?
  • Following δ\delta, in Table 4, when δ=0.15\delta=0.15, both 1-layer and 2-layer configs satisfy it. From my understanding, we should choose 1-layer config instead the 2-layer, but the paper uses 2 layers as the default. How does that come from?
Reviewer_JQugOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates shallow-layer representation dynamics when autoregressive language models are converted to block diffusion. It identifies a large initial update followed by premature stabilization of masked states, and proposes Loop-Distill: an online, stop-gradient teacher repeats selected shallow layers, while a single-pass student receives masked-state distillation only when the teacher improves the masked reconstruction loss. The deployed architecture is unchanged. Experiments on two 30B-A3B MoE backbones, two block sizes, and 13 benchmarks report average accuracy improvements of 2.05 and 2.21 percentage points for GLM and Qwen, respectively. Throughput measurements over nine benchmarks yield an average relative improvement of 11.0% across four model settings. Component ablations and representation, attention, and routing analyses provide additional validation.

Critical strengths

The method concentrates extra computation in the training-time teacher and retains the original inference architecture. Improvements across both backbones and block sizes therefore offer a plausible way to improve quality and decoding efficiency without introducing an inference-time loop. This is the strongest practical argument for acceptance (pp. 4–9).

Critical weaknesses
  1. The experiments establish improvements over a vanilla block-diffusion conversion, but do not report the performances of corresponding ar models. Consequently, it is unclear whether Loop-Distill partially recovers conversion-induced degradation or achieves a favorable quality–efficiency trade-off relative to the original models. Reporting the source AR baselines would clarify the practical significance of the gains. An additional AR model fine-tuned on the same data would help separate the effects of continued training from those of diffusion conversion.
  2. Table 1 reports accuracy across 13 benchmarks, two backbones, and two block sizes, whereas Table 5 reports TPF on only four benchmarks under the default GLM-4.7-Flash, block-size-32 setting. What motivated this subset selection? Please provide corresponding Vanilla and Loop-Distill TPF results for all settings in Table 1, using the same decoding configurations as the accuracy evaluation, either alongside Table 1 or in a companion appendix table. This would clarify whether the accuracy improvements consistently coincide with more efficient decoding and whether the proposed refinement mechanism generalizes beyond the four analyzed tasks.
Questions for authors

Please refer to "critical weakness."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.