acceptodds
Under review as a conference paper at ICLR 2027

ALoDLM: Adaptively Looped Diffusion Language Models

Abstract

Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, yet their practical adoption remains hindered by a persistent quality gap relative to comparably sized autoregressive models. We attribute this gap to a computation–difficulty mismatch: within a partially observed sequence, some unknown tokens are readily predictable, while others require substantially more computation to resolve. Existing DLMs, however, apply uniform computational depth to every unknown position at each denoising step. We introduce ALoDLM, which replaces this uniform computation with token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines representations in latent space, allocating computation based on token difficulty. Tokens ready to commit are fed back as discrete context, while unresolved tokens retain and refine their latent states through additional recurrent passes. To learn both token prediction and computation allocation end-to-end, we formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding autoregressive baselines in average benchmark score at both scales. Importantly, ALoDLM combines superior generation quality with fast parallel decoding, establishing a strong quality–efficiency trade-off among all evaluated autoregressive and diffusion models under optimized inference engines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Model, Looped Transformer, Parallel Decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bt7iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces ALoDLM, a dLLM with token-adaptive latent recurrence. Committed tokens provide context, while unresolved tokens continue to refine their latent states. The authors derive a training objective for token prediction and the halting policy, and evaluate Qwen3-based models at 1.7B and 8B scales on eleven benchmarks.

Critical strengths
  1. The evaluation includes competitive models such as SDAR and WeDLM. ALoDLM-8B achieves an average score of 80.3 across eleven benchmarks, compared with 75.1 for WeDLM-8B in Table 1. This supports the usefulness of the approach beyond early dLLM models.

  2. Figure 3 reports an 8.5% throughput improvement over WeDLM at the same GSM8K accuracy of 93.25%. This is measured evidence of an inference benefit, rather than a claim based only on fewer operations.

Critical weaknesses
  1. The experiments do not establish how much the proposed adaptive computation contributes. Figure 5 studies maximum recurrent depth and loop placement, but does not compare learned halting with fixed-depth recurrence at matched compute. A non-recurrent model trained with the same initialization, data, and training compute is also missing. These comparisons are needed to separate the method’s benefit from the training recipe.

  2. The detailed speed comparison is limited to GSM8K with single-stream inference on one B200. ALoDLM improves throughput in the higher-accuracy range, while WeDLM leads at the fastest settings shown. Without comparable results on other tasks, the evidence for a broader inference advantage remains limited.

Questions for authors
  1. With the same initialization, data, and training compute, how does ALoDLM compare with non-recurrent and fixed-depth models? Does learned halting improve accuracy at matched inference compute?

  2. Appendix D.1 reports eight training epochs for the 8B model. What are the total training FLOPs or GPU-hours? The 5B-token corpus size alone does not establish the training cost.

  3. Does the throughput improvement over WeDLM also hold on a code-generation benchmark at matched accuracy, using the same hardware and the native inference engines?

Reviewer_H744OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the mismatch between token difficulty and computational resource allocation in diffusion language models. Tokens whose predictive entropy falls below a threshold are committed early and fed back as context for the remaining tokens, while unresolved tokens retain and further refine their latent states. A variational objective jointly optimizes token prediction and token-wise computation allocation.

Critical strengths
  1. The formulas and pseudocode are well-presented.
  2. The main results are described in good detail.
Critical weaknesses

ALoDLM builds upon several established techniques: recursive refinement within diffusion steps (R-MDM) [1], token-wise adaptive recurrent depth (MoR) [2], and learned halting mechanisms with depth-dependent training objectives (PonderNet [3] and Ouro [4]). While the proposed joint exit schedule accounts for interactions among tokens, its effectiveness is not sufficiently validated. Consequently, it remains unclear how much of the reported improvements can be attributed to the proposed joint optimization rather than the integration of existing techniques.  

[1] Recursive Scaling in Masked Diffusion Models

[2] Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation.

[3] PonderNet: Learning to Ponder

[4] Scaling Latent Reasoning via Looped Language Models

Questions for authors
  1. Could the authors provide an ablation comparing the proposed sequence-level loss for training the ExitGate with a simpler token-level loss, where each token's halting decision is optimized using only its own prediction loss?
  2. The method combines entropy-based token commitment with a learned exit gate for inner-loop termination. Could the authors provide ablations comparing this joint design with simpler alternatives, such as (1) using the learned exit policy alone to determine token commitment, with a fixed maximum recurrent depth, and (2) using entropy-based commitment alone with a fixed maximum recurrent depth?
1 comment

We sincerely appreciate your time, constructive feedback, and insightful questions. We address your concerns below:

[W1 & Q1: Relation to existing work and effectiveness of sequence-level loss]

MoR, PonderNet, and Ouro are in the autoregressive setting, each decoding step generates a single token, so there is no need to jointly model commitment decisions across multiple unresolved positions within that step.

R-MDM introduces recurrence into masked DLMs but supports step-wise adaptivity: recurrent depth can vary across denoising steps, while all tokens generated within a step share the same depth. It therefore does not model distinct commitment depths for individual tokens within a step.

Our method enables token-level adaptivity, where early-committed tokens become context and influence subsequent predictions at unresolved positions. We therefore model a joint exit schedule for the sequence and use the sequence-level prediction loss to assign credit to each exit decision, accounting for its effects beyond that token’s own prediction.

To empirically validate the effectivenss of this sequence-level loss, we follow your suggestion and conducted a controlled ablation that changes only the prediction-loss credit used to train ExitGate. Let ℓi(s)\ell_i^{(s)} be token ii's prediction cross-entropy at pass ss, ziz_i its sampled exit depth, and Δi=ℓi(zi)−ℓi(1)\Delta_i=\ell_i^{(z_i)}-\ell_i^{(1)} its prediction cost relative to the first-pass baseline. Writing uiu_i for the log-probability of its realized stop/continue decisions, the prediction-credit terms are

\mathcal L_{\mathrm{SEQ}}^{\mathrm{pred}} &=\frac{1}{M}\sum_b \operatorname{sg}\left[\sum_{i\in\mathcal M_b}\Delta_i\right] \sum_{j\in\mathcal M_b}u_j,\\ &\mathcal L_{\mathrm{TOK}}^{\mathrm{pred}} =\frac{1}{M}\sum_b\sum_{i\in\mathcal M_b} \operatorname{sg}[\Delta_i]u_i. \end{aligned}$$ Here, $\mathcal M_b$ contains the supervised masked positions in sequence $b$, $M=\sum_b|\mathcal M_b|$, and $\operatorname{sg}$ denotes stop-gradient. **TOK** assigns each halting decision only its own token's prediction cost; **SEQ** shares the summed cost within each sequence, accounting for how a token's commitment affects other tokens through subsequent recurrent computation. Halting decisions remain token-specific in both variants. Both use identical denoiser supervision, first-pass baselines, KL-based compute regularization (including its sequence-level credit), and training recipes. Using the decoding strategy from Table 1, the sequence-level variant achieves higher accuracy on **9 of 11 benchmarks**, with a reported average 1.60 percentage points higher than the token-level variant (80.30% versus 78.70%), as shown below. | Method | ARC-C | ARC-E | MMLU | MMLU-Pro | GSM8K | MATH | GPQA | MBPP | MBPP+ | HumanEval | HumanEval+ | Avg. | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | SEQ (Ours) | **94.40** | **98.10** | 76.60 | 63.90 | **94.20** | **80.80** | **49.50** | **81.50** | **72.50** | **87.80** | **84.20** | **80.30** | | TOK | 94.08 | 97.34 | **77.59** | **64.15** | 93.93 | 80.40 | 42.42 | 78.90 | 72.22 | 84.15 | 80.49 | 78.70 |
Reviewer_gzLMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ALoDLM introduces adaptive latent recurrence within diffusion decoding steps. Confident tokens are committed as discrete context, while unresolved positions retain continuous representations for further refinement. The method combines entropy-based commitment with learned halting and formulates joint exit schedules through a conditional variational objective. Experiments with Qwen3-based 1.7B and 8B models cover eleven benchmark scores, recurrence and halting analyses, and a GSM8K quality–throughput comparison using optimized inference engines.

Critical strengths
  1. The method combines latent refinement, discrete token commitment, and learned halting in a distinctive architecture. Its joint-schedule formulation addresses the coupling created when committed tokens become context for later passes, providing a substantive contribution beyond simply repeating Transformer layers.

  2. The evaluation includes two model scales and an optimized inference implementation. The eleven-task averages exceed the corresponding AR baselines at both scales. Separately, at matched 93.25% GSM8K accuracy, ALoDLM reports 612.4 versus 564.2 tokens/s for WeDLM on one B200, an 8.5% throughput gain. This supports a practical benefit at that operating point, although WeDLM remains faster in the highest-speed region.

Critical weaknesses

The incremental benefit of adaptive recurrence is not fully isolated from the conversion/SFT recipe. The K=2/4/8 ablations compare only recurrent variants and match training steps rather than training FLOPs. These results support the complete system, but leave the specific contribution of adaptive recurrence uncertain.

Questions for authors

Could you provide further explanation or evidence to distinguish the contribution of adaptive recurrence from the effects of the conversion/SFT recipe and training compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.