- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces ALoDLM, a dLLM with token-adaptive latent recurrence. Committed tokens provide context, while unresolved tokens continue to refine their latent states. The authors derive a training objective for token prediction and the halting policy, and evaluate Qwen3-based models at 1.7B and 8B scales on eleven benchmarks.
Critical strengths
-
The evaluation includes competitive models such as SDAR and WeDLM. ALoDLM-8B achieves an average score of 80.3 across eleven benchmarks, compared with 75.1 for WeDLM-8B in Table 1. This supports the usefulness of the approach beyond early dLLM models.
-
Figure 3 reports an 8.5% throughput improvement over WeDLM at the same GSM8K accuracy of 93.25%. This is measured evidence of an inference benefit, rather than a claim based only on fewer operations.
Critical weaknesses
-
The experiments do not establish how much the proposed adaptive computation contributes. Figure 5 studies maximum recurrent depth and loop placement, but does not compare learned halting with fixed-depth recurrence at matched compute. A non-recurrent model trained with the same initialization, data, and training compute is also missing. These comparisons are needed to separate the method’s benefit from the training recipe.
-
The detailed speed comparison is limited to GSM8K with single-stream inference on one B200. ALoDLM improves throughput in the higher-accuracy range, while WeDLM leads at the fastest settings shown. Without comparable results on other tasks, the evidence for a broader inference advantage remains limited.
Questions for authors
-
With the same initialization, data, and training compute, how does ALoDLM compare with non-recurrent and fixed-depth models? Does learned halting improve accuracy at matched inference compute?
-
Appendix D.1 reports eight training epochs for the 8B model. What are the total training FLOPs or GPU-hours? The 5B-token corpus size alone does not establish the training cost.
-
Does the throughput improvement over WeDLM also hold on a code-generation benchmark at matched accuracy, using the same hardware and the native inference engines?
We sincerely appreciate your time, constructive feedback, and insightful questions. We address your concerns below:
[W1 & Q1: Relation to existing work and effectiveness of sequence-level loss]
MoR, PonderNet, and Ouro are in the autoregressive setting, each decoding step generates a single token, so there is no need to jointly model commitment decisions across multiple unresolved positions within that step.
R-MDM introduces recurrence into masked DLMs but supports step-wise adaptivity: recurrent depth can vary across denoising steps, while all tokens generated within a step share the same depth. It therefore does not model distinct commitment depths for individual tokens within a step.
Our method enables token-level adaptivity, where early-committed tokens become context and influence subsequent predictions at unresolved positions. We therefore model a joint exit schedule for the sequence and use the sequence-level prediction loss to assign credit to each exit decision, accounting for its effects beyond that token’s own prediction.
To empirically validate the effectivenss of this sequence-level loss, we follow your suggestion and conducted a controlled ablation that changes only the prediction-loss credit used to train ExitGate. Let ℓi(s) be token i's prediction cross-entropy at pass s, zi its sampled exit depth, and Δi=ℓi(zi)−ℓi(1) its prediction cost relative to the first-pass baseline. Writing ui for the log-probability of its realized stop/continue decisions, the prediction-credit terms are
\mathcal L_{\mathrm{SEQ}}^{\mathrm{pred}} &=\frac{1}{M}\sum_b \operatorname{sg}\left[\sum_{i\in\mathcal M_b}\Delta_i\right] \sum_{j\in\mathcal M_b}u_j,\\ &\mathcal L_{\mathrm{TOK}}^{\mathrm{pred}} =\frac{1}{M}\sum_b\sum_{i\in\mathcal M_b} \operatorname{sg}[\Delta_i]u_i. \end{aligned}$$ Here, $\mathcal M_b$ contains the supervised masked positions in sequence $b$, $M=\sum_b|\mathcal M_b|$, and $\operatorname{sg}$ denotes stop-gradient. **TOK** assigns each halting decision only its own token's prediction cost; **SEQ** shares the summed cost within each sequence, accounting for how a token's commitment affects other tokens through subsequent recurrent computation. Halting decisions remain token-specific in both variants. Both use identical denoiser supervision, first-pass baselines, KL-based compute regularization (including its sequence-level credit), and training recipes. Using the decoding strategy from Table 1, the sequence-level variant achieves higher accuracy on **9 of 11 benchmarks**, with a reported average 1.60 percentage points higher than the token-level variant (80.30% versus 78.70%), as shown below. | Method | ARC-C | ARC-E | MMLU | MMLU-Pro | GSM8K | MATH | GPQA | MBPP | MBPP+ | HumanEval | HumanEval+ | Avg. | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | SEQ (Ours) | **94.40** | **98.10** | 76.60 | 63.90 | **94.20** | **80.80** | **49.50** | **81.50** | **72.50** | **87.80** | **84.20** | **80.30** | | TOK | 94.08 | 97.34 | **77.59** | **64.15** | 93.93 | 80.40 | 42.42 | 78.90 | 72.22 | 84.15 | 80.49 | 78.70 |