- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the hidden-state dynamics of masked diffusion language models. Through a series of experiments, the authors find that hidden states evolve smoothly across layers, become more predictive with depth and remain consistent across consecutive denoising steps. Motivated by these findings, they propose Cross-Step Recurrent Refinement (CSRR), which feeds the previous step's final hidden state into the first transformer block of the current step through a lightweight cross-attention module.
The authors fine-tune CSRR on the S1K dataset and evaluate it on reasoning (Countdown, GSM8K, SVAMP) and code (HumanEval, HumanEval+, MBPP) benchmarks using LLaDA-8B and Dream-7B. CSRR improves over the baseline across all reported settings, with a small parameter and inference overhead. The appendix also includes a fine-tuning control, KV-cache results, and sweeps over the recurrent scale, block size, and source and injection layers.
Critical strengths
-
The paper is easy to read, and the proposed method is simple and lightweight. CSRR adds only 0.82% parameters and 3.8% inference time and is compatible with KV-caching.
-
The evaluation covers a reasonable range of settings, including two model families (LLaDA and Dream), base and instruct variants, reasoning and code benchmarks, and different block sizes.
Critical weaknesses
-
The primary motivation for studying whether MDLMs "progressively refine" their representations is weak. Layer-wise iterative refinement is already well studied for autoregressive transformers, for example through the logit lens and tuned lens [1, 2] and analyses of residual networks [3]. The paper neither discusses this literature nor explains what is specific to the diffusion setting.
-
Similarly, the findings in Sections 3.1 to 3.4 are largely expected. For example, high adjacent-layer cosine similarity follows from the residual connections, and performance drops from layer removal have been studied for autoregressive LLMs [4]. The cross-step consistency in Section 3.4 is also largely expected. Moreover, all diagnostics use only 10 GSM8K examples, which is a very small sample size.
-
CSRR is conceptually very similar to self-conditioning [5], which is neither cited nor discussed. Self-conditioning feeds back the model's previous prediction, while CSRR feeds back the final hidden state, but training and inference follow the same structure (a stop-gradient pass whose output conditions the next pass).
-
Table 1 shows large improvements, but the baseline is not fine-tuned, while CSRR is fine-tuned on S1K. The matched fine-tuning control (Table S3) shows that much of the gain comes from fine-tuning itself. For instance, on LLaDA-8B-Instruct, CSRR improves over the FT control by only 0.23 on GSM8K and 0.33 on SVAMP. The main table should compare against this control.
References
[1] nostalgebraist. Interpreting GPT: The Logit Lens, '20
[2] Belrose et al. Eliciting Latent Predictions from Transformers with the Tuned Lens, '23
[3] Jastrzębski et al. Residual Connections Encourage Iterative Inference, '18
[4] Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers, '25
[5] Chen et al. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning, '23
Questions for authors
-
Can the authors compare CSRR with standard self-conditioning, where the previous step's predicted logits (or soft tokens) are fed back instead of the final hidden state?
-
Apart from Table S3, all results (including Table 2 and Figure 9) compare fine-tuned CSRR against an untuned baseline. Can the authors report the S1K-matched fine-tuning control for these settings as well?