acceptodds
Withdrawn from ICLR 2027

On the Importance of Hidden-State Refinement in Masked Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Masked diffusion language models (MDLMs) generate text by iteratively denoising masked or corrupted token sequences, allowing token predictions to condition on bidirectional context and update multiple positions in parallel. While each denoising step produces a rich layer-wise hidden-state trajectory, prior work typically focuses only on token-level information and discards these continuous hidden states, leaving their refinement dynamics underexplored. In this work, we analyze hidden-state dynamics in MDLMs and find clear evidence of layer-wise and cross-step refinement: smooth evolution across adjacent layers, greater predictive power in deeper layers, gradual degradation under transformer block removal, and strong final-state alignment across consecutive denoising steps. Motivated by these observations, we propose Cross-Step Recurrent Refinement (CSRR), a mechanism that carries hidden-state information across denoising steps to enhance generation. Extensive experiments show that CSRR improves text generation quality, highlighting the critical role of continuous hidden-state refinement in MDLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vjwnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the hidden-state dynamics of masked diffusion language models. Through a series of experiments, the authors find that hidden states evolve smoothly across layers, become more predictive with depth and remain consistent across consecutive denoising steps. Motivated by these findings, they propose Cross-Step Recurrent Refinement (CSRR), which feeds the previous step's final hidden state into the first transformer block of the current step through a lightweight cross-attention module.

The authors fine-tune CSRR on the S1K dataset and evaluate it on reasoning (Countdown, GSM8K, SVAMP) and code (HumanEval, HumanEval+, MBPP) benchmarks using LLaDA-8B and Dream-7B. CSRR improves over the baseline across all reported settings, with a small parameter and inference overhead. The appendix also includes a fine-tuning control, KV-cache results, and sweeps over the recurrent scale, block size, and source and injection layers.

Critical strengths
  1. The paper is easy to read, and the proposed method is simple and lightweight. CSRR adds only 0.82% parameters and 3.8% inference time and is compatible with KV-caching.

  2. The evaluation covers a reasonable range of settings, including two model families (LLaDA and Dream), base and instruct variants, reasoning and code benchmarks, and different block sizes.

Critical weaknesses
  1. The primary motivation for studying whether MDLMs "progressively refine" their representations is weak. Layer-wise iterative refinement is already well studied for autoregressive transformers, for example through the logit lens and tuned lens [1, 2] and analyses of residual networks [3]. The paper neither discusses this literature nor explains what is specific to the diffusion setting.

  2. Similarly, the findings in Sections 3.1 to 3.4 are largely expected. For example, high adjacent-layer cosine similarity follows from the residual connections, and performance drops from layer removal have been studied for autoregressive LLMs [4]. The cross-step consistency in Section 3.4 is also largely expected. Moreover, all diagnostics use only 10 GSM8K examples, which is a very small sample size.

  3. CSRR is conceptually very similar to self-conditioning [5], which is neither cited nor discussed. Self-conditioning feeds back the model's previous prediction, while CSRR feeds back the final hidden state, but training and inference follow the same structure (a stop-gradient pass whose output conditions the next pass).

  4. Table 1 shows large improvements, but the baseline is not fine-tuned, while CSRR is fine-tuned on S1K. The matched fine-tuning control (Table S3) shows that much of the gain comes from fine-tuning itself. For instance, on LLaDA-8B-Instruct, CSRR improves over the FT control by only 0.23 on GSM8K and 0.33 on SVAMP. The main table should compare against this control.

References

[1] nostalgebraist. Interpreting GPT: The Logit Lens, '20
[2] Belrose et al. Eliciting Latent Predictions from Transformers with the Tuned Lens, '23
[3] Jastrzębski et al. Residual Connections Encourage Iterative Inference, '18
[4] Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers, '25
[5] Chen et al. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning, '23

Questions for authors
  1. Can the authors compare CSRR with standard self-conditioning, where the previous step's predicted logits (or soft tokens) are fed back instead of the final hidden state?

  2. Apart from Table S3, all results (including Table 2 and Figure 9) compare fine-tuned CSRR against an untuned baseline. Can the authors report the S1K-matched fine-tuning control for these settings as well?

Reviewer_B1eLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of hidden states in masked diffusion language models (MDLMs). The authors first analyze how hidden states change across transformer layers and denoising steps. They find that the representations evolve gradually across layers, become more predictive in deeper layers, and remain similar between consecutive denoising steps. Based on these observations, they propose Cross-Step Recurrent Refinement (CSRR), which reuses the final hidden states from the previous denoising step through a cross-attention module added to the first transformer block. The method keeps the original masked-token sampling procedure unchanged.

The authors evaluate CSRR on LLaDA and Dream models using several math reasoning and code generation benchmarks, including Countdown, GSM8K, SVAMP, HumanEval, and MBPP. The results show improvements over the original baselines, with relatively small additional parameter and inference costs. The paper also includes diagnostic studies and ablations to support the proposed design.

Critical strengths
  1. The main strength of the paper is that the proposed method is simple and has relatively low computational cost. CSRR reuses hidden states from previous denoising steps without changing the original sampling procedure. The authors report only 0.82% additional parameters and 3.8% inference time overhead. The method also shows meaningful improvements on some reasoning tasks, especially Countdown and GSM8K for LLaDA-8B-Base, even compared to standard fine-tuning. I think this makes the approach potentially useful, especially given the small additional cost.

  2. I also find the analysis of hidden states across denoising steps useful. In particular, the authors show that the final hidden states remain highly similar between consecutive steps, which provides some motivation for reusing this information instead of discarding it. Although some of the observations are expected from existing transformer architectures, I think studying them in the context of masked diffusion models is still interesting. The paper also connects these observations to a practical method, which I see as a positive aspect of the work.

Critical weaknesses
  1. My main concern is the novelty of the proposed method. The idea of reusing continuous hidden states across denoising steps has already been explored in previous works, such as Loopholing Discrete Diffusion (2025), MetaState (2026), and Learned Relay Representations (2026). These papers address a very similar problem. In particular, MetaState also uses cross-attention to incorporate information from previous steps. CSRR differs in how the hidden states are stored and where they are injected, but I am not fully convinced that these differences provide a significant new contribution. There is also no direct comparison with these methods. In addition, several observations in Section 3, such as the gradual changes in hidden states and the increasing predictive ability of deeper layers, are related to known properties of transformers (e.g., Elhage et al., 2021, Belrose et al., 2023). While studying these properties in diffusion models is useful, I think the paper needs to better explain what new insights are provided compared to existing work.

  2. My second concern is the experimental evaluation, especially whether the improvements actually come from the proposed mechanism. In the main tables, CSRR is compared with the original models without additional fine-tuning, while CSRR itself is fine-tuned on S1K using LoRA and the new module. The authors include a standard fine-tuning control in Table S3, but only for one generation setting. For example, on LLaDA-8B-Instruct, CSRR improves over standard fine-tuning by only 0.23 percentage points on GSM8K and 0.33 on SVAMP. These differences are quite small, and no variance is reported across runs. More importantly, CSRR is trained using consecutive denoising passes (K=2), but there is no control using the same multi-step training without hidden-state reuse. This is particularly important since previous work such as Learned Relay Representations has shown that multi-step training alone can improve performance. Without this comparison, it is difficult to understand how much of the improvement actually comes from the proposed cross-attention mechanism.

Questions for authors
  1. My main question is about the novelty of CSRR compared to existing methods, especially Loopholing Discrete Diffusion, MetaState, and Learned Relay Representations. These methods also reuse continuous hidden states across denoising steps, and MetaState also uses cross-attention. CSRR differs in how the hidden states are stored and where they are injected, but it is not clear what advantage these differences provide. Could the authors explain the main contribution compared to these methods and provide a direct comparison with them? I think this is important for understanding the novelty of the paper.

  2. Regarding the experimental results, Table S3 includes a standard fine-tuning control, but it only covers one generation setting. More importantly, CSRR uses consecutive denoising passes during training (K=2), while the standard fine-tuning control does not appear to use the same procedure. Could the authors include a control that uses the same multi-step training without hidden-state reuse, with the same training data and budget? Previous work such as Learned Relay Representations suggests that multi-step training alone can improve performance. I think this comparison is necessary to understand whether the improvements actually come from the proposed mechanism.

  3. Some of the improvements over standard fine-tuning are quite small, especially on GSM8K and SVAMP for LLaDA-8B-Instruct. Are these improvements consistent across different random seeds? Also, Appendix C.3 mentions that the best cross-attention scale α is selected from four values for the main results. Could the authors clarify whether this selection was based on a separate validation set or the test set? Table S1 provides fixed-scale results, which is useful, but these details would help better understand the reliability of the reported improvements.

Reviewer_bzksOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how hidden states change across transformer layers and denoising steps in masked diffusion language models (MDLMs). Based on the observed similarities between hidden states, the authors propose Cross-Step Recurrent Refinement (CSRR), which uses cross-attention to incorporate the previous denoising step's final hidden states into the current step. The method introduces additional trainable modules and is evaluated on LLaDA and Dream models. The experiments report improvements on mathematical reasoning and code generation tasks with relatively small inference overhead.

Critical strengths

Strengths

  1. The paper provides a useful analysis of hidden-state changes across transformer layers and denoising steps, instead of the token-level behavior typically studied in MDLMs.

  2. The proposed CSRR is relatively simple to implement. It reuses hidden states from the previous denoising step through a cross-attention module without changing the token sampling procedure.

  3. The authors provide several analyses, including hidden-state similarity and block removal experiments, to examine the motivation of their design choices. These help explain their choice of the hidden state source and injection location.

Critical weaknesses

Weakness

  1. Limited Novelty and Missing Comparisons: the core idea of propagating hidden states across denoising steps has already been explored in prior works, such as MetaState [1] and Relay [2]. In particular, MetaState also uses cross-attention to inject information from previous denoising steps. The proposed CSRR appears to be a relatively straightforward architectural variant, and its methodological novelty over these approaches is unclear. More importantly, the paper does not provide direct comparisons with these closely related methods.

  2. Insufficient Evidence for the Proposed Mechanism: the main results involve jointly training the new modules and LoRA adapters, making it difficult to isolate the contribution of hidden-state reuse. Although the authors include a matched fine-tuning baseline, the improvement on GSM8K for LLaDA-8B-Instruct is marginal (83.24 vs. 83.47). Additional controls, such as using shuffled or non-recurrent hidden states, would help establish whether the gains truly come from cross-step refinement.

  3. Limited Hidden-State Analysis: the analysis relies heavily on hidden-state similarity, but high cosine similarity between adjacent Transformer layers or denoising steps does not necessarily demonstrate meaningful refinement. Moreover, the main diagnostic experiments use only 10 GSM8K examples, which is insufficient to support broad conclusions about hidden-state dynamics.

[1] Xia et al. MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models.

[2] Rozonoyer et al. Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

Questions for authors

Please check Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.