acceptodds
Under review as a conference paper at ICLR 2027

Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

Abstract

When Masked Diffusion Models (MDMs) generate sequences through iterative refinement, the rich internal computation over masked positions is discarded, forcing every subsequent refinement step to recompute the valuable internal information stored as model representations. To avoid a hard reset between denoising rounds, we propose Learned Relay Representations (Relay), a method that allows MDMs to be forward-thinking when denoising by explicitly learning how to propagate latent information for the benefit of future denoising steps. Relay introduces a differentiable per-token channel that passes information between forward passes and is trained via truncated backpropagation through time (BPTT). We show that this framework can be scaled to state-of-the-art Diffusion Language Models (DLMs), and is seamlessly compatible with techniques like block diffusion and KV caching. We first provide a thorough justification of the design choices in Relay on a challenging Sudoku-based planning task. We then scale Relay to Fast-dLLM v2, a state-of-the-art DLM, outperforming standard supervised finetuning on coding tasks while reducing inference latency by up to 32%. Our empirical results demonstrate that state-of-the-art DLMs can be explicitly trained to relay latent information forward across decoding steps, advancing the performance-latency Pareto frontier. We provide code for all our experiments.

Primary area:
generative models
Keywords:
discrete diffusion, dLLMs, BPTT

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Pvk9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RELAY to address the issue of MDMs losing hidden state representations across consecutive denoising steps. The proposed method passes the latent representation from the previous step to the current one and trains the model end-to-end using BPTT.

Critical strengths

The method is practically valuable. By integrating hidden state passing with BPTT, RELAY remains compatible with Block Diffusion and KV caching, demonstrating engineering applicability.

Critical weaknesses
  1. Limited empirical evidence of generalization. The experiments are evaluated only on a single pre-trained 1.5B DLM and code generation benchmarks. The lack of evaluation across diverse model architectures, scales, and task domains makes it difficult to demonstrate the generalizability of the proposed method.
  2. Insufficient analysis of training and inference efficiency. RELAY requires full-parameter BPTT, which is not friendly for training. However, the paper does not report actual wall-clock training times to show the overall computational efficiency. Additionally, the inference efficiency is evaluated primarily via the NFE, lacking wall-clock latency or throughput.
Questions for authors
  1. Could the authors provide wall-clock inference latency and throughput? 1. How much additional wall-clock training time and computational cost does RELAY need compared to standard SFT?
1 comment

Dear Reviewer Pvk9,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_vJAbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RELAY, a simple and practical method for mitigating information loss across denoising steps in masked diffusion language models. RELAY passes position-wise hidden states to subsequent steps and trains these representations through truncated BPTT. The method is clearly described, and experiments on Sudoku, HumanEval, and MBPP demonstrate improvements in the accuracy–NFE trade-off.

Critical strengths
  1. The method is well motivated and clearly described: passing position-wise hidden states to future denoising steps through a simple layernorm function, and using BPTT to receive gradients from future K step losses to guide current step's hidden states to be more aware of future learning targets. It is simple to implement and easy to understand.
  2. The ablations show the effectiveness of RELAY for decreasing NFEs while keeping performance, and decompose the contributions of passing current hidden states to future and BPTT training.
Critical weaknesses
  1. Limited evidence across model scales and families. It remains unclear whether RELAY’s gains persist as the base model becomes larger or stronger.
  2. Limited task diversity. The evaluation covers Sudoku and two code-generation benchmarks. Mathematical reasoning benchmarks for the language model would be particularly informative, given that mathematical examples constitute 60% of the SFT data. Evaluation on more open-ended text-generation tasks would also help establish the method’s broader applicability.
Questions for authors
  1. The four SFT configurations appear to share the same main training hyperparameters. How were these hyperparameters selected? Using shared settings is a reasonable controlled comparison, but have the authors examined whether the relative gains are robust to hyperparameter choices, or to tuning each method under a matched search budget?
  2. Table 2 uses a fixed confidence threshold of 0.85. Since this threshold affects both accuracy and NFE, could the authors sweep it for each method and report accuracy–NFE curves, as done for Sudoku? Comparing NFE at similar accuracy levels would help establish whether RELAY’s gains persist beyond this particular threshold.
1 comment

Dear Reviewer vJAb,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_TAckOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a “hard reset” in masked diffusion models (MDMs): hidden representations computed at each denoising step are discarded, so subsequent steps condition only on committed tokens. RELAY addresses this by feeding a learned projection of the previous step’s final-layer hidden state into the next forward pass. It is trained end-to-end via truncated BPTT (K=2) over on-policy, confidence-thresholded unmasking rollouts. Experiments cover a ~7M-parameter model trained from scratch on Sudoku-Extreme, where RELAY dominates MLM, Rollout and a stop-gradient variant, RELAY (sg), on the accuracy–NFE frontier. The paper also covers SFT of Fast-dLLM-v2 (1.5B) on coding datasets, where RELAY matches the Rollout baseline in accuracy with fewer NFEs on HumanEval.

Critical strengths
  1. Well-motivated method with informative ablation studies. Contrasting RELAY (sg) with RELAY isolates the contribution of temporal credit assignment, i.e., learning to write a useful state rather than merely read one. On Sudoku this yields consistent gains in accuracy, NFE and board legality.
  2. Practical compatibility. Because the decoding procedure is untouched, RELAY can be retrofitted onto existing dLLMs with block-wise decoding and KV caching.
Critical weaknesses
  1. Compute is not controlled. BPTT increases per-example training cost, and the authors acknowledge longer training time without quantifying it. Comparisons are matched by samples or steps instead of compute; for instance, it is unclear whether the four models in Sec. 4.1 share a compute budget. Part of the reported improvement may therefore come from additional training compute rather than from the relay mechanism.
  2. Closest prior work is absent from the baselines. Loopholing and Residual Context Diffusion, both discussed in the paper, likewise inject hidden states from preceding denoising steps and are conceptually close to RELAY (sg). Since implementation choices (injection site, projection, training objective) can matter substantially, and the paper’s novelty rests on BPTT adding value beyond these methods, a direct empirical comparison is necessary.
Questions for authors
  1. Is training compute (FLOPs or GPU-hours) matched across MLM, Rollout, RELAY (sg) and RELAY in Sec. 4.1 and 4.2? If not, can compute-matched results be provided?
  2. Can RELAY be compared directly against Loopholing and Residual Context Diffusion under an identical setup?
  3. The gains of RELAY appear concentrated on Sudoku. On coding, the fine-tuned Rollout baseline performs comparably and is stronger on HumanEval (42.7/39.6 vs. 42.1/37.2), and RELAY no longer improves over Rollout. Does this suggest that relayed hidden states carry limited additional information for language, and code in particular? What is the authors’ intuition for this?
  4. A plausible explanation for the limited gains on code is that Fast-dLLM-v2 is not pretrained with relayed states and received only 200 adaptation steps. A from-scratch language-modeling comparison, e.g., training on OpenWebText and reporting generative perplexity of unconditional samples, would clarify whether the Sudoku gains transfer to language.
1 comment

Dear Reviewer TAck,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.