- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RELAY to address the issue of MDMs losing hidden state representations across consecutive denoising steps. The proposed method passes the latent representation from the previous step to the current one and trains the model end-to-end using BPTT.
Critical strengths
The method is practically valuable. By integrating hidden state passing with BPTT, RELAY remains compatible with Block Diffusion and KV caching, demonstrating engineering applicability.
Critical weaknesses
- Limited empirical evidence of generalization. The experiments are evaluated only on a single pre-trained 1.5B DLM and code generation benchmarks. The lack of evaluation across diverse model architectures, scales, and task domains makes it difficult to demonstrate the generalizability of the proposed method.
- Insufficient analysis of training and inference efficiency. RELAY requires full-parameter BPTT, which is not friendly for training. However, the paper does not report actual wall-clock training times to show the overall computational efficiency. Additionally, the inference efficiency is evaluated primarily via the NFE, lacking wall-clock latency or throughput.
Questions for authors
- Could the authors provide wall-clock inference latency and throughput? 1. How much additional wall-clock training time and computational cost does RELAY need compared to standard SFT?
Dear Reviewer Pvk9,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC