- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes HaloBlock, a training-free decoding method for diffusion large language models (dLLMs). It identifies the left-dominance effect and contextual discontinuity at block boundaries of the block-decoding mechanism of dLLMs, and adaptively adds bidirectional halo regions based on token entropy and positional coefficients. Evaluation on LLaDA and Dream models show improved generation quality and inference efficiency in different settings.
Critical strengths
- The proposed method is easy to adapt, and the evaluation shows that HaloBlock can provide advantage over several baselines under various scenarios and settings, while only one hyper-parameter requires tuning.
Critical weaknesses
-
The paper's background and story line are not solid, examples:
- Block-wise decoding was not first introduced by [1], works including [2, 3] had already introduced it before.
- Decoding mechanisms like Fast-dLLM are not the only decoding and caching mechanisms of dLLMs; however, the paper builds its claim based on Fast-dLLM's asynchronous KV cache updates.
- The claim that contextual discontinuity is caused by asynchronous KV cache updates should be verified. In Fig. 2, the fluctuation reduction of HaloBlock is not obvious.
-
The implementation and evaluation scope are limited. While the paper claims to address one structural limitation of block-wise decoding in diffusion language models, the evaluation only uses models that were trained as full-sequence dLLMs. The authors should support and evaluate their method on block diffusion language models like SDAR, LLaDA2, DiffusionGemma, or even the smaller BD3LM paper they bring up at the beginning. Moreover, baselines like DCD [4], which directly tries to address the block boundary problem, should be compared.
References:
[1] Arriola et al., Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, ICLR 2025.
[2] Han et al., SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control, ACL 2023.
[3] Sahoo et al., Simple and Effective Masked Diffusion Language Models, NeurIPS 2024.
[4] Shu et al., Deferred Commitment Decoding for Diffusion Language Models with Confidence-Aware Sliding Windows, arXiv 2026.
Questions for authors
-
Why are linear positional coefficients used in the bidirectional difficulty estimator? Did the authors evaluate alternative weighting functions?
-
Can HaloBlock be implemented on top of real block diffusion language models? How would it impact the KV cache integrity of the previous block?