- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper carries out an empirical study of discrete diffusion language models across four different axes: evidence, selection, pace and verification. The authors evaluate how design choices in these four different axes affect generation quality across a number of models and datasets.
Critical strengths
- A thorough evaluation of different decoding techniques is currently lacking in the literature, to the best of my knowledge, and addressing this gap has a potentially significant impact on the community.
- Splitting the analysis across four different axes is a neat way to approach the problem and inform the readers how to design decoders. The interaction of pace and selection is particularly interesting.
Critical weaknesses
- The main weakness in my view is that the paper reads as a series of empirical observations, some of which are not novel. The paper could still be a valid contribution to the literature if it provided either (1) an analysis that explains the results and informs how to design better decoding strategies or (2) an evaluation protocol to standardize the evaluation of discrete LLMs in the literature. The paper seems to be closer in spirit to the latter but does not propose an actionable and structured protocol, stopping at the identification of the four axes of evidence, selection, pace and verification.
- Some of the empirical observations have already been observed in the literature before. For instance, the fact that generation quality drops with pace is well known in the literature and is the main motivation behind many decoding strategies, including DAPD and ADAS. Reproducibility issues due to batching have also been observed before: [1, Fig. 7] already reports a difference in perplexity due to batching (1 vs 8, in their case).
- I would not suggest the authors to expand the experiments necessarily; improving on the analysis would be much more interesting, as mentioned in my first comment. However, for the purely empirical paper we have now, the experiments are somewhat limited. There are four datasets covering only coding and reasoning, and only three selection methods (confidence, ADAS and DAPD). Other more recent and larger open-weight models, such as DiffusionGemma and LLaDA2.0, are not considered, although I appreciate that their computational requirements may make a thorough empirical evaluation impractical. A discussion of whether the findings generalize to these models would nevertheless strengthen the paper.
- The paper is mostly easy to read, but the observations are not clearly connected to a central research question or overarching conclusion. Sections 6 and 7 are very hard to follow though.
Minor comments
- The abstract contains many acronyms and numbers. This is perhaps a matter of personal preference, but I would avoid those in the abstract so that the main message can shine through more clearly.
- Some of the terms used throughout the text are confusing. For instance, it was not immediately clear to me what was meant by “numerical identity”, “reference fidelity” or “offline discrimination”. Defining these notions more clearly would make the paper read better.
- The mathematical description in the first paragraph of section 2 is never really used, unless I missed it. I would suggest dropping it from the paper entirely, as it does not add much.
References
[1] Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
- Why focus only on ADAS and DAPD? Why not also FastDLLM or EB-Sampler, for instance?
- The effect of batch size and numerical representation in decoding is well known in the literature and not exclusive to diffusion models, as the authors discuss in the related work. Could the authors comment on potential explanations and solutions for this phenomenon in the context of diffusion models? Are there fundamental differences between diffusion and autoregressive LLMs in that regard or could one borrow solutions proposed for autoregressive models [2]?
- In discrete diffusion models, the number of masks allocated for the generation has also been shown to be important [3]. How is that handled in the experiments and why not study the number of masks in this framework too?
- What are JUDGE and VJUDGE? Unless I missed it, these terms are introduced without context or citation.
- Sections 6 and 7 are hard to follow. Could the authors clarify what exactly is evaluated and discussed in those sections?
References
[2] Yuan, Jiayi, et al. "Understanding and mitigating numerical sources of nondeterminism in llm inference." Advances in Neural Information Processing Systems 38 (2026): 169819-169851.
[3] Piskorz, Julianna, et al. “Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models.” ICML 2026, PMLR 306:98965-99002, 2026.