- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a comprehensive evaluation and analysis of decoding methods for dLLMs and reveals the sensitivity of dLLMs to the input prompt. Based on these findings, the paper provides practical guidelines, such as multi-prompt evaluation and detailed reporting of evaluation settings, to improve the reliability of evaluating dLLM decoding methods.
Critical strengths
S1. This paper helps in understanding how reliably dLLMs respond to prompts in real-world deployment.
S2. From an engineering perspective, this is a useful evaluation study.
Critical weaknesses
W1. The novelty of this paper is very limited. It mainly studies the effect of different prompts; it proposes no new method, and its analysis does not yield new insights beyond well-known prompt sensitivity and standard evaluation noise. Both its contributions and its motivation amount to a statistical summary from an engineering perspective. In terms of the significance of its contributions, the importance of the research question and its scientific rigor, I believe the paper falls well short of the ICLR bar.
W2. The paper's central premise is that evaluation inconsistency is a substantive, prompt-driven property of dLLM decoders. The current evidence does not support this convincingly. It remains unclear whether the phenomenon simply reflects standard evaluation statistics, or whether the reported differences fall within evaluation noise. The remedy the paper proposes, "averaging over multiple templates", effectively just increases the sample size. Moreover, since the work is essentially a statistical analysis, the experiments should report more detailed significance tests.
W3. The paper claims that evaluation inconsistency arises because dLLM parallel decoding is more sensitive to prompts. However, the mechanistic explanation relies on a single case study. The authors should provide more cases, including failure cases, and should support the explanation with statistical experiments and causal analysis. Moreover, the experiments show that the output divergence of the autoregressive model is almost as large as that of the dLLM, which does not seem to support the paper's claim that autoregressive models are more stable.
W4. I currently have doubts about the fairness of how the different methods were reproduced. The authors should report all reproduction details thoroughly and in full.
Questions for authors
The experimental results could be presented in more diverse forms. Currently, there are very few qualitative experiments and interpretability analyses.