acceptodds
Under review as a conference paper at ICLR 2027

Understanding Evaluation Inconsistency in Diffusion Large Language Models

Abstract

Despite the capability of parallel decoding, diffusion large language models (dLLMs) require many denoising steps to maintain generation quality, motivating recent research on efficient decoding strategies. However, existing studies have reported inconsistent benchmark results even in seemingly identical evaluation settings, undermining the reliability of conclusions about dLLM decoding. To understand this issue, we perform a comprehensive empirical study of existing dLLM decoding methods across diverse evaluation protocols. Our analysis reveals that the performance gains of decoding methods are highly prompt-dependent, resulting in inconsistent rankings of methods across prompt templates. We further attribute this evaluation inconsistency to the high sensitivity of parallel decoding methods to prompt design, where even minor prompt variations can significantly alter generation behavior. Notably, we find that a well-chosen prompt template can yield markedly greater accuracy gains than simply increasing the decoding budget. Beyond prompt templates, our experiments indicate that other commonly overlooked evaluation settings can also affect the assessment of decoding methods. Based on these findings, we provide practical guidelines for reliably evaluating decoding methods for diffusion large language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evaluation Inconsistency, Diffusion Large Language Model, Decoding Methods

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J769OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a comprehensive evaluation and analysis of decoding methods for dLLMs and reveals the sensitivity of dLLMs to the input prompt. Based on these findings, the paper provides practical guidelines, such as multi-prompt evaluation and detailed reporting of evaluation settings, to improve the reliability of evaluating dLLM decoding methods.

Critical strengths

S1. This paper helps in understanding how reliably dLLMs respond to prompts in real-world deployment.

S2. From an engineering perspective, this is a useful evaluation study.

Critical weaknesses

W1. The novelty of this paper is very limited. It mainly studies the effect of different prompts; it proposes no new method, and its analysis does not yield new insights beyond well-known prompt sensitivity and standard evaluation noise. Both its contributions and its motivation amount to a statistical summary from an engineering perspective. In terms of the significance of its contributions, the importance of the research question and its scientific rigor, I believe the paper falls well short of the ICLR bar.

W2. The paper's central premise is that evaluation inconsistency is a substantive, prompt-driven property of dLLM decoders. The current evidence does not support this convincingly. It remains unclear whether the phenomenon simply reflects standard evaluation statistics, or whether the reported differences fall within evaluation noise. The remedy the paper proposes, "averaging over multiple templates", effectively just increases the sample size. Moreover, since the work is essentially a statistical analysis, the experiments should report more detailed significance tests.

W3. The paper claims that evaluation inconsistency arises because dLLM parallel decoding is more sensitive to prompts. However, the mechanistic explanation relies on a single case study. The authors should provide more cases, including failure cases, and should support the explanation with statistical experiments and causal analysis. Moreover, the experiments show that the output divergence of the autoregressive model is almost as large as that of the dLLM, which does not seem to support the paper's claim that autoregressive models are more stable.

W4. I currently have doubts about the fairness of how the different methods were reproduced. The authors should report all reproduction details thoroughly and in full.

Questions for authors

The experimental results could be presented in more diverse forms. Currently, there are very few qualitative experiments and interpretability analyses.

Reviewer_6e9COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reliability of evaluations of dllm decoding algorithms.

It compares several decoding methods across prompt templates using LLaDA-8B-Instruct, LLaDA-1.5, and supplementary Dream experiments on mathematical and code benchmarks.

The results show that prompt choices can affect accuracy and change the relative rankings of decoding methods.

The authors analyze output similarity and decoding trajectories to investigate this sensitivity and decoding budget, few-shot examples, generation length, and hardware.

The paper recommends evaluation across multiple prompts and detailed reporting of evaluation configurations.

Critical strengths
  • The problem is real, subtle and impactful in diffusion evals.

  • The evaluation is comprehensive.

Critical weaknesses
  • Limited mechanistic insight and conceptual novelty.

    • The paper documents increased prompt sensitivity in the evaluated parallel diffusion decoding strategies, but does not provide a convincing mechanistic explanation for why they are more sensitive than single-token decoding. Its analyses of output similarity and decoding trajectories describe behavioral differences without clearly identifying the underlying mechanism. As a result, the contribution mainly characterizes the observed sensitivity and offers limited new understanding of its causes, which constrains the paper's conceptual novelty. (Section 4.1; Figure 3; Appendices C.6 and D.1.)
  • Insufficient evaluation on more recent diffusion language models.

    • The experiments focus on LLaDA-8B-Instruct and LLaDA-1.5, with supplementary results on Dream, but do not evaluate newer models such as LLaDA2.0 and the SDAR series. Both adapt autoregressive models into block diffusion models, introducing training and generation differences from the main LLaDA baselines. The authors should examine whether the observed prompt sensitivity and decoding-method ranking instability persist in these newer models under comparable evaluation conditions. These effects may be substantially weaker, which would affect the scope and current relevance of the paper's conclusions. Without such evaluation, the generality of the findings remains uncertain.
Questions for authors
  • Clarification of the mechanistic claim:

    • Does any existing experiment isolate the effect of position selection from prompt wording, demonstrations, parallelism, and caching within the same diffusion model? If not, would the authors qualify the causal interpretation of Section 4.1? A controlled intervention on unmasking order would be a targeted way to strengthen this claim, rather than a requirement to evaluate a broad new collection of models.
  • Applicability to more recent models:

    • Do the reported prompt-sensitivity effects and ranking reversals persist in more recent diffusion language models such as LLaDA2.0 and the SDAR series? Under comparable evaluation conditions, are these effects substantially reduced, and how would that affect the scope of the paper's conclusions?
Reviewer_NgkHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that dLLM decoding methods are highly prompt-sensitive, leading to inconsistent rankings across prompts, and that a good prompt matters more than a larger decoding budget; the authors provide guidelines for reliable evaluation.

Critical strengths

This paper provides a thorough analysis on the inference and evaluation of dllms. The provided evidences may help the users of dllms in recent versions.

Critical weaknesses

The problem explored in this paper is of limited value.

A key reason for the poor inference stability of current dllms is insufficient training—both in terms of training data volume and model parameters—and this issue will be greatly alleviated as data and parameters scale up (similar precedents exist in AR models).

Therefore, the conclusions of this paper may only be overfitted to the current state of dllms, making them difficult to apply to future dllms.

Questions for authors

See Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.