acceptodds
Under review as a conference paper at ICLR 2027

Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported

Abstract

Diffusion language models (DLMs) promise fast parallel generation, yet high-quality samples often require large number of refinement steps, which diminishes their advantage in practice. This has led to massive interest in and rapid development of new methods for effective few-step generation. We show that much of the supposed quality gap at few steps can instead arise from a suboptimally configured sampler. Modest sampler sharpening, without any model retraining, enables a couple years old masked DLM to rival supposedly far improved successors. This differently sampled DLM in fact achieves lower generative perplexity in just 16 steps than what its standard sampler obtains with 1024, while improving both judged quality and semantic diversity. We further show that conventional per-output metrics can fundamentally obscure these gains, since any optimal trade-off between two such metrics can be attained by a generator supported on at most two outputs. We subsequently introduce GroupEval, which separately evaluates quality and across-output semantic diversity, and offers fresh insights including uncovering how 1.5-4.7x perplexity gains of a distilled model yield no corresponding quality gain. Finally, we explain why sharpening helps: parallel unmasking destroys dependencies among simultaneously generated tokens, creating a gap between prediction and generation. We prove that pervasive temperature choice of one is generically suboptimal under parallel sampling even for an exact denoiser, and that worse predictions can yield better samples. Through these results, we argue for a broader evaluation principle of treating the deployed generator as the object of comparison, benchmarking it against tuned baselines, and assessing quality and diversity jointly and with more human-aligned measures.

Primary area:
generative models
Keywords:
discrete diffusion, few-step generation, categorical sampling, posterior calibration, generative evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GmtMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the few-step generation capability of Masked Diffusion Language Models (MDLM), and claims that the performance gaps reported in the literature may come from sampler configurations. The authors show that with the pretrained weights of MDLM fixed, the perplexity can be improved by simply adjusting temperature and precision (quality and diversity judged by LLMs). To this end, the authors introduce GroupEval to separately measure output quality and group-level semantic diversity. A finite-step analysis is also conducted to explain the reason why better denoising predictions may not produce better generated distributions. The validation mainly includes experiments on the OpenWebText dataset with various settings, and comparisons with other benchmark methods.

Critical strengths

The evidence that the few-step generation performance of MDLMs can be clearly improved by simply adjusting the sampler configurations, with the weights of pre-trained models remain unchanged. This suggests that existing models may actually be stronger than reported in the literature, and should be properly tuned before concluding that a new method is better. Besides, the theoretical contribution may also be important for principled explanation (but did not check the correctness carefully).

Critical weaknesses

Aside from the limitations acknowledged by the authors in the “Scope and limitations” section, the main weakness of this manuscript is the clarity of its presentation. From my view, it is somewhat verbose and difficult to follow. This manuscript has considerable room for further improvement, e.g., making the narrative more concise and focused. In addition, the sampler tuning comparison experiments for DUO and SEDD should be included in the main submission, as this directly affects the evaluation of the paper's contribution.

Questions for authors

In the manuscript, the authors note that DUO and SEDD were not given comparable sampler tuning. I wonder if MDLM remain competitive if these models were similarly tuned? Additional experimental results or theoretical discussion would help clarify this point.

Reviewer_3esKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that part of the quality gap of MDLMs might be due to how the sampler is configured rather than to the model itself. Using the MDLM-Small public checkpoint, the authors vary the temperature and the Gumbel noise used in the categorical sampling (what they call sharpening). They show that this can lead to an improvement in GenPPL, comparable to more recent models like Duo or DiDi-Instruct. Moreover, the paper also proves that any optimal trade-off between two per-sample metrics can be achieved with just two outputs, so these metrics might not detect collapse. To address this, they then introduce GroupEval, and LLM-as-judge protocol for measuring quality and semantic diversity. Finally, they decompose the KL error into different sources to explain that sharpening can make sampling exact but worsening posterior loss.

Critical strengths
  • Group-level evaluations: to address the limitations of per-sample metrics, the authors propose GroupEval, which can separately measure per-sample quality (GEQ) and semantic diversity across samples (GED). The evaluation protocol is also fixed and deterministic, which makes it easy to reproduce and reuse.
  • More informative DLM evaluations: building on the previous point, the authors show that metrics such as GenPPL and token entropy can be misleading, consistent with findings in the field. This also motivates evaluation with tuned samplers and judging output quality directly.
  • Finite-step analysis: Theorem 1 decomposes the sampling error into a posterior error, a parallelization cost, and a path-mixing term. This clarifies why held-out loss alone cannot guarantee sampler quality.
  • Experimental setting and limitations: experiments are done with fixed checkpoints and frozen samplers. The judges are reported separately and the negative results and limitations are also clearly stated.
Critical weaknesses
  • Limited novelty: the main intervention is to lower the temperature and to use fp32 sampling, both of which are already explored in previous work. Also, related work on temperature tuning and parallel sampling could be better discussed and contextualized.
  • Weak theoretical contribution: the theoretical analysis seems slightly incremental. While the decomposition is useful for intuition (see strengths), it also relies on small problems, and does not cover the adaptive samplers used in practice. I also do not see the connection between the proposed mechanism and the observed gains, that is, whether the theory explains why sharpening improves results.
  • Empirical gains: GEQ improvements seem significant only for one budget. Moreover, GED is judge-dependent and there are no uncertainty estimates. The authors do not report qualitative samples either, which could be useful to better judge the claims.
  • Incomplete comparisons: in the experiments MDLM and DiDi-Instruct are tuned, but several other baselines use the default sampler. In addition, the temperature is selected on the same GenPPL/entropy used to report results, which has a risk of selection bias or overfitting to those metrics. The comparisons are not NFE-matched either, and other set-level metrics (like MAUVE) are mentioned but not reported.
  • Limited practical relevance: the experiments focus on small models and unconditional generation on OpenWebText. In addition, based on the additional results reported, sharpening does not seem to help larger (8B) models, and some gains occur only at very low step counts (where sample quality might be too poor).
Questions for authors
  • Could the authors better clarify what is new relative to Zheng et al. (2025) and in general what is the novelty of their sharpening approach beyond hyperparameter tuning?
  • Regarding the temperature effect: does τ=0.9\tau=0.9 still help when one token is revealed per step? For future work, it would be interesting to explore and compare how much of the gain also appears in autoregressive models of similar size.
  • How would DUO, DUO-DCD, and SEDD compare when given the same temperature/precision tuning? How would the results change under NFE-matched comparisons?
  • To better validate GroupEval, how does it compare with established set-level metrics such as MAUVE? Would it be possible to include qualitative samples for a better evaluation?
  • Could the authors better explain what does Theorem 1 add beyond existing factorization-error bounds? Also the practical consequences of it are a bit unclear.
Reviewer_Q52HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper revisits few-step generation in masked diffusion language models and argues that the performance of existing models has been underestimated due to suboptimal sampling configurations. The authors demonstrate that reducing sampling temperature and using fp32 categorical sampling substantially improves generative perplexity without retraining. They further introduce GroupEval, an LLM-based framework for evaluating sample quality and across-sample semantic diversity, and provide a theoretical analysis explaining why temperature one may be suboptimal for finite-step parallel sampling.

Critical strengths
  1. Relevant evaluation problem. The paper highlights the importance of controlling sampling configurations when comparing generative models. The substantial sensitivity of generative perplexity to temperature and numerical precision is a useful empirical observation.

  2. Interesting theoretical analysis. The finite-step KL decomposition and the analysis of parallel token sampling provide a meaningful explanation for why better posterior predictions do not necessarily translate into better finite-step generation.

Critical weaknesses

1. The main empirical superiority of the new sampler seems scientifically wrong.: Authors claimed the superiority for the fp32 and reduced temperature sampler compared to the existing fp64 based sampler simply by fixing the lower bound of the entropy and argue that it achieves lower gen PPL. I would like to argue that this is not scientifically correct.The substantial gPPL improvements are accompanied by reductions in entropy. The proposed entropy threshold of H ≥ 5 is heuristic and does not constitute a fair comparison at matched diversity. For example, Table 1 shows that 16-step MDLM♯ achieves gPPL 102.2 with H=5.35, whereas the 1,024-step sampler with temperature one and fp32 achieves gPPL 42.5 with H=5.33. Thus, the headline comparison against 1,024 steps does not hold under this approximately entropy-matched alternative. A full quality–diversity frontier comparison is necessary to demonstrate whether the proposed configuration represents an improvement beyond the conventional trade-off between sampling sharpness and diversity.

2.The proposed GroupEval metrics are insufficiently validated.: Although evaluating semantic diversity across samples is well motivated, GroupEval relies on subjective semantic-mode assignments produced by LLM judges. The resulting scores are sensitive to the choice of judge and the limited group size. In particular, GED can saturate when each group contains only eight samples, even if the generator covers a small subset of the underlying distribution. Moreover, the paper lacks direct human validation of the semantic grouping and quality assessments.Consequently, the reported improvements in GEQ and GED do not establish improved distributional fidelity or comprehensive semantic coverage. Comparisons with established diversity metrics and human evaluations would considerably strengthen these claims.

Questions for authors

Q1. Can authors prove that MDLM♯ really improves quality-diversity (gPPL / Entropy) trade-off? If entorpy is not useful metric so we would use the proposed GED, GEQ, how would we choose the temperature and diffusion steps? Also if the proposed sampler is really stronger than the original full-step sampler, why does this not transferred into the downstream tasks. For example, I checked Appendix A.8 where they showed greedy decoding fits best for downstream big dLLMs (which is already a well-known results) and this significantly limits the main contribution of the paper. Can authors provide their thoughts on this aspect?

Q2. Can authors show the scalability and the reliability of the proposed GED, GEQ metrics? either by comparing with diversity metrics or directly task the correlation with the pass@K performance in the test-time scaling in the bigger dLLMs.

Q3. Figure 1 of the paper argues the simple temperature tuning (with fp32 usage) beats distillation methods. But can authors provide full temperature sweep and GED, GEQ metrics for those distilled models? seems like this claim is only based on the gPPL.

To my current understanding, the current version of this paper is scientifically misleading which makes me lean toward clear rejection of the paper. However, if authors can resolve my concerns, i am willing to adjust my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.