acceptodds
Under review as a conference paper at ICLR 2027

When Does Decoding Order Affect Coverage in Diffusion Language Models?

Abstract

Unlike an autoregressive language model, a diffusion language model can choose which positions to fill next. Prior work reports that forcing diffusion language models to decode from left to right can improve repeated-sampling coverage. For dLLMs, coverage (Pass@) matters because it shows how effectively repeated sampling finds successful solutions for test-time scaling and reinforcement-learning exploration. We ask whether left-to-right position order causes the gain. Our central insight is that changing order also changes which tokens commit together, what context they can use, when the decoder resolves uncertain choices, and how sampling temperature affects token choices. We hold model weights and the decoding loop fixed across three open diffusion language models. The left-to-right advantage is strongest with one token per step at low temperature, varies across model–benchmark pairs, and can disappear or reverse as width or temperature increases. At width two, left-to-right decoding commits neighboring tokens together, so the right-hand token cannot use its newly sampled predecessor. Avoiding these adjacent co-commits recovers much of the LLaDA/HumanEval coverage loss, with a compatible but less certain result on Dream. We also observe positive coverage changes without fixed left-to-right priority. Confidence-order decoding at a higher temperature reaches comparable coverage, while committing selected positions earlier yields weaker positive estimates. When we keep the number of early commitments roughly fixed, moving them later lowers coverage and raises single-sample accuracy. Thus, decoding order combines several decoder choices rather than changing position order alone.

Primary area:
generative models
Keywords:
Diffusion Language Models, Decoding Order, Parallel Decoding, Pass@k, Sample Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5CkgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the Pass@k gain of forcing diffusion language models to decode left to right (LTR), reported by Ni et al. (2026), really comes from the position order itself. The authors split decoding order into several coupled choices, including position priority, commit width, adjacent co-commitment, sampling temperature, and commitment timing, and compare them with weights and the decoding loop held fixed. Experiments on LLaDA-8B, Dream-7B, and Nemotron-3B across GSM8K, MATH500, HumanEval, and MBPP show that the LTR advantage is strongest at one token per step and T=0.6. When the width or temperature increases, the advantage shrinks or even reverses. Most of the mechanism analyses, however, are done only on LLaDA/HumanEval.

Critical strengths
  1. The statistical analysis is careful and transparent. Every comparison is paired on the same problems with a bootstrap, the main results are re-checked with additional significance tests, multiple-testing correction, and fresh samples, and the paper clearly marks which analyses were added after seeing the exploratory results. This is useful because readers can tell which conclusions are reliable and which are weaker.
  2. The paper shows that the left-to-right advantage does not hold in parallel decoding. When the model commits one token per step, left-to-right decoding is 6.3 points better, but with two or four tokens per step it becomes 2.8 and 21.6 points worse, while confidence-based order stays nearly unchanged under the same compute budget. This is useful because diffusion language models are usually used with several tokens per step for speed, so results obtained at one token per step should not be directly carried over.
Critical weaknesses
  1. The major concern is that the mechanism conclusions mostly come from a single setting, i.e., LLaDA/HumanEval with 164 problems, and many of them are follow-up analyses chosen after seeing the exploratory results. However, the abstract states the timing result in general terms, i.e., moving commitments later lowers coverage. In fact, within that setting, S=0 and S=8 are almost the same, so the reported decline starts from S=8, a point picked after looking at the sweep. Besides, the width-two control changes more than one thing at a time by the authors' own account, and outside LLaDA its support is weak: the Dream interval has a lower bound of exactly 0.0. In addition, for the main contrasts, coverage is mostly reported only at k=32, while coverage claims are usually checked over larger k. Since the paper itself admits that the mechanism is only partly identified, it is hard to tell which conclusions hold beyond this one setting.

  2. Another concern is that the technical novelty of the paper seems somewhat limited. The paper is mainly an empirical analysis of existing decoders, i.e., LTR and confidence order, under different widths and temperatures, and the width-two control follows the idea of dilated scheduling from prior work. The only new decoding rule, candidate-first, gains just +3.0 points on LLaDA/HumanEval, and its interval includes zero when only fresh samples are used. Besides the useful finding that the LTR advantage depends on width and temperature, the paper does not offer a reliable solution or some potential directions for how the coverage gain could be kept. The authors are encouraged to address these issues.

Questions for authors
  1. Have the authors repeated the timing and width-two controls on another model or benchmark with fresh samples?
  2. Have the authors found any decoding strategy that keeps the coverage gain at two or more tokens per step? Based on the findings, could the authors provide some other guidance or potential directions for designing better decoders or training strategies for diffusion language models?
Reviewer_jrMGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when left-to-right (LTR) decoding improves Pass@k in diffusion language models. Using three frozen pretrained models, it shows that the LTR advantage depends on commit width, local context, commitment timing, and temperature. Non-adjacent commitment partially recovers parallel decoding losses, while alternative policies can improve coverage without enforcing LTR order.

Critical strengths

The paper provides a useful empirical decomposition of decoding-order effects instead of attributing coverage improvements solely to LTR ordering. Evaluations across multiple models, paired uncertainty estimates, and matched-compute comparisons strengthen its findings. I also appreciate the authors' transparency regarding confounding factors and negative results.

Critical weaknesses
  1. Limited mechanistic identification. Although several relevant factors are examined, the interventions do not fully isolate their effects. Confidence-based selection depends on sampled token probabilities, while non-adjacent commitment changes both positional order and available bidirectional context. Moreover, the strongest LTR gains arise in sequential decoding, sacrificing diffusion's parallelism, without remedy like Dspark or relevant dLLM work.

  2. Insufficient connection to training and optimization. The experiments modify only inference-time ordering. Thus, they cannot distinguish intrinsic ordering advantages from differences in denoiser accuracy across the visited masked states. The related works Kim et al. (PUMA, https://arxiv.org/abs/2602.10314) and Bu et al. (DPRM, https://arxiv.org/abs/2604.24357) deserve discussion. In particular, DPRM studies how ordering reshapes optimization difficulty and conditional sample complexity, suggesting that LTR may not be preferable once training efficiency is considered. The current study also provides limited insight into whether its effects generalize to difficult, long-tail reasoning behaviors that require rare order

Questions for authors

Q1. Could the authors better isolate parallel conditional-dependence errors by comparing independent versus sequentially conditioned sampling of the same selected token positions? Could pre-sampling confidence selection also help separate positional priority from sample-dependent selection?

Q2. Why should LTR be preferable for language but potentially suboptimal for reasoning or planning? PUMA and DPRM suggest that train–test alignment and optimization difficulty can substantially affect ordering preferences. Could the authors discuss these connections and, if feasible, evaluate whether DPRM-style confidence-to-reward-guided ordering reproduces their observations in your scenario

Reviewer_22mPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates when and why forced left-to-right (LTR) decoding improves repeated-sampling coverage over confidence-based decoding in diffusion language models. Treating decoding order as a bundle of coupled choices, it evaluates three models across multiple benchmarks and studies commit width, temperature, adjacency, and early commitment, alongside influence and diversity analyses. The LTR advantage is strongest under sequential, low-temperature decoding. Non-adjacent scheduling mitigates parallel LTR degradation, while higher temperature and early commitments can improve coverage without fixed LTR priority, often at a cost to Pass@1.

Critical strengths
  • The paper addresses a relevant question for test-time sampling and exploration across several dLLMs and benchmarks.

  • It reports uncertainty estimates and statistical sensitivity analyses, and discusses limitations of its controls honestly.

Critical weaknesses

The paper does not separate the effect of position order from the preference for high-probability samples.

At width one, confidence decoding samples a token at each masked position and commits the sample with the highest confidence. LTR instead accepts the sample at the leftmost masked position, regardless of its probability. The temperature and early-promotion experiments do not establish how much of the coverage gap comes from this difference and how much comes from commitment timing.

The follow-up controls do not establish which changes produce the coverage gains.

Strided LTR changes both relative order and access to committed neighbors. Masked positions already interact through attention([1]), so the relevant distinction is access to a neighbor’s sampled value. The timing comparisons do not hold promotion counts fixed, targeted selectors do not clearly outperform random promotion, and the influence analysis does not test whether postponing the same decision reduces its influence.

[1] He et al., Reasoning with Latent Tokens in Diffusion Language Models, 2026.

The explanations emphasized as contributions largely repeat ideas from prior work.

Non-adjacent scheduling follows the principle established by [2], while [3] already discuss the higher optimal temperature of confidence decoding. The paper evaluates these ideas across more settings, but I do not see a substantial new explanation emerging from the additional diagnostics.

[2] Luxembourg et al., Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models, in ICML 2026.

[3] Ni et al., The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models, in ICML 2026.

Key results are scattered across too many supplementary tables.

Repetitive numerical reporting and scattered tables make the main comparisons difficult to follow. Much of the results section restates values already reported in tables, while closely related controls and outcomes are spread across different locations. Organizing each intervention together with its controls and main conclusion would make the argument clearer and reduce the need for repeated cross-referencing.

Questions for authors
  1. Does confidence decoding’s preference for high-probability samples contribute to its lower coverage? When committing one token per step, choose the position using the original sampled-token confidence rule, but independently redraw the token at that position from the same tempered distribution before committing it. This preserves the position-selection distribution at a given state while decoupling the committed token from the sample used to select the position. Does the LTR coverage advantage at T=0.6T=0.6 persist under this control?

  2. Does targeting specific positions improve coverage over promoting randomly chosen positions? Can the dictionary, entropy, and learned selectors be compared with uniform random promotion while matching both the exact number of promotions and their within-block step indices by construction? This would distinguish the benefit of choosing particular positions from the benefit of simply moving some commits earlier.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.