- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the Pass@k gain of forcing diffusion language models to decode left to right (LTR), reported by Ni et al. (2026), really comes from the position order itself. The authors split decoding order into several coupled choices, including position priority, commit width, adjacent co-commitment, sampling temperature, and commitment timing, and compare them with weights and the decoding loop held fixed. Experiments on LLaDA-8B, Dream-7B, and Nemotron-3B across GSM8K, MATH500, HumanEval, and MBPP show that the LTR advantage is strongest at one token per step and T=0.6. When the width or temperature increases, the advantage shrinks or even reverses. Most of the mechanism analyses, however, are done only on LLaDA/HumanEval.
Critical strengths
- The statistical analysis is careful and transparent. Every comparison is paired on the same problems with a bootstrap, the main results are re-checked with additional significance tests, multiple-testing correction, and fresh samples, and the paper clearly marks which analyses were added after seeing the exploratory results. This is useful because readers can tell which conclusions are reliable and which are weaker.
- The paper shows that the left-to-right advantage does not hold in parallel decoding. When the model commits one token per step, left-to-right decoding is 6.3 points better, but with two or four tokens per step it becomes 2.8 and 21.6 points worse, while confidence-based order stays nearly unchanged under the same compute budget. This is useful because diffusion language models are usually used with several tokens per step for speed, so results obtained at one token per step should not be directly carried over.
Critical weaknesses
-
The major concern is that the mechanism conclusions mostly come from a single setting, i.e., LLaDA/HumanEval with 164 problems, and many of them are follow-up analyses chosen after seeing the exploratory results. However, the abstract states the timing result in general terms, i.e., moving commitments later lowers coverage. In fact, within that setting, S=0 and S=8 are almost the same, so the reported decline starts from S=8, a point picked after looking at the sweep. Besides, the width-two control changes more than one thing at a time by the authors' own account, and outside LLaDA its support is weak: the Dream interval has a lower bound of exactly 0.0. In addition, for the main contrasts, coverage is mostly reported only at k=32, while coverage claims are usually checked over larger k. Since the paper itself admits that the mechanism is only partly identified, it is hard to tell which conclusions hold beyond this one setting.
-
Another concern is that the technical novelty of the paper seems somewhat limited. The paper is mainly an empirical analysis of existing decoders, i.e., LTR and confidence order, under different widths and temperatures, and the width-two control follows the idea of dilated scheduling from prior work. The only new decoding rule, candidate-first, gains just +3.0 points on LLaDA/HumanEval, and its interval includes zero when only fresh samples are used. Besides the useful finding that the LTR advantage depends on width and temperature, the paper does not offer a reliable solution or some potential directions for how the coverage gain could be kept. The authors are encouraged to address these issues.
Questions for authors
- Have the authors repeated the timing and width-two controls on another model or benchmark with fresh samples?
- Have the authors found any decoding strategy that keeps the coverage gain at two or more tokens per step? Based on the findings, could the authors provide some other guidance or potential directions for designing better decoders or training strategies for diffusion language models?