acceptodds
Under review as a conference paper at ICLR 2027

When to Correct: Update Width Shapes Terminal Refinement in Uniform Diffusion

Abstract

Update width changes the relative benefit of terminal refinement even when model-call counts are fixed. We compare correction during prediction with correction after prediction, using a fixed 33-step predictor grid and eight correction calls. Increasing width moves the terminal-to-late GenPPL ratio from 0.731 to 1.043 on LM1B and from 0.693 to 0.921 on OpenWebText. Larger updates improve both schedules, with late correction gaining more relatively. The direction observed on LM1B was predicted in advance and confirmed on OpenWebText. Later tests at intermediate widths in both models and on fresh seeds at a second correction budget support the same pattern. A separate LM1B study with 3,000 outputs per configuration compares terminal refinement with calibrated nucleus sampling. Terminal achieves 4.96% lower GenPPL and 14.69% shorter measured generation time, while nucleus achieves higher held-out prefix MAUVE. A placement comparison should therefore report update width alongside predictor and correction counts and test more than one width.

Primary area:
generative models
Keywords:
diffusion language models, uniform discrete diffusion, sampling, informed correctors, update width, inference-time computation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VzDvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how update width (the number of token positions resampled per correction call) affects the performance gap between different correction call placements in uniform discrete diffusion language models. Using an existing corrector with a fixed predictor grid and call budget, it finds that wider updates reduce the advantage of terminal refinement (all corrections after prediction finishes) over late-interleaved correction, as measured by GPT-2-large generative perplexity on OpenWebText and One Billion Word Benchmark (LM1B). The paper also compares terminal refinement with calibrated nucleus sampling in a separate LM1B study at a single width (k=3), finding lower generative perplexity and latency for terminal refinement but higher prefix MAUVE for nucleus sampling.

Critical strengths
  1. Controlled design: All placement schedules share the predictor grid, call count, and corrector, and the OpenWebText direction was predicted from the LM1B results before the run.
  2. Transparent reporting: The paper documents all seed cohorts and reports unfavorable results, including higher prefix MAUVE for nucleus sampling and the unresolved LM1B baseline gap.
Critical weaknesses
  1. Unclear presentation and motivation: Key terms are used before they are defined. For example. LM1B is never expanded, and OpenWebText and nucleus sampling are not cited. The motivation consists of a single question at the start of the introduction (“how that update width changes the benefit of finishing prediction first”), followed directly by results. The paper does not explain which existing placement comparisons fixed a single width, why width should be expected to change the comparison, or what a practitioner should do differently; its takeaway, a recommendation to report width, appears only in the conclusion. The nucleus comparison (4.4–4.5) is disconnected from the width question and is run only at k=3, although the paper’s own finding is that terminal refinement’s relative standing depends on width.

  2. The evidence is narrower than the claims: Each schedule is evaluated at a single temperature, although GenPPL rankings can flip with small entropy differences (Pynadath et al., 2026). The paper’s own temperature sweep shows that small changes in entropy produce large changes in GenPPL, yet the compared schedules are not matched in entropy. Terminal refinement produces less diverse text than both late correction (at narrow width) and nucleus sampling, so part of its GenPPL advantage may simply come from lower diversity. Additionally, the authors’ run of the public LM1B checkpoint gives GenPPL 189, far from the published value of 80 (Schiff et al., 2025), for reasons they could not identify, and the paper does not discuss whether its relative findings would hold for a model matching the published quality.

References

Pynadath, P., Shi, J., and Zhang, R. Generative Frontiers: Why Evaluation Matters for Diffusion Language Models. arXiv:2604.02718, 2026.

Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-Torre, H., de Almeida, B. P., Rush, A. M., Pierrot, T., and Kuleshov, V. Simple Guidance Mechanisms for Discrete Diffusion Models. In International Conference on Learning Representations (ICLR), 2025.

Questions for authors
  1. Could you report the width interaction and the terminal-vs-nucleus comparison at matched entropy, for example by sweeping temperature for each schedule? This would show how much of each effect remains once diversity is controlled.
  2. Have you been able to obtain a checkpoint or sampling setup that reproduces the published LM1B GenPPL? If not, why should the relative placement results carry over to a model at the published quality?
  3. Why was the nucleus comparison run only at k = 3? Does terminal refinement still compare favorably with nucleus sampling at k = 12?
Reviewer_5QaMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how correction placement and update width interact in uniform discrete diffusion language models. With predictor and correction call counts fixed, it finds that widening each correction reduces the relative GenPPL advantage of terminal refinement over late-interleaved correction. The study includes a prospective OpenWebText experiment, intermediate-width tests, a second correction budget, and comparisons with temperature and nucleus sampling.

Critical strengths
  • The study fixes predictor and correction call counts, uses paired-seed analyses and multiple-comparison correction, and tests its predicted interaction prospectively on OpenWebText. Intermediate-width experiments and a fresh-seed study with a second correction budget support the same trend.
  • The evaluation goes beyond GenPPL to include entropy, repetition, generation time, and prefix MAUVE. The authors also report negative results and implementation checks, helping clarify the scope of the findings.
Critical weaknesses
  • The main results concern evaluator likelihood, not overall generation quality. At larger widths, entropy decreases and repetition increases. The 3,000-output distribution study covers only width (k=3), where nucleus sampling achieves higher Prefix32 MAUVE than terminal refinement. Distribution quality at larger widths remains untested.
  • Update width is confounded with cumulative proposal budget, and absolute LM1B performance remains uncertain. With the number of correction calls fixed, increasing width also increases the total number of token proposals. The observed interaction therefore does not isolate the effect of width itself. In addition, the native LM1B baseline (GenPPL 189.09) differs substantially from the published result (79.80), and the source of this gap remains unresolved. The relative schedule comparisons are more convincing than the absolute performance claims.
Questions for authors
  • Does the width–placement trend persist when cumulative proposal budgets are matched across settings? Please report the corresponding difference in total model calls.
  • Could the authors report repetition rates and a distribution metric such as Prefix32 MAUVE at larger widths? Please also clarify that the confidence interval for the LM1B terminal/late ratio at the largest width includes 11, so the point estimate does not establish a reversal in scheduling preference.
Reviewer_prC3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the problem of random vocabulary tokens and tries to understand where to spend the corrector calls when sampling from uniform discrete diffusion language models. There are two placements consideration one is at the late stages and one is at the terminal. The finding is that the answer depends on how many tokens each correction step changes. When each step changes only a few tokens, correcting at the end is clearly better; when each step changes many tokens, that advantage mostly disappears.

Critical strengths

The problem sounds

Critical weaknesses
  1. the paper is written in sloppy with a lot of terms not well defined. In method, it jumps directly to PjP_j and CtC_t. no definition on the context, no definition on which model.

  2. The flow is very hard to read. The author seems to use AI to generate the manuscript without checking back what it has written. From the first sentence in abstract, it starts with "Update width changes the relative benefit of terminal refinement even when model-call counts are fixed". I believe that not a normal author would put this sentence at the first of the paper without any context.

  3. The central result is confounded by construction. Widening kk at fixed correction count also raises the total proposal count (d=Ck/Ld = Ck/L moves 0.19 → 0.75). “Wider updates” and “more proposals” are never separated. The authors state this in 2.3 and don’t resolve it.

  4. No mechanism proposed

  5. there are a number of problems in the experimental parts also. However, the paper should not be qualified even before reaching into reviewing exp.

Questions for authors

N/A, the paper needs re-writting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.