acceptodds
Withdrawn from ICLR 2027

Stability-Weighted Decoding for Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion large language models (dLLMs) enable parallel text generation by iteratively denoising a fully masked sequence. However, existing decoding strategies rely on single-step static confidence metrics, ignoring temporal history and often prematurely unmasking unstable tokens. Under idealized ground-truth context revelation, we establish that the expected KL divergence between true conditional distributions equals the information gained about a token and lower-bounds its dependence on the previously masked context. Inspired by this result, we propose Stability-Weighted Decoding (SWD), a training-free, plug-and-play strategy that incorporates temporal stability into token scoring, using model-based temporal KL as an empirical sensitivity signal rather than a dependency guarantee. Experiments on code generation and mathematical reasoning benchmarks demonstrate that SWD improves generation accuracy across representative scoring metrics and exhibits exceptional robustness under aggressive acceleration ratios.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language model, inference method

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8PQHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWD, a training-free decoding rule that combines a base token score with an exponential penalty on temporal KL divergence. An information-theoretic result under ground-truth context revelation motivates the method. Experiments on LLaDA and Dream cover several scoring metrics, full-sequence and block-wise decoding, additional tasks, and the latency and memory overhead of the stability calculation.

Critical strengths
  1. The method is simple and has useful supporting experiments. The approximately NFE-matched comparisons retain substantial full-sequence quality gains, and the profiling study includes KL computation and memory overhead. The training-set validation protocol also gives a concrete way to select the penalty.

  2. The analysis is reasonably broad. The paper studies selection policies, block sizes, KL direction, and longer history. It also distinguishes prediction changes from reference-answer errors and separates the idealized theoretical result from the practical model-based signal.

Critical weaknesses
  1. The entropy scoring rule is unclear. Section 3.3 requires a nonnegative utility, but the detailed tables use negative entropy and negative thresholds. Multiplying a negative score by the proposed penalty raises it toward zero as KL increases, reversing the intended effect under largest-score selection. The exact transformation and threshold conversion are therefore important for interpreting and reproducing these results.

  2. The strongest gains come from full-sequence decoding, while the block-wise results are more mixed. Table 2 shows small average gains and a slight decrease for Margin. With the validation-selected Dream Block-32 penalty, Confidence accuracy decreases on HumanEval, GSM8K, and MATH500. The ordinary LLaDA Block-32 baseline also exceeds full-sequence SWD Confidence on these tasks at lower NFE, limiting the practical scope of the broad robustness claims.

Questions for authors

N/A

Reviewer_Yh6DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work uses changes in token predictions across decoding steps to decide which tokens to generate next. It delays unstable predictions without retraining the model.

Critical strengths
  1. The method is simple and does not require retraining the model.
  2. The reported computational overhead is small,
Critical weaknesses
  1. The gains are much smaller with Block-wise decoding than with full-sequence decoding. But block-wise decoding is very useful for dLLMs in practice.
  2. The Dream Block-32 results should be included in the main text. Evaluation on newer models released in 2026 would also help.
Questions for authors
  1. Will the authors release the code?
Reviewer_HLmLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a stability weighted decoding (SWD) strategy for dLLM. The authors introduce KL divergence between adjacent distributions to measure the temperal stability, which is well motivated and illustrated. Extensive experiments are performed on LLaDA and Dream models to show its improvement under the full-sequence settings.

Critical strengths
  1. The motivation is clear and the example in Figure 1 explains things well. The design of the SWD method is a reasonable solution to handle such instabilities.
  2. The SWD can be combined with various methods and is demonstrated to be effective under the full-sequence setting.
Critical weaknesses

The main weaknesses lie in the experiments.

  • The datasets used are relatively out-dated, e.g. the HumanEval and MBPP was released 5 years ago. Though dLLMs are far from being mature in comparison with AR LLMs, the reviewer hopes the evaluation can be performed on more recent benchmarks like LiveCodeBench v6. It would be even better if the authors could add some results on agentic tasks, which can serve as a complementary scenario.
  • The results of other methods seem to be lower than those reported in the corresponding original papers, e.g. KLASS-LLaDA/Dream, CreditDecoding-LLaDA. There exist gaps of over 10%. Some are due to different test settings like block partitioning. The large improvement can be quite misleading, since the gains from SWD in cases of blocks/semi-AR are marginal or even diminished, which is shown later.
  • There lack a systematic analysis of error attribution. Before diving deep into the method, what are the proportions of errors that are directly related to the mentioned instability? The authors give a good example in Figure 1, but it would be more convincing that this is a key problem if an statistical analysis is provided. This will help establish the anticipation of and explain the performance change when using the SWD.
  • The proposed SWD brings little gains in semi-AR settings. The reviewer hopes the authors could provide an in-depth analysis of this phenonmenon and improve the method rather than stopping at the current place, which makes the work and its impact quite limited.
Questions for authors

Datasets like HumanEval are far from "reasoning-heavy tasks".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.