acceptodds
Under review as a conference paper at ICLR 2027

Fill-and-Refine Decoding for Diffusion Large Language Models

Abstract

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive models. However, the vanilla decoding strategy suffers from a critical limitation: once an incorrect token is accepted, it cannot be revised, causing error propagation. Existing approaches perform refinement within the fill-up process, which limits their ability to exploit posterior information. In light of this, we propose FIRE, a two-stage decoding strategy consisting of a sequence fill-up stage followed by iterative refinement, where tokens are remasked and re-decoded conditioned on the remaining context. This mechanism allows previously accepted tokens to be reconsidered and corrected. Experiments on five benchmarks spanning language understanding, code generation, and mathematics demonstrate consistent improvements under identical computational budgets. These results highlight the importance of decoding algorithms in fully unlocking the potential of diffusion language models.

Primary area:
generative models
Keywords:
Diffusion Language Models, Inference-time Methods, Decoding Algorithms

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FQxZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates ReMDM, an unmasking and remasking schedule for diffusion language models. With several engineering tricks (optimizing the EoT penalty, and refinement rate annealing), they outperform a different setup of ReMDM and as well as RCR on two dLLMs across several benchmarks spanning knowledge, coding, and mathematical reasoning.

Critical strengths

The authors show generally strong knowledge of prior work and the models as well as the benchmarks (tasks) are extensive with experiments conducted up to statistical significance.

Critical weaknesses

There is little novelty. There is no theory and barely any methodological advancement. While the authors try to reframe their ReMDM setup with a different name, calling it cross-validation refinement, it is conceptually isomorphic to ReMDM under a different refinement scheduler.

The additional engineering tricks (such as EoT penalty and refinement rate annealing) are established in prior art, yet not applied to the baselines (such as RCR and ReMDM). Also, the authors should compare against threshold-based methods such as WINO etc.

Questions for authors
  1. The authors claim that "our approach departs from prior work by explicitly decoupling fill-up and refinement.". ReMDM already does this, see Section 4.2 "Design Strategies for ‘Turning On’ ReMDM". In fact, FIRE is isomorphic to ReMDM.

  2. The authors write: "Concretely, we penalize the logit of the EoT token by a factor λeot>1\lambda_{eot} >1 before softmax, using the sign-aware form of the standard repetition penalty (Keskar et al., 2019). While this adjustment does not directly improve draft quality, it effectively prevents early termination and produces drafts that are better suited for subsequent refinement.". Can the authors provide strong evidence that optimizing an EoT penalty does not directly improve draft quality? This would directly contradict the findings on the Dream models where an EoT token penalty was proven crucial for strong parallel generation under confidence/entropy based decoding. The claim on the EoT penalty is repeated in line 424-426, which likewise may have to be revised.

  3. The sign-aware choice of multiplicative penalty for the EoS token is odd, since logits are shift invariant. So the behavior of the penalty depends on a degree of freedom (dof) in the convergence of weights that does not affect model behavior, and said dof cannot be factored out by adjusting the penalty. Can the authors ablate on this and compare against a well-tuned additive shift on the logits at EOT positions?

  4. The authors do not compare against WINO, arguing that it was designed for block-attention and is threshold based. The argument is not convincing, WINO simply uses different threshold for drafting and verification (without 2-stage separation) and should be included. The threshold can be tuned to achieve comparable average NFEs. Moreover, WINO could also be applied under 2-stage draft-then-refine separation to compare threshold-based with fixed budget generation.

  5. In the experiments, the authors compare against "ReMDM" and RCR. While the former is just the same algorithm but for a different draft and refine window, RCR is actually a different algorithm. Moreover, the comparison is not entirely fair, since both methods seem to be missing an EOT penalty (with schedule) and the same annealing for the refinement rate. There are also no ablations identifying which degrees of freedom (changing tont_{on} and tofft_{off}, other annealing options, etc.) actually resulted in the accuracy improvements.

  6. On line 216 the authors state that they "remask a random subset", yet they do not indicate under what distribution. In this review I assumed they select the subset uniformly at random just like ReMDM. Can the authors provide clarification on this?

Reviewer_EgP6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FIRE, a training-free two-stage decoding strategy for diffusion large language models. It first generates a complete draft using standard diffusion decoding, then iteratively refines the output by randomly remasking and re-predicting subsets of tokens conditioned on the remaining context. The method also incorporates an EoT penalty and an annealed refinement schedule. Experiments on LLaDA and Dream across five benchmarks demonstrate improvements in generation quality, particularly under limited inference budgets.

Critical strengths
  • Simple and practical decoding framework. FIRE introduces a training-free two-stage decoding strategy that explicitly separates draft generation from refinement. The method is easy to implement and provides a practical way to correct previously accepted tokens.

  • Strong empirical results. The experiments demonstrate substantial improvements across multiple benchmarks, particularly under limited inference budgets. The evaluation covers different models and computational budgets, with useful ablation studies supporting the effectiveness of the proposed approach.

Critical weaknesses
  • Limited methodological novelty. The proposed method builds upon existing remasking and iterative refinement techniques. Although separating fill-up and refinement is a meaningful design choice, the core algorithm remains relatively simple, with limited methodological innovation beyond prior approaches.

  • Limited robustness of refinement. The performance gains are less consistent under larger inference budgets, sometimes resulting in accuracy degradation. Moreover, the refinement process does not naturally converge and may oscillate, raising concerns about its stability and general applicability.

Questions for authors
  1. Could the authors further clarify what makes the two-stage refinement strategy fundamentally different from existing remasking approaches? In particular, how much of the improvement comes from performing refinement after completing the draft, rather than from the remasking schedule itself?

  2. The paper acknowledges that refinement does not naturally converge and may oscillate. Could the authors provide further analysis of how refinement iterations affect output quality, particularly under larger inference budgets where performance occasionally degrades? This would help assess the robustness of the proposed approach.

Reviewer_VJ1QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a test-time decoding for diffusion language models (DLMs). The core motivation of this paper is to argue that existing DLM decoding methods fail to revise revealed tokens. To this end, this paper proposes a two-stage decoding strategy consisting of a sequence fill-up stage followed by iterative refinement, where tokens are remasked and redecoded conditioned on the remaining context. This mechanism allows previously accepted tokens to be reconsidered and corrected. Experimental results show the effectiveness of the proposed method.

Critical strengths
  1. The topic is interesting where how to improve the decoding quality of DLMs is a core challenge of modern DLMs.
  2. The motivation is clear and the uncorrected tokens need to be revised.
  3. The presentation is good and it is easy to understand.
Critical weaknesses
  1. This paper is outdated and its novelty is explored by existing open-source models. The core idea of drafting is already implemented by LLADA 2.1, which includes mask-to-token generation with token-to-token editing and supports revisiting earlier blocks. Meanwhile, LLADA 2.2 extends editing and insertion ability. These open-source models are not discussed. Although FIRE differs through test-timeremasking of a completed draft, presenting fill-and-refine as a fundamentally new paradigm is insufficiently justified.

  2. LLADA 2.1 fundamentally introduce the drafting ability and intuitively is better than a training-free method. The author may compare with it, but it may be very difficult to compare a training-based method and a training-free one.

  3. Evaluation uses only the original LLaDA and Dream models. It remains unclear whether FIRE offers additional benefits for newer diffusion models that already support editing.

Questions for authors

This paper is outdated and its novelty is explored by existing latest open-source models. The author can address my question in the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.