- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Diffusion language models (dLLMs) generate text by iteratively filling masked positions, often committing multiple tokens in parallel. Tokens filled in the same step may be mutually inconsistent as they are not conditioned on each other. The work proposes Token-to-Mask (T2M), a training-free inference-time correction method that remasks tokens with probabilities less than a threshold and reconstructs them in later denoising steps. T2M improves performance on 15 out of 17 benchmarks on LLaDA2.1-mini against its T2T decoding. The work emphasizes improvements of remasking over immediate editing at the cost of additional inference computation.
Critical strengths
- The research question of whether an unstable token should be edited or remasked is an interesting direction in improving inference-time diffusion language model correction. While both directions exist in existing literature, the emphasize put by this work on this distinction is well motivated.
- The proposed method T2M is training-free and it is simple, intuitive, and easy to implement. The evaluation shows consistent improvements over a broad range of benchmarks which demonstrates the method's effectiveness.
Critical weaknesses
-
T2M requires approximately 2-3x times as many forward evaluations as the baseline T2T. Figure 3 explores different threshold settings, but does not fully establish that T2M is preferable under identical inference budgets. It may be expected that increasing compute during inference in diffusion models can improve results. Hence, the work is missing evaluations comparing T2M against different baselines under similar compute budget, which would better distinguish its capabilities.
-
The motivating problem of mutual inconsistency is not fully connected to the proposed solution. The paper starts by motivating the mutual inconsistency problem: positions filled in the same step are not conditioned on each other. However, the rest of the work, including the proposed solution, does not directly connect back to this problem. It is unclear why remasking based on a confidence threshold would resolve this specific problem rather than other types of inconsistencies.
Questions for authors
-
How does T2M compare with ReMDM-conf, CORE, and T2T when using similar inference budget? For some methods the budget can be increased by increasing the number of decoding timesteps.
-
How frequently do parallel commitments actually produce inconsistent tokens, and how often does T2M successfully resolve those inconsistencies? An (either qualitative or quantitative) analysis of how frequently such inconsistencies occur, how often T2M resolves or recreates them, and actual generation examples would strengthen the motivation.
-
Wide-In, Narrow-Out (WINO) (Hong et al., ICLR 2026) seems to be a closely related work that also uses a confidence threshold to remask previously committed tokens and reconstruct them later (please refer to Eq. 3). WINO uses shadow-based verification, whereas T2M directly evaluates the current token without masking it. WINO also combines remasking with aggressive parallel decoding and reports inference speedups over its baseline. Could the authors clarify what T2M contributes beyond WINO and, if feasible, compare their accuracy and inference efficiency under similar settings? In particular, how would WINO's shadow-based verification and decoding settings perform when applied to LLaDA2.1, compared to T2M?
3 comments
Rebuttal response
I thank the authors for the clarifications.
My primary concerns were threefold: novelty, motivation, and computation. After reviewing the rebuttal, I find that these concerns remain mostly unresolved.
Novelty: The rebuttal clarifies the distinction from existing methods such as WINO, however, I remain unconvinced that directly using visible-token probabilities for detection rather than existing similar verification mechanism constitutes sufficient methodological novelty.
Motivation: The hypothetical example is plausible but provides no qualitative or quantitative evidence that T2M resolves parallel inconsistencies.
Computation: While Table 3 provides useful comparisons under the same detector, it does not establish whether existing methods could achieve comparable or better performance under matched inference budgets.
Given these unresolved concerns, I have decided to lower our score from 2 (Weak Rejection) to 1 (Clear Rejection).
Thank you for the follow-up. Our earlier reply used a hypothetical example where we should have pointed to the evidence. Parallel inconsistency is a documented motivation for both remasking and token editing. Our question is what the decoder should do once it flags a committed token.
D3PM (§3) factorizes each reverse step across positions given the current sequence. Tokens filled together can therefore each be plausible under that context and still be incompatible with one another. ParallelBench gives a concrete test: when shuffling a list, two positions filled together can select the same item, violating the requirement that each item appear exactly once. Its analysis and LLaDA experiments show that even filling two positions per step becomes unreliable as the list grows (Eq. 5, Fig. 2). Structured Denoising Diffusion Models in Discrete State-Spaces · ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
WINO, which you cite, explicitly motivates revocable decoding with contradictory tokens produced during early parallel decoding (§4.1). It verifies draft tokens and remasks those that lose support. LLaDA2.1 identifies the same problem and introduces token-to-token editing so that later steps can overwrite earlier commitments (§§1–2). Our contribution is to use the correction-trained model’s probability of the visible token to trigger remasking and to test the choice of correction action while holding the detector fixed. Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs · LLaDA2.1: Speeding Up Text Diffusion via Token Editing
Our paper provides both qualitative and quantitative evidence for this choice. In the controlled example in Fig. 1(c), the model flags gasoline in “the firefighter extinguished the fire with gasoline.” It assigns water probability 0.27, below the edit threshold. After remasking, that probability rises to 0.87 and the model fills in water. Fig. 1(b) shows an actual GSM8K trace: the editor overwrites the correct units digit 5 with 2 and ends at 225, whereas T2M reopens the position and recovers the 5. On full GSM8K with the same LOWPROB detector, direct replacement gives 89.39% accuracy and deferred remasking gives 91.13% (Table 3). These experiments show how the correction action affects predictions and final answers. Estimating the fraction of same-step conflicts that T2M repairs would require labeling their origin in generation traces.
On computation, Table 3 also permits a comparison across detectors. On LLaDA2.1-mini, default T2M scores 91.13% at 0.584 forward evaluations per token, while CORE with direct replacement at interval 2 scores 85.22% at 0.642. Thus T2M uses fewer evaluations per token at these measured settings. How WINO and the other baselines would rank if each were optimized at a common inference budget remains open.
We understand why the absence of new experiments leaves some of your questions open. In your original review, you identified the choice between editing and remasking as a strength of the paper. In your follow-up, you acknowledge that the distinction from WINO is clearer and that Table 3 provides a useful comparison. Given that the remaining concerns were already part of your initial assessment, could you clarify what changed in your judgment of the paper to warrant lowering the rating from Weak Rejection to Clear Rejection? We would appreciate your reconsidering the lower rating in light of the evidence above.
Thank you for highlighting the connection to WINO and for asking how remasking can resolve inconsistencies between tokens generated in parallel.
We distinguish two generations of diffusion language models by how they are trained. Early models receive prediction supervision only at masked positions. Later models trained for token correction also receive prediction supervision at visible positions.
In the first generation, mask-only training applies the prediction loss only at masked positions. These models can output logits at visible positions, but those outputs receive no corresponding supervision and therefore need not provide a meaningful correction signal. Some correction methods for these models already use remasking. To identify which tokens need correction, they must obtain a signal indirectly or learn one through additional training, because they cannot rely directly on visible-token probabilities that have received no such supervision.
In the second generation, visible positions also receive prediction supervision. Applying softmax to the logits at these positions therefore yields prediction confidences grounded in that training. This makes it possible to use the probability assigned to the current visible token directly for detection. The native correction strategy for these models, however, is to edit the current token by replacing it directly with another token proposed by the model.
Our primary innovation is the combination of directly using the probability of the visible token and remasking it (prob + remask). T2M reuses the confidence estimates at visible positions learned during correction training. It uses the probability assigned to the current token to identify positions for correction, remasks those positions, and leaves reconstruction to subsequent denoising steps. This requires no additional training or indirect construction of a detection signal. Earlier remasking methods draw their signal from other sources, while the backbone that provides our signal natively uses it for editing. We pair this signal with remasking and use controlled detector–action comparisons to evaluate how editing and remasking perform with the same detection signal.
(LLaDA, §2.1, LLaDA2.1, §3.1)
WINO and matched-budget comparisons
To check a visible token x, WINO puts a mask at the corresponding position in a shadow block and prevents that position from attending directly to x. It reads the probability of predicting x itself at that mask. Only if this probability is below the verification threshold does WINO turn x back into a mask in the original sequence. T2M reads the probability of x at its original, still-visible position. A latency comparison must include WINO’s shadow verification and T2M’s subsequent denoising.
We agree that we should compare WINO and T2M on the same LLaDA2.1 checkpoint and evaluate T2M against ReMDM-conf, CORE, and T2T under matched inference budgets. We will run these comparisons in follow-up work and report accuracy and inference cost.
Default T2M uses 2.0–3.1× as many forward evaluations as native T2T. Table 3 compares actions under the same detector.
With the T2T trigger the costs are similar. With CORE, remasking improves accuracy and reduces NFE/token at both intervals. Our ReMDM result is for the random variant (91.28 accuracy at 2.88 NFE/token), while LOWPROB with τ=0.9 reaches 91.66 at 0.672 NFE/token.
Parallel inconsistency
Parallel filling is one way a token can lose support as context develops. Suppose a city and a tower name are filled in the same step. Each may be plausible on its own, yet the pair may be inconsistent. Once the city is visible, the model may assign the chosen tower name a probability below the LOWPROB threshold. T2M can then reopen the tower name and predict it again with the city in view. If both tokens receive low support, both can be reopened within the remasking budget. This repeated updating in light of the other positions is the intuition shared with Gibbs sampling. LOWPROB measures support for a token in its current context. The score alone cannot tell us whether parallel filling caused a conflict.
These statistics would directly test our motivation. We did not collect them originally. During rebuttal, we allocated our time and compute to another submission we considered more likely to be accepted and deferred this analysis and the additional baselines. We will analyze generation traces in follow-up work to measure how often tokens filled together conflict and how often T2M resolves or reintroduces those conflicts.