acceptodds
Withdrawn from ICLR 2027

When to Revise: A Scaling Study of Masked Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free remasking overlays let masked diffusion language models (MDMs) return committed tokens to the mask and decode them again. Each overlay is evaluated in its own harness and usually at one operating point, so it is unclear whether a reported gain reflects the method or the step budget, the compute spent, and the sample of benchmark problems. We study this question for four overlays (CoRe, ReMDM-conf, STaRR and T2M) and vanilla decoding on LLaDA-8B-Base with a common-harness protocol. It runs all methods in one decoder at three step budgets, counts each method's forward passes from its implementation, adds equal-cost vanilla controls, and estimates paired uncertainty for the equal-weight benchmark mean. CoRe, the only overlay that spends extra forward passes to verify tokens, gains about two to three points over vanilla at every budget, and the gain does not shrink under equal-cost controls. Its lead over the best of the three compute-free overlays, however, is within benchmark sampling uncertainty at two of three budgets. The overlays differ far more in how many answers they change than in net accuracy, and on a second backbone CoRe's effect on one benchmark reverses sign. Code is available at https://anonymous.4open.science/r/dLMM-benchmark-iclr27-81D2/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Masked diffusion language models, inference-time revision (remasking) overlays, inference-budget scaling, training-free decoding, difficulty-stratified evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_K3iFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Diffusion language models (DLMs) promise various advantages over commonly used autoregressive models, including parallel generation, bidirectional modeling, and self-correction. The most prominent solution is masked DLM. Strictly following its backward formulation would require to replace randomly selected (still masked) positions with tokens sampled from the denoiser's distribution. This is, however, problematic in parallel decoding, as the denoiser models the conditional joint probability as the product of the per-token marginals.

To this end, various position selection heuristics have been proposed, which leverage per-token marginal scores or even cross-token dependency-aware scores (Kim et al., ICML 2026). As an orthogonal direction, various works have started to explore remasking already committed tokens, initially proposed in ReMDM, and revisited by CORE, T2M, and also Llada 2.1 (they have direct token-to-token transitions). Even though remasking diverges even more from the original training objective, it enables some level of self-correction, enabling to fix initial parallel sampling errors due to the modeling approximation (product of marginals).

This empirical study aims to compare different remasking methods at an equal number of function evaluations (NFE), particularly penalizing remasking methods that use additional forward passes for remasking (such as CORE). Main experiments are conducted on Llada-8B across five tasks (math and coding). Results show that CORE outperforms a vanilla confidence-based baseline at equalized NFEs. However, CORE cannot significantly outperform other remasking methods (T2M) across all sampling budgets.

Critical strengths

None

Critical weaknesses

My main reservation with this study is the very limited scope. As an empirical study that exclusively compares existing methods, I would expect much broader evaluations that may give actionable insights for future work. Here are some dimensions I would expect from such a study:

  • Various models, including native token-to-token models (T2T). Evaluating more models is strictly required to show the generality across backbone models. This goes beyond adding full evaluations on Dream. Recent T2T models (e.g., Llada-2.1) should be incorporated into the study too, maybe compared to their M2T predecessors augmented with the considered remasking methods (e.g., adding CORE on Llada-2.0).
  • Comparison to unmasking methods This study should also consider different unmasking methods, ranging from marginal (EB, KLASS) to dependency-aware (DAPD). Considering only simple static baselines (i.e., fixed number of commitments per step) based on a single confidence is not sufficient for a purely empirical study. Maybe, these unmasking methods can even be combined with the considered remasking strategies?

Kim et al., "DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs," ICML 2026.

Questions for authors
  • As mentioned in the paper, CORE did already compare to the vanilla equal-cost baseline. Hence, not much additional insight is gained (despite generalization across budgets). Instead, this paper could compare other remasking methods at equal budget (e.g., running T2M at 68 NFEs instead of 64 in Table 2).
  • Line 366--368: If the McNemar test weights problems across all benchmarks, why not equalizing the test per benchmark for this measure?
  • Figure 2: Section 4.1 concluded that one cannot simply interpolate between sampling budgets to get estimates of the accuracy. Why does Figure 2 do it anyways?
Reviewer_w7zsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Remasking strategies for masked diffusion language models (MDMs) can achieve performance gains without additional training, and many studies have explored these strategies. This paper argues that evaluations in prior work are misleading and statistically flawed. Accordingly, the paper re-evaluates existing remasking methods, including CoRe, ReMDM-conf, STaRR, and T2M, and shows that CoRe achieves the best performance.

Critical strengths

None

Critical weaknesses
  1. The main weakness of this paper is that its contribution is too marginal for acceptance at ICLR.
  • It only evaluates existing methods and does not propose any new ones. The only claim of this paper (including Takeaways 1, 2, and 3) is that CoRe is the best strategy among existing methods. This kind of work could be done for any SOTA paper, which makes me think that the contribution is marginal.
  • The topic itself is too narrow. It only focuses on masked diffusion models, which are just one type of diffusion language model, and remasking strategies, which are just one type of sampling strategy.
  1. The experimental results themselves also do not provide meaningful insights.
  • The equal-compute advantage of CoRe over vanilla has already been demonstrated by the original CoRe authors themselves. This paper primarily extends that evaluation to an additional sampling budget and even acknowledges this.
  1. The writing quality is poor.
  • A three-page introduction in an eight-page paper makes no sense.
  • The PDF title and the OpenReview title are different.
  • The same code URL is repeated three times.
Questions for authors

Please see weaknesses.

Reviewer_pY7KOpenReview ↗
Rating
2
Confidence
4
Informativeness
1(−1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares remasking overlays (i.e., mechanisms that correct or revise previously generated tokens) for masked diffusion language models across different decoding budgets. The authors examine whether improvements in accuracy hold at matched compute and whether the differences between methods are statistically distinguishable. The paper evaluates four existing overlays using LLaDA-8B-Base, reporting paired confidence intervals and counting rescues and breakages. CORE improves on vanilla in both equal-cost comparisons, but its lead over T2M is not statistically resolved at the two lower step budgets. The authors also find that methods with similar net gains differ substantially in how often they correct or introduce errors.

Critical strengths

The paper provides a direct comparison of existing remasking overlays that their separate evaluations do not provide. It provides a holistic assessment regarding both how confidently the methods can be ranked and how their gains divide into rescues (fixing the response) and breakages (making the response incorrect). The paired comparisons show that some apparent advantages are unresolved, while the correctness-flip analysis distinguishes methods with similar net accuracy. Both findings persist on the benchmarks with matched prompts (Tables 2–3).

Critical weaknesses
  • The comparison of CORE, ReMDM-conf, STaRR, and T2M adds limited insight beyond existing evaluations. The paired intervals and correctness-flip counts describe the methods’ differences more precisely, but the main conclusions remain that rankings require uncertainty estimates and similar net gains can involve different numbers of rescues and breakages. Figure 1 also overstates what prior evaluations omit. CORE’s published paper already includes compute-matched comparisons and statistical analyses in Table 5 and Appendices B and E, some of which the text below Figure 1 acknowledges.

  • The Dream results overstate the improvement because HumanEval outputs were post-processed differently for CORE and vanilla. The README acknowledges that only CORE's completions were truncated at a markdown fence. The authors report that consistent scoring reduces the five-benchmark mean gain from 4.41 to 1.48 points, but Tables 1 and 11 still contain the original values. The reported corrected gain remains positive, and this error does not affect the separate LLaDA comparison.

Questions for authors

None,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.