- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Diffusion language models (DLMs) promise various advantages over commonly used autoregressive models, including parallel generation, bidirectional modeling, and self-correction. The most prominent solution is masked DLM. Strictly following its backward formulation would require to replace randomly selected (still masked) positions with tokens sampled from the denoiser's distribution. This is, however, problematic in parallel decoding, as the denoiser models the conditional joint probability as the product of the per-token marginals.
To this end, various position selection heuristics have been proposed, which leverage per-token marginal scores or even cross-token dependency-aware scores (Kim et al., ICML 2026). As an orthogonal direction, various works have started to explore remasking already committed tokens, initially proposed in ReMDM, and revisited by CORE, T2M, and also Llada 2.1 (they have direct token-to-token transitions). Even though remasking diverges even more from the original training objective, it enables some level of self-correction, enabling to fix initial parallel sampling errors due to the modeling approximation (product of marginals).
This empirical study aims to compare different remasking methods at an equal number of function evaluations (NFE), particularly penalizing remasking methods that use additional forward passes for remasking (such as CORE). Main experiments are conducted on Llada-8B across five tasks (math and coding). Results show that CORE outperforms a vanilla confidence-based baseline at equalized NFEs. However, CORE cannot significantly outperform other remasking methods (T2M) across all sampling budgets.
Critical strengths
None
Critical weaknesses
My main reservation with this study is the very limited scope. As an empirical study that exclusively compares existing methods, I would expect much broader evaluations that may give actionable insights for future work. Here are some dimensions I would expect from such a study:
- Various models, including native token-to-token models (T2T). Evaluating more models is strictly required to show the generality across backbone models. This goes beyond adding full evaluations on Dream. Recent T2T models (e.g., Llada-2.1) should be incorporated into the study too, maybe compared to their M2T predecessors augmented with the considered remasking methods (e.g., adding CORE on Llada-2.0).
- Comparison to unmasking methods This study should also consider different unmasking methods, ranging from marginal (EB, KLASS) to dependency-aware (DAPD). Considering only simple static baselines (i.e., fixed number of commitments per step) based on a single confidence is not sufficient for a purely empirical study. Maybe, these unmasking methods can even be combined with the considered remasking strategies?
Kim et al., "DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs," ICML 2026.
Questions for authors
- As mentioned in the paper, CORE did already compare to the vanilla equal-cost baseline. Hence, not much additional insight is gained (despite generalization across budgets). Instead, this paper could compare other remasking methods at equal budget (e.g., running T2M at 68 NFEs instead of 64 in Table 2).
- Line 366--368: If the McNemar test weights problems across all benchmarks, why not equalizing the test per benchmark for this measure?
- Figure 2: Section 4.1 concluded that one cannot simply interpolate between sampling budgets to get estimates of the accuracy. Why does Figure 2 do it anyways?