- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper’s main point is that existing evaluations of approaches that propose strategies for decoding masked diffusion language models conflate two sources of performance gains. The first is how many tokens to unmask at a given step; the second is how to decide which tokens to unmask at a given step. The main contribution is an evaluation framework to separate the measurement of performance gains. The “substrate” defines the performance gains stemming from confidence-based selection over random token selection when the number of tokens decoded at a given step is kept fixed. The “overlay” is defined as the additional gain stemming from a given method over confidence-based selection. A divergence upper bound is also included to measure the difference between the likelihood of the final tokens being changed compared to what would be selected based on confidence-ranking. Based on the results, the claim is that confidence-based selection is the primary source of performance gains and that benefits from more sophisticated selection criteria are not statistically significant. This leads to the conclusion that deciding the number of tokens to decode at a given step may matter more than the particular strategy for selecting tokens. While the paper’s evaluation framework is useful, I find the empirical validation and scope insufficiently deep. Evaluating a wider range of tasks, datasets, and models is needed to generalise the findings. Especially given that the primary contribution of this work is on the evaluation side. I do not believe the paper has sufficient empirical validation for acceptance.
Critical strengths
- The method is addressing an important problem in the language diffusion area regarding performance evaluation of decoding strategies.
- The paper proposes simple and straightforward metrics to disentangle the performance gains.
Critical weaknesses
-
The scope of the paper is too limited. The paper's conclusions are based on tasks where the order of token unmasking has a limited effect. Sec. 7 itself notes that strongly structured tasks such as Sudoku may behave differently, since the order of unmasking can be important. Hence, the main issue is about whether the findings generalise beyond the evaluated settings. A broader evaluation across diverse models, datasets, and tasks is needed to establish when and to what extent sophisticated selection criteria provide benefits. Without this, the framework has limited practical value.
-
There is insufficient background information on masked diffusion models and how the existing decoding strategies work. This is essential for readers who are from outside this domain because the manuscript must be self-contained.
-
The math notation is overloaded. The symbol t is used for both the diffusion step and the tokens, e.g. t_{<j} in Proposition 2.
-
The code is not available for review, which is necessary for review.
-
Questions for authors
See weaknesses W1-W4