- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an extension to the standard masked diffusion modelling paradigm, in which the model is trained not only to unmask tokens, but also to insert, remove and correct tokens during the generation process. Specifically, the authors build on a method called FlexMDM, which combined insertion with unmasking, and additionally introduce deletion and correction into the decoding process. They argue that this removes ambiguity compared to FlexMDM. The authors describe the deletion and correction mechanisms with a lot of details, and then experimentally demonstrate that the obtained method achieves better generation perplexity at smaller sampling budgets than the baselines, across different datasets.
Critical strengths
- The proposed deletion and correction scheme seem well-designed and (mostly) well-explained, carefully accounting for the changes to the token positions as a result of the insertion and deletion operations. The proposed notation makes the training and generation pipelines easy to follow.
- The experiments demonstrate meaningful performance improvements across OpenWebText and TinyGSM. The authors compare generative perplexity and entropy, and show that MDM performs well on the Pareto-front. I also appreciate that the baselines are trained for twice more optimisation steps, to match the compute used.
- The authors run some of the important ablations, including evaluating the effect of the insertion scale factor, as well as evaluating how often the corrections and deletions happen.
Critical weaknesses
For me, the biggest weakness of the paper is that the current empirical evaluation does not make clear what the sources of gain are. Some of the details of the deletion and correction schemes, as well as some hyperparameters of the baselines, could mean that the reported performance gains are artefacts of simple choices in the generation process, rather than strictly from introducing deletion and correction. If the authors could provide additional evidence and clarifications, which justifies that indeed adding deletion and correction leads to better performance under fair comparisons, I would be happy to increase my score.
- Deletion. Line 305 seems to suggest that the deletion operation is used only for the newly generated mask tokens. Based on this, it seems that tokens which have already been unmasked cannot be deleted? This limits the usefulness of the deletion operation and raises the following questions:
- Is the deletion operation actually helpful? Given the above constraint to newly generated mask tokens, it seems that the deletion mechanism works strictly as a correction to the insertion mechanism utilised in the immediately preceding step of the generation process. However, it is unclear to me whether the deletion mechanism actually uses any additional information which was not available to the insertion mechanism? If not, is it useful at all? Could the authors run an ablation where they run their MDM model with and without the deletions (while defaulting to the FlexMDM’s insertion scheme, which does not use the aggressive insertions). This is to provide some justification for the use of deletion, which is currently missing.
- FlexMDM paper shows that a significant advantage of their model is that they can recover the length distribution of samples from OpenWebText. With the aggressive insertion and deletion operations, does the same hold for MDM? The plots in Figure 14 suggest that the length distribution of MDM is highly dependent on the insertion scale factor, which does not seem like a desirable property.
- Corrections. The correction mechanism introduced in the model always relies on taking the argmax over the vocabulary (l.303). That seems to suggest that the generation just effectively defaults to greedy sampling rather than nucleus sampling, as all non-argmax tokens sampled in the proposal stage can be corrected in the correction stage to the argmax. Is this correct? If yes, this raises several questions:
- If MDM uses greedy sampling, while FlexMDM and MDM use nucleus sampling, are the comparisons fair? Or are the differences in performance largely due to to the use of a different sampling scheme? To measure that, it would be great if the authors could compare the performance of MDM to FlexMDM and MDM when the latter two also use greedy sampling. That would allow to make clear where the performance gains are coming from.
- Comparisons to baselines. The hyperparameters of the baseline generation process are not described in details, making it again quite difficult to understand the sources of gain.
- What are the exact parameters of the MDM generation here? What is the initial number of masks given to the MDM in each of the settings? I can imagine that if MDM does not use block-wise decoding, while also using random unmasking order, then the performance would indeed be quite bad and lead to very favourable comparison for MDM and FlexMDM, where the length of the sequence is increased systematically throughout the generation process. If block-wise decoding is not used, I would strongly encourage the authors to report that comparison.
- Additionally, the performance of standard MDMs can be improved significantly by using adaptive token unmasking (e.g. unmasking the most confident tokens). Similar scheme was also proposed by FlexMDM. However, it is unclear to me how this could be implemented in MDM, which uses aggressive unmasking. If there is no space for adaptive unmasking in MDM, could the authors also report comparison to MDMs and FlexMDMs when the latter two use confidence-based unmasking? To allow for a more fair (optimised) comparison.
Less important but worth addressing for revised manuscript
- Related works: I think a discussion of related works which consider edit-operations (deletion, correction) in masked diffusion models could be strengthened, to include works such as Edit Flows (https://arxiv.org/abs/2506.09018), DiffusER (https://arxiv.org/abs/2210.16886) or ME-DLM (https://arxiv.org/abs/2605.09603), which seem highly relevant.
- Motivation. The main motivation for the paper is resolving the “multiple alignments problem” identified in FlexMDM (l.162). However, it is unclear to me whether the proposed method has managed to actually resolve this problem. First, note that the proposal training part of the algorithm is the same for MDM as for FlexMDM, which suggests that will suffer from largely similar challenges. This is somewhat supported by the plot in figures 7 and 8, where the difference between FlexMDM and MDM seem very small, particularly compared to MDM (the fact that the two plots have different y-axis makes the comparisons quite difficult). Here, the decrease in entropy could potentially be attributed to the use of argmax operations during corrections (see W3). Second, it seems to me that the deletions and corrections can actually introduce more ambiguity into the decoding process (there is multiple different ways in which the final token sequence could be obtained from an intermediate one). Could the authors comment on that?
Questions for authors
- Do the networks and correspond to two different heads of the underlying transformers, which share most of the hyperparameters? The architectural details seem to be missing.
- How is the sampling budget defined exactly?
Minor
- How exactly is deletion instantiated in the model? Section 3.2 indicates that tokens are deleted when the models predicts the deletion token
[DEL], but line 131 suggests that there is no deletion token. I’m assuming that the deletion token is a part of MDM, but not FlexMDM? If yes, it would be great if you could please make that clear. - In line 202, shouldn’t the length of the insertion count vector be as in line 180? Or are we not allowing mask insertions at the very end of the sequence?
- Is there a maximal number of masks that can be inserted in step 2.2 of section 3.2? Or is the allowed sequence length not really bounded?
Thank you for taking the time to review our paper
(1) Could you please clarify what you mean by "actual tasks"? To our best understanding, Generative perplexity and entorpy is are widely used and standard metrics for evaluating generative models at this scale on unconditional generation.
(2) We also train models on TinyGSM and evaluate on GSM8K. The evaluation for this is accuracy and not generative perplexity.
(3) Please note that we are using the same evaluation protocol as was used in both the main baselines in our paper. Human evaluation is expensive and none of the baselines report human evaluation protocols or numbers.
Could you please clarify what you mean by "argmax" inference? Please refer to our response title "Corrections and Comparisons to Baselines" to reviewer d2zs, where we explain that the generation is not purely argmax based.
Could you please clarify what the objective question or concern is in these statements? We would like to be able to understand you concern better in order to address it as best as we can.
Thank you for taking the time to review our paper. We would be grateful if you could provide some objective details in your questions.