acceptodds
Under review as a conference paper at ICLR 2027

Birth, Death and Diff: Insertion-Based Sequence Generation with Corrections

Abstract

Insertion-based masked diffusion models enable flexible-length, parallel generation by interleaving mask insertion and token unmasking. However, because their generation decisions are irreversible, aggressive parallel decoding can introduce errors that later steps cannot fix, forcing conservative decoding with more forward passes (NFEs). We show that, due to multiple alignments, insertion-based models have higher entropy than MDMs and, without a correction mechanism, require even more conservative decoding. To address this, we introduce δMDM, which makes intermediate generations of an insertion-based model revisable by equipping it with substitution and deletion operations. During training, we expose the model to errors produced by aggressive unmasking and insertions to teach it to correct its own mistakes. At inference time, we interleave insertion and unmasking with correction steps that can modify previously unmasked tokens and delete unnecessary masks. This correction mechanism makes aggressive decoding practical, improving both generation speed and quality. δMDM Pareto-dominates FlexMDM, a vanilla insertion-based model, on the speed–quality trade-off. On high-diversity unconditional text generation on OpenWebText, it halves the generative perplexity relative to FlexMDM at high NFEs. It also achieves lower generative perplexity at all inference budgets compared to MDM trained from scratch with variable-length padded sequences. On low-diversity conditional text generation on TinyGSM, it matches FlexMDM accuracy with half as many NFEs.

Primary area:
generative models
Keywords:
Insertion-based generation, Discrete Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d2zsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an extension to the standard masked diffusion modelling paradigm, in which the model is trained not only to unmask tokens, but also to insert, remove and correct tokens during the generation process. Specifically, the authors build on a method called FlexMDM, which combined insertion with unmasking, and additionally introduce deletion and correction into the decoding process. They argue that this removes ambiguity compared to FlexMDM. The authors describe the deletion and correction mechanisms with a lot of details, and then experimentally demonstrate that the obtained method achieves better generation perplexity at smaller sampling budgets than the baselines, across different datasets.

Critical strengths
  1. The proposed deletion and correction scheme seem well-designed and (mostly) well-explained, carefully accounting for the changes to the token positions as a result of the insertion and deletion operations. The proposed notation makes the training and generation pipelines easy to follow.
  2. The experiments demonstrate meaningful performance improvements across OpenWebText and TinyGSM. The authors compare generative perplexity and entropy, and show that δ\deltaMDM performs well on the Pareto-front. I also appreciate that the baselines are trained for twice more optimisation steps, to match the compute used.
  3. The authors run some of the important ablations, including evaluating the effect of the insertion scale factor, as well as evaluating how often the corrections and deletions happen.
Critical weaknesses

For me, the biggest weakness of the paper is that the current empirical evaluation does not make clear what the sources of gain are. Some of the details of the deletion and correction schemes, as well as some hyperparameters of the baselines, could mean that the reported performance gains are artefacts of simple choices in the generation process, rather than strictly from introducing deletion and correction. If the authors could provide additional evidence and clarifications, which justifies that indeed adding deletion and correction leads to better performance under fair comparisons, I would be happy to increase my score.

  1. Deletion. Line 305 seems to suggest that the deletion operation is used only for the newly generated mask tokens. Based on this, it seems that tokens which have already been unmasked cannot be deleted? This limits the usefulness of the deletion operation and raises the following questions:
    1. Is the deletion operation actually helpful? Given the above constraint to newly generated mask tokens, it seems that the deletion mechanism works strictly as a correction to the insertion mechanism utilised in the immediately preceding step of the generation process. However, it is unclear to me whether the deletion mechanism actually uses any additional information which was not available to the insertion mechanism? If not, is it useful at all? Could the authors run an ablation where they run their δ\deltaMDM model with and without the deletions (while defaulting to the FlexMDM’s insertion scheme, which does not use the aggressive insertions). This is to provide some justification for the use of deletion, which is currently missing.
    2. FlexMDM paper shows that a significant advantage of their model is that they can recover the length distribution of samples from OpenWebText. With the aggressive insertion and deletion operations, does the same hold for δ\deltaMDM? The plots in Figure 14 suggest that the length distribution of δ\deltaMDM is highly dependent on the insertion scale factor, which does not seem like a desirable property.
  2. Corrections. The correction mechanism introduced in the model always relies on taking the argmax over the vocabulary (l.303). That seems to suggest that the generation just effectively defaults to greedy sampling rather than nucleus sampling, as all non-argmax tokens sampled in the proposal stage can be corrected in the correction stage to the argmax. Is this correct? If yes, this raises several questions:
    1. If δ\deltaMDM uses greedy sampling, while FlexMDM and MDM use nucleus sampling, are the comparisons fair? Or are the differences in performance largely due to to the use of a different sampling scheme? To measure that, it would be great if the authors could compare the performance of δ\deltaMDM to FlexMDM and MDM when the latter two also use greedy sampling. That would allow to make clear where the performance gains are coming from.
  3. Comparisons to baselines. The hyperparameters of the baseline generation process are not described in details, making it again quite difficult to understand the sources of gain.
    1. What are the exact parameters of the MDM generation here? What is the initial number of masks given to the MDM in each of the settings? I can imagine that if MDM does not use block-wise decoding, while also using random unmasking order, then the performance would indeed be quite bad and lead to very favourable comparison for δ\deltaMDM and FlexMDM, where the length of the sequence is increased systematically throughout the generation process. If block-wise decoding is not used, I would strongly encourage the authors to report that comparison.
    2. Additionally, the performance of standard MDMs can be improved significantly by using adaptive token unmasking (e.g. unmasking the most confident tokens). Similar scheme was also proposed by FlexMDM. However, it is unclear to me how this could be implemented in δ\deltaMDM, which uses aggressive unmasking. If there is no space for adaptive unmasking in δ\deltaMDM, could the authors also report comparison to MDMs and FlexMDMs when the latter two use confidence-based unmasking? To allow for a more fair (optimised) comparison.

Less important but worth addressing for revised manuscript

  1. Related works: I think a discussion of related works which consider edit-operations (deletion, correction) in masked diffusion models could be strengthened, to include works such as Edit Flows (https://arxiv.org/abs/2506.09018), DiffusER (https://arxiv.org/abs/2210.16886) or ME-DLM (https://arxiv.org/abs/2605.09603), which seem highly relevant.
  2. Motivation. The main motivation for the paper is resolving the “multiple alignments problem” identified in FlexMDM (l.162). However, it is unclear to me whether the proposed method has managed to actually resolve this problem. First, note that the proposal training part of the algorithm is the same for δ\deltaMDM as for FlexMDM, which suggests that KθK^\theta will suffer from largely similar challenges. This is somewhat supported by the plot in figures 7 and 8, where the difference between FlexMDM and δ\deltaMDM seem very small, particularly compared to MDM (the fact that the two plots have different y-axis makes the comparisons quite difficult). Here, the decrease in entropy could potentially be attributed to the use of argmax operations during corrections (see W3). Second, it seems to me that the deletions and corrections can actually introduce more ambiguity into the decoding process (there is multiple different ways in which the final token sequence could be obtained from an intermediate one). Could the authors comment on that?
Questions for authors
  1. Do the networks KθK^\theta and gθg^\theta correspond to two different heads of the underlying transformers, which share most of the hyperparameters? The architectural details seem to be missing.
  2. How is the sampling budget TT defined exactly?

Minor

  1. How exactly is deletion instantiated in the model? Section 3.2 indicates that tokens are deleted when the models predicts the deletion token [DEL], but line 131 suggests that there is no deletion token. I’m assuming that the deletion token is a part of δ\deltaMDM, but not FlexMDM? If yes, it would be great if you could please make that clear.
  2. In line 202, shouldn’t the length of the insertion count vector be ℓ(x)+1\ell(x)+1 as in line 180? Or are we not allowing mask insertions at the very end of the sequence?
  3. Is there a maximal number of masks that can be inserted in step 2.2 of section 3.2? Or is the allowed sequence length not really bounded?
Reviewer_MpneOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes δ\deltaMDM, which extends insertion-based masked diffusion models such as FlexMDM with substitution and deletion so that errors from aggressive parallel decoding can be revised. A single time-blind network is trained both to generate and to correct its own aggressive proposals. At inference, proposal and correction steps alternate, and both count toward the sampling budget. With models trained from scratch, the paper finds lower GPT-2-large generative perplexity than FlexMDM on OpenWebText. On TinyGSM-trained code generation evaluated on GSM8K, δ\deltaMDM has higher pass@1 than FlexMDM at every budget. On synthetic star-graph planning, it also shows a large gain over FlexMDM on the hardest split.

Critical strengths
  1. Compared with FlexMDM, where unmasking and insertion decisions are irreversible, δ\deltaMDM lets the same network revise its own intermediate generations through substitution and deletion. Unlike ProSeCo (Schiff et al., 2026), which only substitutes tokens in fixed-length MDMs, δ\deltaMDM can also delete surplus inserted masks, so over-insertion can be undone. It also needs no auxiliary model, in contrast to approaches that learn the unmasking order with a separate variational model. This makes aggressive insertion practical: when corrections start early, GSM8K accuracy keeps improving as the insertion scale λ\lambda increases.

  2. Under matched training and inference compute, this translates to clear empirical gains over FlexMDM. On GSM8K, δ\deltaMDM improves pass@1 at every sampling budget, for example from 31.77% to 42.23% at T=500T=500, and it matches FlexMDM’s accuracy with about half the steps. On the hardest star-graph split, exact match rises from 7.27% to 96.47%. On OpenWebText, it roughly halves generative perplexity at the highest budget and remains better at matched entropy.

References

Yair Schiff, Omer Belhasin, Roy Uziel, Guanghan Wang, Marianne Arriola, Gilad Turok, Michael Elad, and Volodymyr Kuleshov. Learn from Your Mistakes: Self-Correcting Masked Diffusion Models. In International Conference on Machine Learning (ICML), 2026.

Critical weaknesses
  1. Edit Flows (Havasi et al., 2025), a discrete flow model that generates variable-length sequences through insertion, deletion, and substitution, is not discussed, although the combination of birth, death, and diff is presented as the core of the method. δ\deltaMDM differs in that its corrections are trained on its own proposals, while Edit Flows trains on edit paths sampled from alignments between source and target sequences. The paper should make this distinction explicit, and a comparison with Edit Flows would show whether this form of correction offers an advantage over modeling edits directly. Earlier edit-based non-autoregressive models such as the Levenshtein Transformer (Gu et al., 2019) are also relevant background.

  2. The paper does not isolate which part of the correction mechanism drives the gains over FlexMDM. These gains combine substitution, for which self-correction has been studied in fixed-length MDMs such as GIDD (von Rütte et al., 2025) and ProSeCo (Schiff et al., 2026), with the deletion of surplus inserted masks, which is specific to the insertion setting. Without a substitution-only variant, it remains unclear how much deletion contributes.

References

Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein Transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2019.

Marton Havasi, Brian Karrer, Itai Gat, and Ricky T. Q. Chen. Edit Flows: Variable Length Discrete Flow Matching with Sequence-Level Edit Operations. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

Yair Schiff, Omer Belhasin, Roy Uziel, Guanghan Wang, Marianne Arriola, Gilad Turok, Michael Elad, and Volodymyr Kuleshov. Learn from Your Mistakes: Self-Correcting Masked Diffusion Models. In International Conference on Machine Learning (ICML), 2026.

Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized Interpolating Discrete Diffusion. In International Conference on Machine Learning (ICML), 2025.

Questions for authors
  1. Could the authors add a discussion of Edit Flows (Havasi et al., 2025) and, if feasible, compare with it on one of the tasks, to show whether training corrections on the model’s own proposals offers an advantage over modeling edits directly?

  2. How does a substitution-only variant compare with δ\deltaMDM on GSM8K under the same training and inference budget, even if that budget is reduced (e.g., both fine-tuned from the FlexMDM checkpoint)? The variant would be trained without the deletion loss (e.g., with insertions capped at the true gap length) and have its insertion scale tuned separately. Disabling deletion only at inference would not answer this, since the model is trained to map surplus masks to [DEL].

I would be glad to raise my score if the substitution-only comparison shows that deletion accounts for a meaningful part of the gain over FlexMDM, and if the relation to Edit Flows is clarified.

Reviewer_wNoHOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces δMDM, a mask diffusion language model which may edit previously unmasked position, as well as add and delete positions in the sequence, while being less prone to a "multiple alignment problem" that may affect other models with position insertion capabilities. Trainings are done from scratch at the GPT-2 Large scale using a four-terms training loss for four different mechanisms: unmasking mask tokens; substituting (or leaving unaltered) already unmasked positions; inserting new positions (initially containing mask tokens); and deleting positions. Generation proceeds by alternating proposal and correction steps. Evaluations are done on generative perplexity and token entropy.

Critical strengths

A visible effort is made to communicate the training and inference algorithms.

Critical weaknesses
  • Evaluations are performed solely on generative perplexity and entropy: no actual tasks (except Appendix C.1, which is a quite toy-ish case), no human evaluations.
  • The model systematically has has lower entropy than FlexMDM; not doing argmax-based inference would likely increase entropy, but at the cost of some generative perplexity.
  • There is no block decoding, which would minimally improve the baselines.

"Edit Flows: Variable Length Discrete Flow Matching with Sequence-Level Edit Operations." (Havasi et al., NeurIPS 2025) is not an MDM, but otherwise has insertion, deletion and edits (it can do image captions, but does not need to). LLaDA2.2 is a proper MDM with all four of δMDM's operations; it should probably be considered concurrent work, but it gives an idea of what δMDM is competing with.

Questions for authors

Do you disagree with my listed critical weaknesses?

3 comments
AuthorsOpenReview ↗

Thank you for taking the time to review our paper

Evaluations are performed solely on generative perplexity and entropy: no actual tasks (except Appendix C.1, which is a quite toy-ish case), no human evaluations.

(1) Could you please clarify what you mean by "actual tasks"? To our best understanding, Generative perplexity and entorpy is are widely used and standard metrics for evaluating generative models at this scale on unconditional generation.

(2) We also train models on TinyGSM and evaluate on GSM8K. The evaluation for this is accuracy and not generative perplexity.

(3) Please note that we are using the same evaluation protocol as was used in both the main baselines in our paper. Human evaluation is expensive and none of the baselines report human evaluation protocols or numbers.

not doing argmax-based inference would likely increase entropy

Could you please clarify what you mean by "argmax" inference? Please refer to our response title "Corrections and Comparisons to Baselines" to reviewer d2zs, where we explain that the generation is not purely argmax based.

Editflows...(it can do image captions, but does not need to) LLADA 2.2 .... it gives an idea of what δMDM is competing with

Could you please clarify what the objective question or concern is in these statements? We would like to be able to understand you concern better in order to address it as best as we can.

Thank you for taking the time to review our paper. We would be grateful if you could provide some objective details in your questions.

AuthorsOpenReview ↗

Request for clarification

We have tried our best to understand your critical concerns and want to follow up on our previously posted clarification questions. We would greatly appreciate any kind of response.

(1) Could you please clarify what you mean by "actual tasks"?

By "actual task" I mean a downstream task that has some inherent value for business/human use, or some ersatz task for which we have reasons to believe that doing good on that ersatz has a genuine positive correlation with business/human use.

To our best understanding, Generative perplexity and entorpy is are widely used and standard metrics for evaluating generative models at this scale on unconditional generation.

They are widely used, and it is true that at that scale it may be difficult to do actual downstream tasks. But perplexity has historically had issues (Zheng et al. 2025), and doing better on the perplexity-entropy Pareto front often doesn't correlate with a human's perception of "good" generation (my personal experience, best reference I found is Franca and Tong 2026). As of 2026, the standard is to have downstream tasks, even for small models.

(2) We also train models on TinyGSM and evaluate on GSM8K. The evaluation for this is accuracy and not generative perplexity.

GSM8K qualifies as an "actual task": there was a delay between my careful reading of the manuscript and my production of the review, and I must have gotten confused in my notes. I am sincerely sorry for this confusion. I am raising my score to 2, and I decrease my confidence to 3.

(3) Please note that we are using the same evaluation protocol as was used in both the main baselines in our paper. Human evaluation is expensive and none of the baselines report human evaluation protocols or numbers.

You may provide sample outputs in appendix for the different model/baselines, and the readers can judge for themselves. I find that conditional generation is often more instructive for this type of experiment.

Could you please clarify what you mean by "argmax" inference? Please refer to our response title "Corrections and Comparisons to Baselines" to reviewer d2zs, where we explain that the generation is not purely argmax based.

I read the mentioned discussion. I understand that newly unmasked tokens are sampled using SampleUnmask, which is nucleus sampling for OWT and greedy (argmax) for GSM8K. I also understand that the baselines use the same SampleUnmask, so in that respect the comparisons are fair.

However, when apply_correction is true, the positions that are already unmasked in xtx_t go through two steps of argmax before making their way into xt+τx_{t+\tau}: first on line 2 of algorithm 1, then on line 3 of algorithm 2. Taking argmax of a distribution conditioned on the argmax of a distribution is a deterministic function, and deterministic functions may only decrease, or leave unaffected, the entropy of a random variable.

Editflows...(it can do image captions, but does not need to) LLADA 2.2 .... it gives an idea of what δMDM is competing with

Could you please clarify what the objective question or concern is in these statements? We would like to be able to understand you concern better in order to address it as best as we can.

In the first part, I was pre-empting the "Although EditFlow looks similar to δ\deltaMDM..." reply you gave to reviewer Mpne. Yes, I agree that δ\deltaMDM is different from EditFlow, but I deem EditFlow needs to be cited (which I see that you now do at the end of section 5) and, like reviewer Mpne, I think that EditFlow should have been among the evaluated baselines.

Then I said that LLaDA2.2 "should probably be considered concurrent work", because of the short time elapsed between its release and the submission deadline for ICLR: I do not consider LLaDA2.2 as prior work for the purpose of the present submission. Then when I said "but it gives an idea of what δ\deltaMDM is competing with", I insinuated that LLaDA2.2 was superior to δ\deltaMDM, a higher standard of overall quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.