Birth, Death and Diff: Insertion-Based Sequence Generation with Corrections

Insertion-based masked diffusion models enable flexible-length, parallel generation by interleaving mask insertion and token unmasking. However, because their generation decisions are irreversible, aggressive parallel decoding can introduce errors that later steps cannot fix, forcing conservative decoding with more forward passes (NFEs). We show that, due to multiple alignments, insertion-based models have higher entropy than MDMs and, without a correction mechanism, require even more conservative decoding. To address this, we introduce δMDM, which makes intermediate generations of an insertion-based model revisable by equipping it with substitution and deletion operations. During training, we expose the model to errors produced by aggressive unmasking and insertions to teach it to correct its own mistakes. At inference time, we interleave insertion and unmasking with correction steps that can modify previously unmasked tokens and delete unnecessary masks. This correction mechanism makes aggressive decoding practical, improving both generation speed and quality. δMDM Pareto-dominates FlexMDM, a vanilla insertion-based model, on the speed–quality trade-off. On high-diversity unconditional text generation on OpenWebText, it halves the generative perplexity relative to FlexMDM at high NFEs. It also achieves lower generative perplexity at all inference budgets compared to MDM trained from scratch with variable-length padded sequences. On low-diversity conditional text generation on TinyGSM, it matches FlexMDM accuracy with half as many NFEs.