acceptodds
Under review as a conference paper at ICLR 2027

Beyond Replacement: Levenshtein Editing for Diffusion Language Models

Abstract

Diffusion language models (DLMs) offer a flexible alternative to autoregressive models, enabling any-order infilling without specialized prompting. While most existing decoding methods treat decoded tokens as fixed, some recent models allow previously decoded tokens to be revised only by replacing them with alternative tokens, a paradigm we refer to as *replacement-only editing* (ROE). Although ROE can correct individual tokens, it remains inefficient for sequence shifts caused by missing or extra tokens and struggles when errors occur across many positions. To address these limitations, we propose LED (**L**evenshtein **E**diting for **D**iffusion Language Models), a post-training framework that extends ROE with insertion and deletion, enabling Levenshtein-style refinement of generated sequences. It uses two key training designs. First, it reconstructs edit supervision on the fly during supervised fine-tuning using longest-common-subsequence alignment, enabling the model to learn insertion and deletion operations without manually annotated labels. Second, it adopts multi-stage training, where the state from one stage becomes the input to the next, exposing the model to its own earlier errors and enabling it to learn how to correct them. When applied to the 16B-parameter LLaDA2.1-Mini Base, LED raises the average score across twelve benchmarks from to under confidence-guided decoding while increasing tokens per forward from to , whereas under confidence-free decoding it raises the average score from to . We further show that, in this setting, LED achieves higher pass@ at large than thresholded decoding without editing, demonstrating improved generation diversity and revealing the diversity potential of DLMs without confidence guidance. LED also improves robustness to large block sizes, raising the average score from to at block size . Together, these results suggest that LED supports more robust self-correction in DLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, sequence editing, self-correction, Levenshtein editing, non-autoregressive generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_F7HwOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a post-training framework for llada 2.1 to enable it to perform Levenshtein editing, including operations like insertion and deletion. It has two key designs: the first is to use the longest common substring (LCS) algorithm to construct the label for edit supervision; the second is the multi-stage training to let the model revise its own errors. Experiments demonstrate the performance and efficiency superiority over llada 2.1 on standard benchmarks.

Critical strengths
  1. This paper proposes an innovative solution to enable Levenshtein editing in DLMs, based on LCS labeling and multi-stage training.

  2. Compared to llada 2.1, which is one of the most advanced SOTA DLMs, LED improves the generation accuracy and speed on various language benchmarks.

Critical weaknesses

This paper has substantial overlap with llada 2.2, which is a concurrent TMLR submission (https://openreview.net/forum?id=kTa5ZayvmJ), the shared mechanisms include deletion and insertion control tokens, insertion of a mask before the current token, fixed-length restoration through padding or truncation, LCS-derived supervision assigning at most one insertion per gap to the following matched anchor, and multi-stage training to revise the model-generated errors. See ICLR §§3.1–3.2 and Algorithms 1–2, and TMLR §3.1. The mechanisms are central to both papers, and neither supplied manuscript explicitly acknowledges the other.

This raises concerns about potential dual submission or unattributed reuse.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer F7Hw: Relationship with LLaDA2.2

Thank you for pointing out the overlap with [LLaDA2.2] (https://openreview.net/forum?id=kTa5ZayvmJ). We acknowledge that LLaDA2.2 directly incorporates the LED algorithm studied in this submission and that the two manuscripts are under review concurrently. We agree that this relationship should have been made explicit. We will cite LLaDA2.2 and clarify the shared mechanism and the contributions of each paper in the revised manuscript.

Although the core editing mechanism is shared, the manuscripts address different research questions and provide distinct contributions:

  1. Research objectives. LLaDA2.2 describes the development and evaluation of an agentic diffusion model. Our submission studies LED as an editing algorithm, focusing on its motivation, design, and ability to address sequence misalignment that replacement-only editing struggles to correct.

  2. Methodological specification. LLaDA2.2 describes the insertion/deletion interface and LCS-based supervision. Our submission explicitly specifies training trajectory construction, mixed teacher forcing, the stage-wise loss, and cycle detection and resampling (§3.2;§3.3;§3.4; Algorithms 2–4). These procedures are not detailed in the LLaDA2.2 report.

  3. Additional empirical findings. On a 16B model, we study controlled omission and duplication repair (Fig. 1; App. A.2), confidence-free decoding and pass@k solution coverage (§4.3–4.4; Figs. 3–4), and robustness when decoding blocks exceed the training block size (§4.6; Table 2). These analyses complement LLaDA2.2’s model-level benchmarks and editing ablation by examining LED’s behavior under controlled error and decoding settings.

We believe these focused methodological and empirical contributions make the manuscripts sufficiently distinct that their concurrent submission does not constitute prohibited dual submission.

Reviewer_QLqGOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Based on LLADA 2.1, a prominent large-scale DLM realization, this paper argues the replacementonly editing (ROE) equipped by LLADA 2.1 exhibits two failure modes that limit its editing capability, including i) ROE cannot efficiently correct sequence misalignment, which arises when token omission or duplication shifts the remaining suffix relative to the target sequence. ii) ROE remains ineffective when errors occupy a large portion of the sequence. To this end, this paper post-trains LLaDA2.1-Mini Base and this paper call LED (Levenshtein Editing for Diffusion Language Models). LED involves insertion and deletion by introducing a special output token for each operation.

Critical strengths
  • This paper is well-written and targets an interesting problem, i.e., how to integrate the insertion and deletion ability into the existing DLMs.

  • The experimental results are very strong and convincing.

  • I like this paper and it is very clear.

Critical weaknesses
  • I like this paper becuase of its good presentation and obvious efforts. However, this paper has substantial overlap with LLaDA 2.2 (https://openreview.net/pdf?id=kTa5ZayvmJ). Its technical report already describes insertion/deletion tokens, LCS-derived supervision, one insertion per gap per round, and fixed-block padding/truncation. Therefore, the novelty of this paper is insufficient. I thanks the authors' efforts, but I am not sure if this an overlapped paper can be accepted by ICLR (this may be selected by the AC). I will keep a relatively low score and wait the discussion phase.

  • The ablation study is not completed and should be enhanced. For example, the components such the insertion, deletion, LCS supervision can be isolated and report the performance individually.

  • For effeiciency, tokens per forward does not establish end-to-end speedup, especially with additional editing rounds, sequence shifts, and cycle-handling overhead.

Questions for authors

please see the weakness.

1 comment

Thank you for your thoughtful review and for recognizing the paper's clarity and experimental results. We address your three concerns below.

1. Relationship to LLaDA2.2. The two submissions pursue different research questions. Our paper studies LED as an editing algorithm and examines its behavior under controlled error and decoding settings, while LLaDA2.2 develops and evaluates an agentic diffusion model. They share the core editing mechanism, which we will acknowledge and cite explicitly in the revision. Please see our response to F7Hw for a detailed comparison of the contributions.

2. Component ablation. Because one LED post-training run costs 3.7k H200 GPU-hours (§4.1), we could not complete additional 16B training runs to separate insertion, deletion, and LCS supervision during the rebuttal. To assess the edit actions, we instead disabled them at inference while holding the LED checkpoint, mask threshold (0.5), and generation budget (32k tokens) fixed. In the table, Re, I, and D denote replacement, insertion, and deletion. All scores are percentages, and the Re+I+D column contains the original w/ LED results reported in Table 1.

BenchmarkReRe+IRe+DRe+I+D
MMLU-Pro47.0652.3246.2455.33
GPQA-D39.4342.0838.7040.91
KOR-Bench38.9640.4039.5244.00
GSM+75.9279.3078.0083.06
OlympiadBench52.8151.2250.4454.11
AIME2516.6717.1424.3824.01
HumanEval+52.4453.8156.6379.19
MBPP+46.3052.8151.8870.90
LiveCodeBench v69.696.177.6524.34
Coding mean (3)36.1437.6038.7258.14
BFCL v359.1760.0861.9264.85
IFEval61.0062.2957.4968.39
C-Eval71.6774.1575.3076.15
12-task mean47.5949.3149.0157.10

Relative to Re alone, Re+I and Re+D raise the 12-task mean by 1.72 and 1.42 points, respectively. The gap between full LED and the restricted settings is largest on coding tasks, whose mean is 58.14 with all actions and at most 38.72 with a restricted action set. This pattern is consistent with prior work showing that code generation depends on preserving syntax and data-flow structure (StructCoder) and that mismatched fixed lengths can impair code infilling in diffusion models (DreamOn). Overall, these results provide evidence for the importance of both edit actions in LED.

3. End-to-end inference time. To assess inference speed beyond tokens per forward, we analyzed the per-request timing records from the existing 12-benchmark evaluations without rerunning inference. We define effective output TPS as total generated output tokens divided by summed per-request times. Timing runs from a request's entry into the inference engine to its final output. It includes scheduling and queueing, prefill, decoding and editing rounds with their sequence shifts and cycle handling, and detokenization. Model loading is excluded. Both settings used NVIDIA H20-3e GPUs.

Benchmarkw/o LEDw/ LED
MMLU-Pro123.5138.0
GPQA-D114.4138.7
KOR-Bench166.5190.3
GSM+112.6113.4
OlympiadBench137.1130.7
AIME25123.0123.6
HumanEval+176.6158.6
MBPP+163.3178.6
LiveCodeBench v6185.1207.0
BFCL-Live29.518.7
IFEval128.5155.1
C-Eval161.9209.1
Pooled TPS127.5139.3

The pooled rate uses all requests rather than an average of benchmark rates. LED has a higher observed TPS on 9 of the 12 benchmarks, with lower rates on OlympiadBench, HumanEval+, and BFCL-Live. Overall, LED achieves higher observed effective TPS than the replacement-only (ROE) baseline, indicating a speed advantage under this evaluation setup.

Reviewer_zyvtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that replacement-only self-correction in block-diffusion language models corrects sequence misalignment only inefficiently. It proposes LED, which adds deletion and insertion tokens, derives their labels from standard SFT data through LCS alignment, and uses multi-stage training. LED is applied to a 16B model and compared with a replacement-only baseline on twelve benchmarks under several decoding settings.

Critical strengths
  1. Under aggressive decoding, LED improves every benchmark and raises the average from 27.52 to 47.19. It also degrades much less at block size 128, which shows much better recovery from corrupted drafts, although both models collapse at block size 256.
  2. The design is simple and scales to 16B. Two extra tokens keep the fixed-block interface, and a controlled probe shows that LLaDA-LED repairs single-token shifts far more often than the replacement-only baseline.
Critical weaknesses
  1. LED introduces insertion and deletion together with multi-stage training, and no ablation separates the two. The baseline is also described inconsistently across Section 4.1, Appendix A.2 and Appendix E, so it is unclear whether the benchmark gains in Table 1, Figure 3 and Table 2 come from the new edit actions or from the training pipeline. The central "beyond replacement" claim is therefore not isolated.
  2. Under the default decoding setting, the gains are mixed. All three math benchmarks decline, most of the average gain comes from two benchmarks, and all results come from a single greedy run without uncertainty estimates.
Questions for authors

How exactly is the baseline trained? How does LED compare with a replacement-only model trained on the same multi-stage trajectories? Please also report item-level bootstrap confidence intervals or paired tests for Table 1.

1 comment

Thank you for the careful review and for highlighting the need to clarify the baseline and quantify uncertainty. We address these points with an action ablation and item-level bootstrap estimates.

Matched baseline. For Table 1, Fig. 3, and Table 2, both models start from LLaDA2.1-Mini Base and use the same 745k SFT examples, four-stage training scheme, teacher-forcing ratio (0.5), and optimization schedule. Both are trained on multi-stage trajectories. The baseline learns replacements only. LED adds [Ins]/[Del] with LCS-derived edit labels. The main comparison therefore controls for the use of multi-stage training. We will make this matched training setup explicit in the revised manuscript and clarify the baseline descriptions.

Action ablation. Our compute budget during rebuttal did not permit an additional 16B training run since the LED run required 3.7k H200 GPU-hours (§4.1). Therefore, we disabled edit actions at inference on the same LED checkpoint, with threshold 0.5 and a 32k token budget. The full LED score is from Table 1:

Enabled actionsReplacement onlyReplacement + insertionReplacement + deletionReplacement + insertion + deletion (LED; Table 1)
12-task mean47.5949.3149.0157.10

Enabling insertion or deletion alone raises the mean over replacement only by 1.72 or 1.42 points, respectively. The per-benchmark scores for all four test-time action settings are reported in §2 (“Component ablation”) of our response to Reviewer QLqG.

Item-level uncertainty. Using the saved item-level results underlying Table 1, we drew 10,000 bootstrap resamples per model. The table reports observed scores and 95% bootstrap confidence intervals (CIs), defined by the 2.5th and 97.5th percentiles of the resampled scores. Δ denotes the score difference in percentage points.

Taskw/o LED: score [95% CI]w/ LED: score [95% CI]Δ
MMLU-Pro53.70 [52.74, 54.63]55.33 [54.40, 56.27]+1.63
GPQA Diamond39.65 [34.60, 44.70]40.91 [36.46, 45.49]+1.26
KOR-Bench32.88 [30.72, 34.96]44.00 [41.92, 46.16]+11.12
GSM+83.57 [82.80, 84.33]83.06 [82.29, 83.82]−0.51
OlympiadBench55.85 [52.15, 59.56]54.11 [50.63, 57.52]−1.74
AIME2530.00 [13.33, 46.67]24.01 [10.99, 38.39]−5.99
HumanEval+76.22 [69.51, 82.93]79.19 [72.79, 85.14]+2.97
MBPP+65.08 [60.05, 69.84]70.90 [66.47, 75.46]+5.82
LiveCodeBench v613.88 [10.96, 16.96]24.34 [20.48, 28.19]+10.46
BFCL v365.39 [63.48, 67.35]64.85 [62.91, 66.79]−0.54
IFEval69.69 [65.80, 73.57]68.39 [64.51, 72.27]−1.30
C-Eval72.36 [69.88, 74.77]76.15 [73.85, 78.43]+3.79
12-task mean54.86 [53.18, 56.54]57.10 [55.62, 58.67]+2.24

Taken together with the matched baseline and action ablation, these results support our conclusion that LED improves over ROE training method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.