acceptodds
Under review as a conference paper at ICLR 2027

LED: Improving Masked Diffusion Language Models by Lookahead Editing

Abstract

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive language models, enabling non-autoregressive text generation through parallel decoding. Existing dLLMs can be broadly categorized into two families: masked diffusion models (MDMs) and uniform diffusion models (UDMs). In this work, we theoretically show that MDMs are easier to optimize due to their mask-based transition schedule but have lower computational expressivity, whereas UDMs enable iterative state revision and greater expressivity but are substantially harder to train. Motivated by this analysis, we propose Lookahead Editing (), which augments masked diffusion decoding with a rewritable proposal state. At each decoding step, the model performs several lookahead updates to iteratively refine future predictions in this proposal state before committing tokens to the masked sequence. This design introduces a rewritable proposal for exploring and refining future predictions while preserving the simple mask-based transition schedule of MDMs. We fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-7B and LED-8B, respectively. Across eight benchmarks, LED SFT yields average gains of and points over standard MDM SFT. Moreover, using SFT alone, LED-8B outperforms LLaDA2.1-16B Q-mode, which is trained with continued pre-training, supervised fine-tuning, and reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eGUbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LED adds a draft proposal branch to a masked diffusion model and fine-tunes the model to correct imperfect proposals. At inference time, it refines these proposals over several forward passes before committing tokens. The paper reports gains on eight benchmarks and presents theoretical analyses to motivate the approach.

Critical strengths
  1. The method is straightforward, with a clear distinction between revisable drafts and committed tokens.

  2. The lookahead and proposal-threshold experiments provide useful information about how the method behaves.

Critical weaknesses
  1. The main comparison is unfair because it does not account for inference cost. LED introduces additional forward passes, but the paper reports neither total NFE nor a compute-matched comparison against the corresponding MDM-SFT baseline. The lookahead depth used in the main table is also not explicitly stated.

  2. The source of the gains remains unclear. The method combines correction training with extra inference computation. The experiments do not adequately separate their contributions, so the main results cannot establish how much the training method itself helps.

  3. The theory does not directly support the proposed method. The circuit analysis compares MDMs with logarithmically many steps against UDMs with polynomially many revisions. This does not isolate the benefit of rewriting, and no corresponding result is established for LED’s limited lookahead.

Questions for authors

Please see the weaknesses section.

6 comments

W1. The main comparison is unfair because it does not account for inference cost. LED introduces additional forward passes, but the paper reports neither total NFE nor a compute-matched comparison against the corresponding MDM-SFT baseline. The lookahead depth used in the main table is also not explicitly stated.

To provide a compute-matched comparison against the MDM-SFT baseline, we report the average number of function evaluations (Avg. NFE) and benchmark scores for the 8B models across different lookahead depths K∈{0,1,2}K \in \lbrace 0, 1, 2 \rbrace. As shown in the table below, LED consistently achieves comparable or higher accuracy with comparable or fewer NFEs than MDM-SFT on all datasets. For example, on HumanEval/HumanEval+, LED achieves 82.3%/81.7% with 1009.65 NFEs, outperforming MDM-SFT's 79.9%/74.4% with 1656.53 NFEs. These results demonstrate that LED's performance improvements cannot be solely attributed to additional inference computation.

Regarding the lookahead depth in Table 1, all main results use K=2K=2. We have included this in the revised manuscript.

BenchmarkMethodAvg. NFEAccuracy (%)
HumanEval / HumanEval+MDM-SFT1656.5379.9 / 74.4
LED (K=0K=0)1009.6582.3 / 81.7
LED (K=1K=1)1927.8885.3 / 82.3
LED (K=2K=2)3007.6587.2 / 84.1
MATH500MDM-SFT4060.9588.8
LED (K=0K=0)4050.7488.8
LED (K=1K=1)9104.5390.2
LED (K=2K=2)13746.0492.6
AIME 2024MDM-SFT9322.4736.7
LED (K=0K=0)4429.9040.0
LED (K=1K=1)8010.2046.7
LED (K=2K=2)11691.3750.0
AIME 2025MDM-SFT9286.5723.3
LED (K=0K=0)4076.5333.3
LED (K=1K=1)7426.1336.7
LED (K=2K=2)11520.8340.0
GPQA DiamondMDM-SFT9132.7346.5
LED (K=0K=0)5819.7445.9
LED (K=1K=1)9219.7451.2
LED (K=2K=2)13951.4753.6
MBPP / MBPP+MDM-SFT1507.8674.6 / 61.9
LED (K=0K=0)942.9375.4 / 64.6
LED (K=1K=1)1817.7875.4 / 63.8
LED (K=2K=2)2737.4475.4 / 65.1

W2. The source of the gains remains unclear. The method combines correction training with extra inference computation. The experiments do not adequately separate their contributions, so the main results cannot establish how much the training method itself helps.

In Figure 4 in our main paper, we evaluated lookahead depths K∈{0,1,2}K \in \lbrace 0, 1, 2 \rbrace and reported the corresponding accuracy–throughput trade-offs with and without lookahead inference. To further separate the two contributions across all eight benchmarks, we provide additional experiments in the table below, comparing MDM-SFT, LED training with standard MDM inference, and LED training with LED inference. Specifically, LED training alone improves the average score from 45.445.4 to 47.347.3 (+1.9+1.9) for 7B and from 60.860.8 to 64.064.0 (+3.2+3.2) for 8B. Incorporating LED inference further improves the average score to 56.856.8 (+9.5+9.5) and 68.568.5 (+4.5+4.5), respectively. These results demonstrate that both proposal-conditioned training and iterative lookahead contribute to the overall improvements.

MethodSizeMATH500AIME24AIME25GPQAHumanEvalHumanEval+MBPPMBPP+Avg.
MDM-SFT7B72.013.313.322.762.859.864.055.045.4
LED training7B80.8 (+8.8)16.7 (+3.4)23.3 (+10.0)27.2 (+4.5)60.6 (-2.2)57.6 (-2.2)60.6 (-3.4)51.3 (-3.7)47.3 (+1.9)
LED training + LED inference7B86.0 (+14.0)26.7 (+13.3)23.3 (+10.0)33.8 (+11.1)75.0 (+12.2)70.7 (+10.9)74.6 (+10.6)64.0 (+9.0)56.8 (+11.4)
MDM-SFT8B88.836.723.346.579.974.474.661.960.8
LED training8B88.840.0 (+3.3)33.3 (+10.0)45.9 (-0.6)82.3 (+2.4)81.7 (+7.3)75.4 (+0.8)64.6 (+2.7)64.0 (+3.2)
LED training + LED inference8B92.6 (+3.8)50.0 (+13.3)40.0 (+16.7)53.6 (+7.1)87.2 (+7.3)84.1 (+9.7)75.4 (+0.8)65.1 (+3.2)68.5 (+7.7)

Table: Ablation separating LED training and LED lookahead inference across mathematical, scientific, and code generation benchmarks. Parentheses denote differences relative to the corresponding MDM-SFT baseline.

W3. The theory does not directly support the proposed method. The circuit analysis compares MDMs with logarithmically many steps against UDMs with polynomially many revisions. This does not isolate the benefit of rewriting, and no corresponding result is established for LED’s limited lookahead.

  1. Isolating the benefit of rewriting: We would like to clarify that even when MDMs are allowed the same poly(n)\mathrm{poly}(n) forward-pass budget as UDMs, their effective decoding steps remain limited by irreversible token commitment. Specifically, with a bounded block size O(1)O(1), each masked position can be committed at most once, allowing only O(1)O(1) state-changing updates per block. Under the O(log⁡n)O(\log n) CoT-length setting of Theorem 3.5, a poly(n)\mathrm{poly}(n) forward-pass budget yields at most O(log⁡n)O(\log n) effective commitments, keeping MDMs within TC0\mathsf{TC}^0. Additional forward passes without new writable positions cannot provide further state revisions. In contrast, rewriting allows UDMs to repeatedly update the same O(1)O(1) workspace across all poly(n)\mathrm{poly}(n) steps without expanding sequence length.

  2. Connection to LED and limited lookahead: Our theory shows that asymptotically scaling iterative state revisions can increase expressivity. It does not establish an expressivity guarantee specifically at K=2K=2. Our empirical validation of the theory is presented in Figure 5 of the main paper, where lookahead depths K=1,2,3,4K = 1,2,3,4, which are within the O(log⁡n)O(\log n) regime, all outperform the setting without lookahead K=0K=0. We did not evaluate larger lookahead steps because K=4K=4 already requires 23 hours on MATH500. We select K=2K=2 only because it achieves the best performance among the evaluated settings, and we acknowledge that identifying the optimal number of steps within the O(log⁡n)O(\log n) regime is not solved by our theory and it is an interesting direction for future work.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

Thank you again for your efforts and valuable comments in reviewing our paper. We have carefully addressed your comments and concerns in detail in the rebuttal and we hope you might find the response satisfactory. We are very much looking forward to hearing from you about any further feedback. We will be very happy to clarify further concerns (if any).

Best,

Authors.

Reviewer_PjZPOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript propose Lookahead Editing (LED), an hybrid between masked diffusion models (MDMs) and uniform diffusion models (UDMs). All discussions assume block decoding. LED treats clean past blocks normally. However, every token of the block being actively decoded is associated with two different transformer positions. One copy behaves essentially like a standard MDM for both training and inference. The other copy is inspired by UDM, with the requirement that its token value must match the first's when the corresponding first copy's token is unmasked. During training, these masked positions are filled in the second copy by calling a standard MDM. During inference, what the model predicted in the last round is fed as input in the next round; multiple such steps of look-ahead may be performed before a token is actually unmasked in the first (MDM) copy. Through different theorems, the manuscript claims that these changes confer LED a different (better) computational expressivity class than MDM and autoregressive language models. Experiments on mathematical reasoning, scientific reasoning and code generation benchmark are favorable to LED. Similar favorable results are obtained for speed-accuracy trade-offs.

Critical strengths

This is a refreshingly clean/simple way to hybridize MDMs and UDMs. LED is basically a stateful MDM. Indeed, one could write a wrapper interface around LED and MDM that allows for a "scratch space": LED could use that space to track its proposal branch's predictions across rounds, and the MDM would just ignore it. Both could share the same interface and schedule.

The computational expressivity theorems are a nice touch, but I'm not well placed to assess their practical implications. I didn't check any of these theorems.

Critical weaknesses

While it is my understanding that LED itself is novel, I believe that some of the (implicit or explicit) claims of novelty/primacy may be overstated. This is the topic of my main question below. I'm not sure if the claims do have to be toned down; I'm ready to hear the author's perspective. In any case, this is fixable by addressing prior work and/or toning down.

A quick search revealed that, although TC0≠NC1TC^0 \neq NC^1 is a widely held belief, it is still an open problem. The manuscript implicitly takes the separation for granted; I think it should make it explicit. (Allender and Koucký. Amplifying Lower Bounds by Means of Self-Reducibility. ACM 2010 https://dl.acm.org/doi/pdf/10.1145/1706591.1706594) I would also have welcomed concrete examples of what kind of operations the different classes can efficiently compute. Again, fixable.

Questions for authors

Main question: Are there diffusion models that already do NC1NC^1?

My understanding is that the current manuscripts is, in effect, tracking (token, is_mask) for each position. When is_mask is true, the tokens are those of the proposal branch; when it is false, both branches agree on the token value and this value won't change anymore during inference.

There is a whole literature of papers doing "remasking", with or without training involved. Many of those amount to predicting (token, p_mask), using p_mask to determine whether to remask token in the next step. Here p_mask is discarded after each forward: does this suffice for NC1NC^1?

Then there are models that pass the full (token, p_mask) information across rounds. DMax (https://arxiv.org/pdf/2604.08302) does it by interpolating (1-p_mask)*token+p_mask*mask). To me, this looks as expressive as LED, if not more. What do you think?

The authors need not provide a definitive answer: LED's simplicity/"neatness" may play an important role in its algorithmic tractability. But in any case, I believe that this connection to the literature should be tightened.

Reviewer's curiosity, comments, typos, etc.

  1. Do the two branches' logits tend to agree? Have you looked at the DKLD_{KL} between them? Have you tried to include that divergence as a loss term? Does it makes things worse or better?
  2. The two branches are just concatenation of their corresponding blocks, with no special treatment of positional encoding? If yes, does that degrade the model reacts to a change in inference-time block size? (I know the manuscript says that all models were trained with block size 32, but standard block diffusion models can cope with some inference-time variability.)
  3. Figure 3, missing word(s) after "Orange arrow denotes".
4 comments

W1: Some of the (implicit or explicit) claims of novelty/primacy may be overstated. Are there diffusion models that already can do NC¹?

We would like to clarify that we do not claim LED is the first or only diffusion model capable of NC1\mathsf{NC}^1 computation. In fact, Theorem 3.6 of our paper explicitly proves that existing UDMs already support NC1\mathsf{NC}^1 computation via iterative state revision.

However, as shown in Theorems 3.1–3.3, enabling vocabulary-level rewriting directly on the main diffusion trajectory introduces denoiser-dependent transition schedules (Lwhen(θ)>0L_{\mathrm{when}}(\theta) > 0) and additional optimization difficulty. This motivates our proposed LED, which preserves the standard MDM transition schedule while introducing a rewritable proposal branch to enable iterative token correction.

Therefore, our contribution is not claiming to be the first diffusion model to achieve NC1\mathsf{NC}^1 expressivity, but rather: (i) formally characterizing the optimization–expressivity trade-off between MDMs and UDMs, and (ii) bridging these two properties through a decoupled lookahead proposal mechanism.

W2.1: A quick search revealed that, although TC0≠NC1TC^0 \neq NC^1 is a widely held belief, it is still an open problem. The manuscript implicitly takes the separation for granted; I think it should make it explicit.

We thank the reviewer for pointing this out. In the revised manuscript, we have explicitly added the standard assumption TC0⊊NC1\mathsf{TC}^0 \subsetneq \mathsf{NC}^1.

W2.2: I would also have welcomed concrete examples of what kind of operations the different classes can efficiently compute. Again, fixable.

TC0\mathsf{TC}^0 can efficiently compute constant-depth threshold and counting operations such as the Majority problem (as well as parity and integer addition). For example, given a binary sequence x=(1,0,1,1,0)x=(1,0,1,1,0), determining whether more than half of the bits are 11 yields MAJ(x)=1\mathrm{MAJ}(x)=1.

NC1\mathsf{NC}^1 further supports logarithmic-depth hierarchical computations, such as the NC1\mathsf{NC}^1-complete Boolean Formula Evaluation problem. For example, given a nested formula F=(a∧b)∨(c∧(d∨e))F=(a\land b)\lor(c\land(d\lor e)) with (a,b,c,d,e)=(1,0,1,0,1)(a,b,c,d,e)=(1,0,1,0,1), evaluating it yields F=1F=1. We have added these concrete examples to Section 2.2 of the revised manuscript.

Q2. There is a whole literature of papers doing "remasking", with or without training involved. Many of those amount to predicting (token, p_mask), using p_mask to determine whether to remask token in the next step. Here p_mask is discarded after each forward: does this suffice for NC1NC^1? Then there are models that pass the full (token, p_mask) information across rounds. DMax does it by interpolating (1-p_mask)*token+p_mask*mask). To me, this looks as expressive as LED, if not more. What do you think?

  1. Whether a method supports NC1\mathsf{NC}^1 depends on whether it maintains a rewritable state and permits poly(n)\mathrm{poly}(n) revisions. Under poly(n)\mathrm{poly}(n) steps, DMax can also support NC1\mathsf{NC}^1 computation. We would like to clarify that our contribution is not claiming to be the first/only diffusion model to achieve NC1\mathsf{NC}^1 expressivity, but (i) formally characterizing the optimization–expressivity trade-off between MDMs and UDMs, and (ii) bridging these two properties through a decoupled lookahead proposal mechanism.

  2. Regarding token embedding interpolation, DMax represents intermediate states by interpolating between predicted token embeddings and the mask embedding. However, since the vocabulary size (typically over 150K tokens in the Qwen series) is much larger than the embedding dimension (typically 1K–4K in the Qwen series), such interpolation may introduce representation ambiguity, where different token predictions and interpolation weights can produce the same representations.

To empirically evaluate discrete dual-branch editing against token-embedding interpolation, we conduct a controlled experiment from the same Qwen2.5-1.5B initialization comparing MDM SFT, DiffusionGemma SFT (which similarly uses weighted interpolation of token embeddings to incorporate previous predictions), and LED.

As shown in the table below, DiffusionGemma SFT improves over MDM SFT on GSM8K (68.0→69.268.0 \to 69.2) and MATH500 (36.4→39.036.4 \to 39.0), whereas LED SFT already achieves 70.670.6 and 42.342.3, and adding LED lookahead inference further improves accuracy to 71.571.5 and 44.644.6. These results confirm that LED's discrete proposal branch is more effective in practice than embedding interpolation.

MethodGSM8KMATH500
MDM SFT68.036.4
DiffusionGemma SFT69.239.0
LED SFT + MDM Inference70.642.3
LED SFT + LED Inference71.544.6

Comment 1: Do the two branches' logits tend to agree? Have you looked at the DKLD_{\mathrm{KL}} between them? Have you tried to include that divergence as a loss term? Does it make things worse or better?

The two branches are not constrained to produce identical predictions, as they serve different roles in LED: the proposal branch maintains rewritable candidate tokens, while the masked branch predicts tokens for commitment and corrects erroneous proposals.

Specifically, consider a problem from the AIME24 benchmark (finding the remainder when ∏k=012(2−2ωk+ω2k)\prod_{k=0}^{12}(2 - 2\omega^k + \omega^{2k}) is divided by 10001000, where ω≠1\omega \neq 1 is a 1313-th root of unity).

In an intermediate complex-multiplication step evaluating (−65−64i)(−65+64i)=4225+4096=8321(-65-64i)(-65+64i)=4225+4096=8321, the proposal branch erroneously predicts "8821", whereas the masked branch corrects this error and predicts "8321". This real trace from AIME24 illustrates that the two branches do not always agree, allowing the masked branch to revise inaccurate proposal predictions prior to commitment.

Regarding DKLD _ {\mathrm{KL}}, we intentionally do not include DKLD _ {\mathrm{KL}} as an auxiliary loss. LED is trained on proposal sequences X^0\hat{X} _ {0} that contain erroneous tokens, and the objective of the masked branch is precisely to identify and correct those errors toward the ground-truth X0X _ {0}. Penalizing DKLD _ {\mathrm{KL}} between the two branches would force the masked branch to match the proposal branch even on erroneous positions (such as 8821 above), which directly conflicts with error correction.

Comment 2: The two branches are just concatenation of their corresponding blocks, with no special treatment of positional encoding? If yes, does that degrade the model's performance when the inference-time block size changes? (I know the manuscript says that all models were trained with block size 32, but standard block diffusion models can cope with some inference-time variability.)

Regarding positional encoding, corresponding token positions in the masked and proposal branches share the exact same positional encodings. Because this token-level alignment is independent of the block size BB, varying BB at inference time does not break the positional correspondence between masked tokens and their proposals.

To empirically evaluate LED's sensitivity to inference-time block size, we test LED-8B (trained exclusively with B=32B=32) under B=16B=16 and B=64B=64, and compare them against the B=32B=32 results from Table 1. For B=16B=16 and B=64B=64, we use K=2K=2 with proposal thresholds τ=0.8\tau=0.8 on math/GPQA and τ=0.7\tau=0.7 on code benchmarks.

As shown in the table below, when scaling up to a larger block size of B=64B=64, LED-8B achieves an average score of 65.065.0 (vs. 68.568.5 at B=32B=32), which still outperforms the matched-B=32B=32 MDM-SFT-8B baseline (60.860.8) by +4.2+4.2 points and even surpasses B=32B=32 on MBPP (81.581.5 vs. 75.475.4) and MBPP+ (69.369.3 vs. 65.165.1). In contrast, halving the block size to B=16B=16 reduces the average score to 51.951.9. This asymmetric behavior is consistent with LED's lookahead mechanism: since lookahead refinement operates within the active block, shrinking BB to 1616 truncates the lookahead horizon available for previewing and coordinating future tokens, whereas B=64B=64 preserves a sufficient lookahead context and generalizes well with modest degradation.

BenchmarkB=16B=32B=64
MATH50082.892.691.0
AIME2426.750.036.7
AIME2533.340.033.3
GPQA46.053.647.0
HumanEval53.787.283.5
HumanEval+50.084.178.0
MBPP65.675.481.5
MBPP+56.965.169.3
Avg.51.968.565.0

Comment 3: Figure 3, missing word(s) after "Orange arrow denotes".

We thank the reviewer for pointing out this typo and have corrected it in the revised manuscript.

Thank you for your reply.

We would like to clarify that we do not claim LED is the first or only diffusion model capable of NC1\mathsf{NC}^1 computation.

With the understanding that this gets clearly stated in the text, together with the already added comment on the TC0⊊NC1\mathsf{TC}^0 \subsetneq \mathsf{NC}^1 assumption, then I support accepting this contribution.

we intentionally do not include DKLD_{\textup{KL}} as an auxiliary loss.

Your motivations to not use it as a loss make sense, but it could still be informative to look at it as a probe. For example, are the DKLD_{\textup{KL}} statistics of successful tasks significantly different from failed ones? Are mismatches indicator of when more/less compute should be spent in the decoding? I'm not asking you to do it before the end of the discussion period; I'm encouraging you to look into it in the future.

Comment 2: The two branches are just concatenation of their corresponding blocks, with no special treatment of positional encoding? [...]

Regarding positional encoding, corresponding token positions in the masked and proposal branches share the exact same positional encodings.

So I think that the answer to my question is "No, this is not just concatenation." By "just concatenation", I meant (writing "C" a clean context token, "M" a mask-or-unmasked token, and "U" an initially-uniformly-sampled token, here using a block size 3):

C C C M M M U U U
0 1 2 3 4 5 6 7 8

Here "just concatenating" gives different positional encodings to the three M tokens and the three U tokens. Had this been what you were doing, then I would predict performance degradations to a change in block size.

But my understanding from your answer is that what you're doing instead is to "reset" positional encoding of the U tokens as if the M tokens were not there:

C C C M M M U U U
0 1 2 3 4 5 3 4 5

I don't know how to call this, but it's not just concatenating. In any case, please make sure to make the positional encoding matters clear in an eventual camera ready.

Thank you very much for your question and for supporting our work.

DKLD _ {\mathrm{KL}} as an inference-time probe:

Thank you for this insightful suggestion. Using inter-branch DKLD _ {\mathrm{KL}} as a probe is a very promising direction that we look forward to exploring in future work.

Positional encoding:

Yes, your understanding is correct. The positional indexing scheme we use is to "reset" positional encoding:

C C C M M M U U U
0 1 2 3 4 5 3 4 5

We will make sure to explain this positional encoding design clearly in the final version.

Thank you again for your careful review and positive score!

Best,

Authors

Reviewer_5xQAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LED adds an editable draft to masked diffusion language models. The model refines this draft before permanently committing tokens to the output. The paper motivates this design through a theoretical comparison of masked and uniform diffusion. The paper shows comparable practical results.

Critical strengths
  1. Substantial and consistent empirical gains. LED improves all eight benchmark metrics over the corresponding MDM-SFT baselines at both model sizes

  2. A concrete mechanism for refinement before commitment. LED separates an editable draft from the permanently committed output and trains the model using imperfect, model-generated proposals. This directly addresses premature token commitment while retaining a standard cross-entropy objective, making the approach a potentially useful extension to masked diffusion.

Critical weaknesses
  1. Limited conceptual novelty relative to prior work. The central motivation is already well established. GIDD [1] and ReMDM [2] address irreversible token decisions through iterative correction, while AUDM [3] develop masked-style reformulations of uniform diffusion. The paper does not sufficiently clarify what new theoretical insight its analysis adds beyond these established ideas.

  2. Unclear relevance of the circuit-complexity analysis. The comparison allows logarithmically many steps for MDMs but polynomially many revisions for UDMs, so it does not isolate the benefit of rewriting under comparable computational budgets. Furthermore, the connection between these asymptotic results and LED’s practical, fixed-depth lookahead is not established. It is therefore unclear how this analysis justifies the particular proposed design.

  3. Missing theoretical bridge to LED itself. The theoretical results characterize the baseline model families, but do not establish that LED inherits the expressivity attributed to UDMs while retaining the claimed optimization advantages of MDMs. Since this combination is central to the paper’s motivation, the absence of a corresponding result for LED leaves a substantial gap between the diagnosed limitations and the proposed solution.

  4. The source of the empirical gains is insufficiently isolated. Although the authors report common Qwen initialization and training settings, the experiments do not disentangle the contributions of teacher-generated proposal quality, proposal-conditioned training, and additional inference computation. Please report the full training budget, including the proposal generator, and provide matched-budget controls and ablations of proposal quality and conditioning. Also the training stage is inconsistent with inference, during the training authors give the clean sequence as first argument to the model, during the inference only prompt. It is inclear why it even should work.

  5. The relationship to self-conditioning is unclear. Feeding an estimated clean sequence back into the denoiser is an established self-conditioning strategy, including in [5]. LED follows a related principle through its discrete proposal branch and repeated refinement. The authors should explain what the proposed mechanism adds beyond self-conditioning and compare against a suitably adapted self-conditioned MDM under a matched computational budget.

  6. Inconsistencies in the algorithm description hinder reproducibility. In Algorithm 2, lines 241–242, the first model output is assigned to $ LC^{t1} $, but the next line uses the undefined $ LO^{t0} $. In §4.2, lines 294–296, the text should distinguish the one-time fully masked initialization from subsequent decoding steps. Lines 304–306 also use $ X^{t-1} $ as an input, whereas Algorithm 2 uses $ X^t $ to produce $ X^{t-1} $. Please correct these inconsistencies, unify indexing across the pseudocode, text, and Figure 3, and explicitly explain how the packed clean branch during training corresponds to the growing context during inference. The sampler’s interface should also specify how already committed tokens are preserved.

[1] GIDD: https://arxiv.org/abs/2503.04482

[2] ReMDM: https://arxiv.org/abs/2503.00307

[3] AUDM: https://arxiv.org/abs/2605.22765

[4] https://arxiv.org/abs/2506.08316

[5] https://arxiv.org/abs/2211.04236

Questions for authors
  1. Clarification of novelty. What is the precise contribution beyond the correction mechanisms, specifically in comparison with existing self-conditioning approaches? Which theoretical result or algorithmic component constitutes the main advance?

  2. Clarification of the theoretical scope. Do any of the theoretical results establish an expressivity or optimization advantage for LED itself under its practical lookahead budget?

  3. Request for evidence isolating the gains. Can you specify the initialization and complete training pipeline for both LED and its MDM baseline, including the proposal generator’s training cost? Do you have matched-budget controls that separate the effects of proposal quality, proposal-conditioned training, and additional refinement passes? A comparison with a simple self-conditioned MDM would materially help assess whether LED’s specific mechanism is necessary.

  4. Clarification of the executable decoding procedure. Could you provide a corrected Algorithm 2 consistent with §4.2 and Figure 3? Please also specify how the sampler preserves previously committed tokens.

10 comments

W1: Limited conceptual novelty relative to prior work. The central motivation is already well established. GIDD and ReMDM address irreversible token decisions through iterative correction, while AUDM develop masked-style reformulations of uniform diffusion. The paper does not sufficiently clarify what new theoretical insight its analysis adds beyond these established ideas.

We would like to clarify that the novelty of our method is providing a framework that combines the optimization advantages of MDMs with the expressive capabilities enabled by rewriting in UDMs. While GIDD and AUDM introduce mechanisms for token correction, they differ fundamentally from our approach in how rewriting is incorporated into the diffusion process. In particular, both methods involve uniform transition kernels, which do not eliminate the denoiser-dependent schedule-matching loss Lwhen\mathcal{L}_{\mathrm{when}}, as characterized in our Theorem 3.2.

GIDD. GIDD incorporates uniform noise into masked diffusion: qt(xt∣x0)=(1−t)δx0+tδm+ctU1+ctq_t(x_t\mid x_0)=\frac{(1-t)\delta_{x_0}+t\delta_m+c_t U}{1+c_t}, where UU denotes the uniform distribution over a vocabulary of size VV. For any ordinary token aa, the reverse jump rate is λθ(a,t)=ρt1−qt(a∣fθ)qt(a∣fθ)\lambda_\theta(a,t)=\rho_t\frac{1-q_t(a\mid f_\theta)}{q_t(a\mid f_\theta)}, where ρt=ct′+ct/(1−t)V(1+ct)\rho_t=\frac{c_t'+c_t/(1-t)}{V(1+c_t)} and qt(a∣fθ)=(1−t)fθ(a∣a,t)+ct/V1+ctq_t(a\mid f_\theta)=\frac{(1-t)f_\theta(a\mid a,t)+c_t/V}{1+c_t}. For ct>0c_t>0 and ρt>0\rho_t>0, choosing fθ1(a∣a,t)=1f_{\theta_1}(a\mid a,t)=1 and fθ2(a∣a,t)=0f_{\theta_2}(a\mid a,t)=0 yields different reverse jump rates. Applying these choices across ordinary tokens while keeping the denoisers identical at the mask state leads to different distributions over subsequent transition times: qθ1(S)≠qθ2(S)q_{\theta_1}(S)\neq q_{\theta_2}(S). Since the forward schedule distribution p(S)p(S) is fixed, it cannot match both distributions. Therefore, LwhenGIDD(θ)≢0\mathcal{L}_{\mathrm{when}}^{\mathrm{GIDD}}(\theta)\not\equiv 0.

AUDM. AUDM reformulates uniform diffusion using a random absorbing token U∼Uniform(V)U\sim\mathrm{Uniform}(\mathcal{V}): qt(xt∣x0,U)=αtδx0+(1−αt)δUq_t(x_t\mid x_0,U)=\alpha_t\delta_{x_0}+(1-\alpha_t)\delta_U. Unlike standard masked diffusion, the absorbing token may coincide with the clean token. Assuming α1=0\alpha_1=0, the probability of observing no state transitions under the forward process is p(S=∅)=Pr⁡(U=X0)=1/Vp(S=\varnothing)=\Pr(U=X_0)=1/V. Now consider a denoiser that always predicts the absorbing token: fθ(X0=U∣xt=U,t,U)=1f_\theta(X_0=U\mid x_t=U,t,U)=1. Under the corresponding AUDM reverse bridge, this denoiser produces no observable state changes, yielding qθ(S=∅)=1≠1/V=p(S=∅)q_\theta(S=\varnothing)=1\neq 1/V=p(S=\varnothing). Consequently, qθ(S)≠p(S)q_\theta(S)\neq p(S), and hence LwhenAUDM(θ)≢0\mathcal{L}_{\mathrm{when}}^{\mathrm{AUDM}}(\theta)\not\equiv 0.

Distinction from LED. The analyses above show that although GIDD and AUDM introduce mechanisms related to token correction, their reverse processes do not structurally guarantee a denoiser-independent distribution of observable transition times. In contrast, LED preserves the original masking kernel of MDMs, for which Lwhen=0\mathcal{L}_{\mathrm{when}}=0 holds independently of the denoiser, while introducing rewriting through a separate proposal branch (see W3 for details). Thus, LED enables iterative correction without modifying the underlying mask-based transition schedule.

Since GIDD and AUDM are evaluated on small-scale models and cannot provide meaningful comparisons, subsequent works, such as LLaDA 2.1 and XDLM, explore related ideas at larger scales, and we have already included both methods in our experimental comparisons. Moreover, by incorporating uniform transition kernels, these approaches do not inherently eliminate the denoiser-dependent schedule-matching loss, consistent with the optimization challenges identified for UDMs in Theorem 3.2. Our experimental results show that LED outperforms both LLaDA 2.1 and XDLM, providing empirical evidence for the effectiveness of preserving the mask-based transition schedule while enabling iterative token correction.

Regarding ReMDM, it is a training-free method. We evaluate its effectiveness on LLaDA across four benchmarks and find that it degrades performance on three of them. As shown in the table below, it reduces accuracy on MATH, HumanEval, and MBPP by 0.4, 1.1, and 1.8 points, respectively.

DatasetLLaDALLaDA + ReMDMΔ\Delta
GSM8K79.380.4+1.1
MATH39.439.0-0.4
HumanEval46.345.2-1.1
MBPP40.138.3-1.8

W2.1. The comparison allows logarithmically many steps for MDMs but polynomially many revisions for UDMs, so it does not isolate the benefit of rewriting under comparable computational budgets.

We would like to clarify that even when MDMs are allowed the same poly(n)\mathrm{poly}(n) forward-pass budget as UDMs, their effective decoding steps remain limited by irreversible token commitment. Specifically, with a bounded block size O(1)O(1), each masked position can be committed at most once, allowing only O(1)O(1) state-changing updates per block. Under the O(log⁡n)O(\log n) CoT-length setting of Theorem 3.5, a poly(n)\mathrm{poly}(n) forward-pass budget yields at most O(log⁡n)O(\log n) effective commitments, keeping MDMs within TC0\mathsf{TC}^0. Additional forward passes without new writable positions cannot provide further state revisions. In contrast, rewriting allows UDMs to repeatedly update the same O(1)O(1) workspace across all poly(n)\mathrm{poly}(n) steps without expanding sequence length.

W2.2: Furthermore, the connection between these asymptotic results and LED’s practical, fixed-depth lookahead is not established.

We would like to clarify that Theorem 3.6 establishes a worst-case sufficient condition for simulating arbitrary NC1\mathsf{NC}^1 circuits, rather than implying that poly(n)\mathrm{poly}(n) steps are necessary for practical tasks. While LED's formulation supports arbitrary lookahead depth KK, real-world token generation within each local block rarely requires worst-case poly(n)\mathrm{poly}(n) revisions. Therefore, using a small fixed KK is a practical accuracy–throughput choice: as shown in Figure 4, increasing the lookahead steps KK from 00 to 11 and 22 consistently improves performance.

W3 & Q2. Missing theoretical bridge to LED itself. The theoretical results characterize the baseline model families, but do not establish that LED inherits the expressivity attributed to UDMs while retaining the claimed optimization advantages of MDMs. Since this combination is central to the paper’s motivation, the absence of a corresponding result for LED leaves a substantial gap between the diagnosed limitations and the proposed solution.

We would like to clarify that LED retains the optimization advantages of MDMs while introducing the rewritable workspace mechanism of UDMs.

1. Optimization. LED preserves the standard MDM reverse transition schedule. For any 0≤s<t≤10\leq s<t\leq 1, conditioning the denoiser on the proposal CtC^t gives qθLED(Xs=m∣Xt=m,Ct)=∑x0fθ(x0∣Xt,Ct,t)1−αs1−αt=1−αs1−αtq_\theta^{\mathrm{LED}}(X_s=m\mid X_t=m,C^t)=\sum_{x_0}f_\theta(x_0\mid X_t,C^t,t)\frac{1-\alpha_s}{1-\alpha_t}=\frac{1-\alpha_s}{1-\alpha_t}, where the last equality follows from ∑x0fθ(x0∣Xt,Ct,t)=1\sum_{x_0}f_\theta(x_0\mid X_t,C^t,t)=1. Therefore, qθLED(S)=p(S)q_\theta^{\mathrm{LED}}(S)=p(S) and LwhenLED=0\mathcal{L}_{\mathrm{when}}^{\mathrm{LED}}=0, as in Theorem 3.1.

2. Expressivity. Theorem 3.6 establishes that UDMs can support NC1NC^1 computation through polynomially many revisions of a constant-size workspace. Following its proof in Appendix E, by Barrington's theorem, any f∈NC1f\in NC^1 can be represented by a polynomial-length width-5 branching program with instructions Ij=(ij,πj0,πj1)I_j=(i_j,\pi_j^0,\pi_j^1). Let σj∈S5\sigma_j\in S_5 denote the accumulated permutation after jj instructions, defined by σ0=e\sigma_0=e and σj=σj−1∘πjxij\sigma_j=\sigma_{j-1}\circ\pi_j^{x_{i_j}}, where ee is the identity permutation. LED can maintain this state in its proposal branch, C(j)=⟨σj⟩C^{(j)}=\langle\sigma_j\rangle, while keeping the corresponding masked position uncommitted. Using the same attention and feed-forward construction as in Appendix E, together with its step-dependent instruction encoding, each refinement implements the required state update. Thus, the NC1NC^1 construction extends to LED when polynomially many proposal updates and the required step-dependent control are allowed.

W4 & Q3. The source of the empirical gains is insufficiently isolated. Although the authors report common Qwen initialization and training settings, the experiments do not disentangle the contributions of teacher-generated proposal quality, proposal-conditioned training, and additional inference computation. Please report the full training budget, including the proposal generator, and provide matched-budget controls and ablations of proposal quality and conditioning. Also the training stage is inconsistent with inference, during the training authors give the clean sequence as first argument to the model, during the inference only prompt. It is inclear why it even should work.

Regarding the insufficiently isolated gains of proposal-conditioned training and additional inference computation, we would like to clarify that this ablation is already provided in our experiments. As shown in Fig. 4, we evaluate 0, 1, 2 lookahead depths and report the corresponding accuracy–throughput trade-offs, demonstrating the benefits w/ and w/o lookahead inference. To further address your concern, we provide full evaluation in the table below, comparing MDM-SFT, LED training with standard MDM inference, and LED training with LED inference. Specifically, LED training alone improves the average accuracy from 45.4 to 47.3 (+1.9) for 7B and from 60.8 to 64.0 (+3.2) for 8B. Incorporating LED inference further improves the accuracy to 56.8 (+9.5) and 68.5 (+4.5), respectively. These results demonstrate that both proposal-conditioned training and iterative lookahead contribute to the overall improvements.

Regarding different proposal generators, our objective is to improve the performance of the corresponding MDM-SFT (proposal generator) baselines, and we have already validated our approach using both the 7B and 8B MDM-SFT models as proposal generators in Table 1 in our main paper. The consistent improvements across both model scales demonstrate the effectiveness of LED with different MDM-SFT proposal generators.

Regarding the training budget, we train both MDM-SFT models using 512 NVIDIA A100 GPUs for 72 hours and select the 30-hour checkpoint to train LED for an additional 24 hours. Therefore, the effective training duration along the LED training trajectory is 54 hours (30 + 24), which is less than the 72 hours for MDM-SFT.

Regarding the training–inference inconsistency, we would like to clarify that this follows the standard block diffusion training paradigm adopted by BDLM [1], Fast-dLLM v2 [2], SDAR [3], and Nemotron [4]. As illustrated by the attention mask in Figure 1 in our main paper, each masked–proposal block can only attend to preceding clean blocks and cannot access the ground-truth tokens of the current or future blocks. During inference, these preceding clean blocks correspond to previously generated blocks in the prompt. Therefore, as in [1–4], our training and inference procedures are consistent.

MethodSizeMATH500AIME24AIME25GPQAHumanEvalHumanEval+MBPPMBPP+Avg.
MDM-SFT7B72.013.313.322.762.859.864.055.045.4
LED training7B80.8 (+8.8)16.7 (+3.4)23.3 (+10.0)27.2 (+4.5)60.6 (-2.2)57.6 (-2.2)60.6 (-3.4)51.3 (-3.7)47.3
LED training + LED inference7B86.0 (+14.0)26.7 (+13.3)23.3 (+10.0)33.8 (+11.1)75.0 (+12.2)70.7 (+10.9)74.6 (+10.6)64.0 (+9.0)56.8
MDM-SFT8B88.836.723.346.579.974.474.661.960.8
LED training8B88.840.0 (+3.3)33.3 (+10.0)45.9 (-0.6)82.3 (+2.4)81.7 (+7.3)75.4 (+0.8)64.6 (+2.7)64.0
LED training + LED inference8B92.6 (+3.8)50.0 (+13.3)40.0 (+16.7)53.6 (+7.1)87.2 (+7.3)84.1 (+9.7)75.4 (+0.8)65.1 (+3.2)68.5

[1] Arriola et al., Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, ICLR 2025.

[2] Wu et al., Fast-dLLM v2: Efficient Block-Diffusion LLM, ICLR 2026.

[3] Cheng et al., SDAR: A Synergistic Diffusion-Autoregression Paradigm for Scalable Sequence Generation, ACL Findings 2026.

[4] Fu et al., Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding, EMNLP 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.