- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
LED adds a draft proposal branch to a masked diffusion model and fine-tunes the model to correct imperfect proposals. At inference time, it refines these proposals over several forward passes before committing tokens. The paper reports gains on eight benchmarks and presents theoretical analyses to motivate the approach.
Critical strengths
-
The method is straightforward, with a clear distinction between revisable drafts and committed tokens.
-
The lookahead and proposal-threshold experiments provide useful information about how the method behaves.
Critical weaknesses
-
The main comparison is unfair because it does not account for inference cost. LED introduces additional forward passes, but the paper reports neither total NFE nor a compute-matched comparison against the corresponding MDM-SFT baseline. The lookahead depth used in the main table is also not explicitly stated.
-
The source of the gains remains unclear. The method combines correction training with extra inference computation. The experiments do not adequately separate their contributions, so the main results cannot establish how much the training method itself helps.
-
The theory does not directly support the proposed method. The circuit analysis compares MDMs with logarithmically many steps against UDMs with polynomially many revisions. This does not isolate the benefit of rewriting, and no corresponding result is established for LED’s limited lookahead.
Questions for authors
Please see the weaknesses section.
6 comments
Part 2
W3. The theory does not directly support the proposed method. The circuit analysis compares MDMs with logarithmically many steps against UDMs with polynomially many revisions. This does not isolate the benefit of rewriting, and no corresponding result is established for LED’s limited lookahead.
-
Isolating the benefit of rewriting: We would like to clarify that even when MDMs are allowed the same forward-pass budget as UDMs, their effective decoding steps remain limited by irreversible token commitment. Specifically, with a bounded block size , each masked position can be committed at most once, allowing only state-changing updates per block. Under the CoT-length setting of Theorem 3.5, a forward-pass budget yields at most effective commitments, keeping MDMs within . Additional forward passes without new writable positions cannot provide further state revisions. In contrast, rewriting allows UDMs to repeatedly update the same workspace across all steps without expanding sequence length.
-
Connection to LED and limited lookahead: Our theory shows that asymptotically scaling iterative state revisions can increase expressivity. It does not establish an expressivity guarantee specifically at . Our empirical validation of the theory is presented in Figure 5 of the main paper, where lookahead depths , which are within the regime, all outperform the setting without lookahead . We did not evaluate larger lookahead steps because already requires 23 hours on MATH500. We select only because it achieves the best performance among the evaluated settings, and we acknowledge that identifying the optimal number of steps within the regime is not solved by our theory and it is an interesting direction for future work.
Kind Reminder
Dear Reviewer,
Thank you again for your efforts and valuable comments in reviewing our paper. We have carefully addressed your comments and concerns in detail in the rebuttal and we hope you might find the response satisfactory. We are very much looking forward to hearing from you about any further feedback. We will be very happy to clarify further concerns (if any).
Best,
Authors.
Part 1
To provide a compute-matched comparison against the MDM-SFT baseline, we report the average number of function evaluations (Avg. NFE) and benchmark scores for the 8B models across different lookahead depths K∈{0,1,2}. As shown in the table below, LED consistently achieves comparable or higher accuracy with comparable or fewer NFEs than MDM-SFT on all datasets. For example, on HumanEval/HumanEval+, LED achieves 82.3%/81.7% with 1009.65 NFEs, outperforming MDM-SFT's 79.9%/74.4% with 1656.53 NFEs. These results demonstrate that LED's performance improvements cannot be solely attributed to additional inference computation.
Regarding the lookahead depth in Table 1, all main results use K=2. We have included this in the revised manuscript.
In Figure 4 in our main paper, we evaluated lookahead depths K∈{0,1,2} and reported the corresponding accuracy–throughput trade-offs with and without lookahead inference. To further separate the two contributions across all eight benchmarks, we provide additional experiments in the table below, comparing MDM-SFT, LED training with standard MDM inference, and LED training with LED inference. Specifically, LED training alone improves the average score from 45.4 to 47.3 (+1.9) for 7B and from 60.8 to 64.0 (+3.2) for 8B. Incorporating LED inference further improves the average score to 56.8 (+9.5) and 68.5 (+4.5), respectively. These results demonstrate that both proposal-conditioned training and iterative lookahead contribute to the overall improvements.
Table: Ablation separating LED training and LED lookahead inference across mathematical, scientific, and code generation benchmarks. Parentheses denote differences relative to the corresponding MDM-SFT baseline.