- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(−1)
Read review
Paper summary
The paper empirically analyzes the Multiple Block Editing (MBE) capabilities of a recent diffusion language model LLaDA 2.1 after a full draft has been generated via block-diffusion. Its findings can be summarized into three major points: (1) global refinement after block-diffusion improves accuracy, (2) spending less compute on drafting and more compute on global refinement is beneficial, (3) drafting does not need to use the same model as refinement.
Critical strengths
None
Critical weaknesses
There is little novelty in the analysis. There is no theory or methodological advancements, and the empirical analysis is not particularly informative either. In particular:
- MBE exists and already has been analyzed even in the technical report of the LLaDA 2.1 model to which the paper restricts itself to. ReMDM (Wang et al. 2025) also already considered a draft-then-refine strategy (see Section 4.2) which acts globally.
- The experiments are not extensive and only consider a single model family. There are no uniform diffusion models, flow language models, etc., yet all of the same-model draft+refine experiments could be conducted also on those.
- Competing methods are dismissed either as contemporary or neglected entirely. E.g., there is no comparison against prior methods on speculative decoding in DLMs nor against speculative decoding with autoregressive models.
- The two-model approach for drafting and refinement requires 2 GPUs rather than 1 because both models have to be stored in VRAM.
Wang, G., Schiff, Y., Sahoo, S., & Kuleshov, V. (2025). Remasking discrete diffusion models with inference-time scaling. Advances in Neural Information Processing Systems, 38, 147282-147339.
Questions for authors
-
You write in related works: "LLaDA2.1 introduces token-to-token editing and describes Multiple Block Editing (MBE), which revisits earlier blocks during generation (Bie et al., 2026)". So what exactly is the novelty of this paper? Just applying MBE after drafting rather than at intermediate steps? That would be very incremental.
-
In the introduction you write: "Our work is closely aligned with concurrent work on Diffusion in Diffusion (Ma et al., 2026), which also identifies draft-then-refine as a promising structure for block diffusion models.". This paper is already 8 months old. Please correct your claims across the manuscript on concurrency with prior art.
-
Speculative correction in DLMs should be compared with both speculative decoding on diffusion language models. E.g., you cite Gao et al., 2025; Han et al., 2026; Cui et al., 2026 in the related work but do not compare against any of them. None of these are contemporary.
-
You should also compare against speculative decoding of standard AR models, e.g., using DFlash2 on a similarly-sized AR model with credible inference backends such as sglang or vllm. Currently the paper is missing any absolute tokens/second numbers, which makes it unclear how competitive the proposed method actually is. Please provide them.
5 comments
2. Model-family scope. We agree that our evaluation is limited to LLaDA2.1. Our plug-and-play approach requires a block-autoregressive drafting path and a bidirectional interface capable of editing existing tokens. Small-to-large correction additionally requires different checkpoint sizes and compatible tokenization. These requirements motivated our choice of LLaDA2.1. Suitable open-source options remain limited.
3. Speculative-decoding comparisons. Despite the name, our method is not a substitute for speculative verification. Classical exact speculative decoding accelerates a target model while preserving its distribution. Our method deliberately changes a completed draft, returns the edited output directly, and makes no losslessness or distribution-preservation claim. We acknowledge that some DLM speculative methods also trade quality for speed.
Comparisons to DFlash2 on a different AR model and serving backend would test broader system competitiveness, not isolate whether post-draft correction helps the same pretrained DLM. Thus, standalone Mini and Flash are our primary controls, with other speculative methods discussed in Related Work. We acknowledge that such comparisons would be useful for assessing broader systems competitiveness, and do not claim superiority over AR speculative-decoding systems.
Our results are not simply accuracy gained by spending extra computation. New matched measurements on all 95 held-out GSM8K problems show Flash→Flash improving accuracy from 78.9% to 91.6%, with a 1.16× speedup. In a matched 100-problem MATH-384 comparison, Mini→Flash achieves 31% at 19.39 s, compared with 29% at 34.64 s for a faster standalone Flash alternative. The latency ratio is 1.79×. That being said We do not claim globally exhaustive quality–latency frontiers.
4. Absolute throughput and deployment. We now explicitly report absolute throughput from the completed joint quality–latency measurements:
| Panel | Problems | Mini | Flash | Mini→Flash | Flash→Flash |
|---|---|---|---|---|---|
| GSM8K-384 | 95 | 15.69 | 8.99 | 13.89 | 10.52 |
| MATH-384 | 100 | 19.37 | 7.55 | 18.03 | 10.67 |
| MATH-768 | 50 | 13.09 | 7.32 | 14.04 | 11.81 |
| MBPP+-384 | 153 | 21.60 | 12.68 | 17.09 | 12.83 |
Throughput is measured in final output tokens/s: total effective output tokens divided by total end-to-end generation time. Tokens are counted through the first EOS. drafting and refinement both contribute to latency. Model loading, warmup, and task scoring are excluded.
We acknowledge heterogeneous weights introducing memory overhead in Section 7. However, using two checkpoints does not intrinsically require two GPUs or additional dedicated hardware. The necessary device count depends on model size, precision, and available memory. Furthermore, in modern multi-GPU serving systems, drafting and refinement can be scheduled across existing GPUs, distributing the computational load rather than necessarily requiring additional devices.
5. Prior-work chronology and revisions. We thank the reviewer for pointing out shortcomings in our treatment of prior work. We agree that describing DiD as "concurrent" was imprecise and that we should have discussed ReMDM more explicitly. We acknowledge the relevance of ReMDM's draft-based remasking and DiD's draft-then-refine approach with specialized training. We do not claim novelty for the general principle of draft-then-refine itself. We will correct the chronology, improve our coverage of related work, and more clearly distinguish our training-free approach and heterogeneous small-to-large correction in the revised manuscript.
We hope these explanations clarify the distinctions between our work and the existing literature, as well as the practical contributions of our approach. We thank the reviewer again for their feedback and welcome any further questions or discussion.
Thank you for your response, I read it carefully. However, at this stage, my concerns are not resolved:
(1) The paper claims way too much space. Since you now agree that "LLaDA2.1 provides token-to-token editing, that MBE revises earlier blocks, and that the general principle of draft-then-refine already exists.", the only remaining novelty is not using the same model size for drafting and verification/editing. The paper should, from the ground up, be rewritten to make clear that this is the only contribution, and focus the experiments on actually novel angles rather than repeating well-known analysis from prior art. In my opinion, such large changes should warrant another review cycle.
Also, using a differently sized drafter and verifier is standard practice in speculative decoding. So, since we agree that the contribution of the paper is to use differently sized models, even in this aspect novelty is limited.
(2) What about Dream / Fast-dLLM? They use a Qwen tokenizer. Trends on a single model family cannot be assumed to generalize.
(3 & 4) Well, we agree that speculative correction, in contrast to standard speculative decoding, is not lossless. So it becomes all the more important to demonstrate faster inference latency and/or throughput compared to the established lossless baseline of D-Flash on autoregressive LLMs and speculative decoding on DLMs. If not provided, it can be assumed that the method has both worse latency & throughput as well as lower accuracy. Your newly added throughput numbers actually point towards this being the case, your system has almost 100x slower latency than modern D-Flash based speculative decoding approaches. This is a significant weakness I cannot overlook.
We thank the reviewer for the feedback. We believe our contribution extends beyond analyzing LLaDA2.1's MBE capability.
1. MBE and novelty. We agree that LLaDA2.1 provides token-to-token editing, that MBE revises earlier blocks, and that the general principle of draft-then-refine already exists. However, our method is not simply MBE applied after generation. MBE interleaves decoding with repeated edits to previously generated blocks. Our method instead completes a fast block-AR draft, then globally refines the entire response using bidirectional context with a separate refinement budget. This allows us to explore different allocations of computation between drafting and refinement, including the use of independently pretrained models.
The latter is central to speculative correction and deserves particular consideration in assessing novelty. Mini→Flash composes separately released DLMs without joint training, distillation, or adaptation, despite neither model being optimized for its role in the cascade. On MATH-384, it achieves 0.282 versus Flash's 0.292 at 2.32× speedup; on MBPP+, it achieves 0.621 versus 0.484 Plus pass@1.
We consider these results both practically useful and an important motivation for future model design. Our experiments demonstrate two working training-free inference configurations. Flash→Flash uses the same checkpoint for drafting and refinement, while Mini→Flash combines independently pretrained models of different sizes. Both provide concrete ways to implement draft-then-refine decoding using existing models, rather than requiring specialized training.
Furthermore, our results motivate two directions for developing models specifically around this paradigm. First, a small drafting model and a larger refinement model could be trained for complementary roles, potentially overcoming the distribution mismatch in our current cascade. Second, our Flash→Flash results motivate training two models of the same size but with separate weights and objectives, one specialized for drafting and the other for refinement. These are realistic model-design directions building directly on the architectures we have demonstrated. Exploring such specialized training was beyond our available compute, but our findings provide concrete motivation for doing so.