acceptodds
Under review as a conference paper at ICLR 2027

dSpec: Efficient Speculative Decoding for Block-Diffusion Language Models

Abstract

Speculative decoding accelerates a target language model by letting a lightweight drafter propose several tokens that the target verifies together. For block-diffusion language models, which decode one block of tokens at a time, it offers an alternative to the prevailing approach of lowering a confidence threshold, which trades generation quality for speed. Existing speculative decoders for these models, however, use trajectory-based or autoregressive-style verification, whose draft quality and continuation-state costs both affect acceleration. We present dSpec, which rests on the observation that a target pass over a fully masked block already predicts every position and thus yields a complete draft. dSpec uses this draft directly or corrects it with a one-layer head that reads the target's representation at each position. A single folded target pass then verifies the draft while preparing the block's key–value state and the next block's draft, so a fully accepted block costs only one pass. Requests rejected at different positions are recovered jointly in one shared pass. In SGLang, dSpec achieves 1.6–4.3× the throughput of Fast-dLLM's threshold-1.0 decoding on LLaDA2.0-mini and SDAR-8B at batch sizes 1–128. At batch one, its seven-task mean quality is within about one percentage point of that baseline. On matched hardware, it is roughly twice as fast as the fastest of three existing speculative decoders. Our analysis further shows that longer acceptance does not guarantee higher throughput, so proposal quality and continuation cost must be optimized together.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Diffusion LLM, Efficient AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UsHDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes dSpec, a speculative decoding method for block-diffusion language models such as LLaDA2.0-mini and SDAR-8B. It uses the target model’s own predictions on a fully masked block as the draft, and can optionally refine these predictions with a small one-layer head. It then runs one target forward pass that does two things concurrently. It verifies the draft from left to right, and it prepares the cache and the next block’s draft in case the whole block is accepted. When requests in a batch are rejected at different positions, their states are rebuilt together in one shared pass. Experiments in SGLang show clear throughput gains over confidence-threshold decoding across batch sizes, with similar average quality.

Critical strengths

The method is implemented in SGLang and test it at batch sizes from 1 to 128, which is closer to real serving settings, making the evaluation part comprehensive. In addition, the manuscript reports its results honestly. It gives a simple rule for when the learned head is worth its cost, and it admits that the head does not help on SDAR. The ablations also show clearly how much each part of the system contributes.

Critical weaknesses

(1) In terms of Appendix H, when the model checks a draft token, it can indirectly see that same token through the shared attention layout. As a result, the output is not guaranteed to match the target model. Lossless output is the main advantage of speculative decoding, so this is a serious limitation.

(2) TiDAR (arXiv:2511.08923) also verifies the current draft and drafts the next block from mask tokens in one forward pass with a structured attention mask. TiDAR also prepares drafts for every possible acceptance length and uses exact causal verification. The paper should cite TiDAR and explain which parts of folded verification are new.

(3) I find that the baseline LC1 decodes in confidence order, while dSpec verifies in left-to-right order. So dSpec does not reproduce the baseline’s output, even approximately. The paper reports that the average quality is similar, but some tasks drop clearly.

Questions for authors

I wanna know how much compute does training the Learned Draft head take? Given that the head improves throughput by only about 5% on LLaDA2 and slows down decoding on SDAR, this training cost should be justified.

3 comments

Thank you for the careful review, and for noting the SGLang evaluation at various batch sizes and the clear ablations. W1 and W3 both concern what dSpec's output should match. In the revision we implement an exact version of dSpec's left-to-right(L2R) verification.

W1. The shared layout does not guarantee the target's output

We agree. The shared layout approximates L2R verification over independent partial blocks (Appendix H). We implemented that exact form and now recommend it as the default. Each verified position gets its own copy of the block, and the copies do not attend to one another, so a candidate cannot influence its own verification. For any draft, the committed tokens are then those of the L2R decoding of the same target, one token at a time on the model's own block grid.

Both columns below use Self-Draft, with no training, so they differ only in the verification layout.

Lossless layoutShared layout
Same tokens as the greedy L2R decoding from the same state (BF16, batch 1), SDAR / LLaDA299.15% / 96.97% of passes97.06% / 60.87%
Throughput over LC1 at batch 1, SDAR / LLaDA22.92x / 4.58x2.93x / 4.15x
Throughput over LC1 at batch 128, SDAR / LLaDA21.40x / 1.06x1.72x / 1.83x
Target passes per output token (SDAR)0.3680.378
Query tokens per pass (SDAR)2014

The lossless layout disagrees with the L2R decoding only where the two highest logits are within 0.75 (median gap 10-12), that is, at BF16 near-ties. Because its passes are wider, its gain shrinks at large batch, but it stays faster than LC1 at every batch size we tested. We keep the shared layout as an option for large batches and describe it as approximate.

W2. TiDAR

We now cite TiDAR in related work (Section 5). It is an important prior design, since it also verifies the current draft and pre-drafts the next block in one pass. The differences:

TiDARdSpec
TargetA hybrid model trained for this purpose, with a causal verification pathExisting block-diffusion models with bidirectional attention inside the block
Exact verificationOne sequence gives the prediction for every prefixOne copy of the block per verified position
After a rejectionKV is truncated to the accepted prefixKV cannot be truncated; it is rebuilt only when a block ends on a correction
Next draftPre-drafted for every acceptance lengthThe copy at the mismatch already holds predictions for the rest of the block

What is new in dSpec is how this pass is built for existing bidirectional targets, where prefixes cannot share representations and KV cannot be truncated: exact verification uses one block copy per verified position, and the next block's state is prepared in the same pass.

W3. LC1 commits in confidence order; dSpec verifies left to right

You are right that dSpec does not reproduce LC1. The revision now states the claim as follows: dSpec is lossless with respect to the L2R decoding defined above, and LC1 serves only as the speed baseline. We measured what this choice costs:

SDAR-8BLLaDA2.0-mini
greedy L2R decoding minus LC1 (seven tasks, points, 95% CI)-1.45 [-3.91, +0.95]+2.02 [-0.03, +4.17]
Exact verification in LC1's confidence order: throughput relative to exact L2R0.79-0.83x0.54-0.68x
Draft token equals the committed token90.3%82.7 / 65.6 / 46.2 / 25.1% at positions 1-3 / 4-7 / 8-15 / 16-31

The greedy L2R decoding differs from LC1 mainly in the order of commitment, and on neither target is its quality difference from LC1 statistically significant. Following LC1's order exactly is possible but slower (0.461 vs 0.368 target passes per token), because the draft must then predict the unmasking order as well as the tokens (Appendix K.4). L2R order removes this constraint at comparable quality, and verification starts where the draft is most accurate (third row).

On the task-level drops you noticed: for LLaDA2 IFEval, Table 2 had 82.07 for LC1 and 78.62 for dSpec with the shared layout. In the new runs, the greedy L2R decoding gives 81.33 and the lossless layout 81.70, against 81.89 for LC1. The drop therefore does not come from the L2R order, but from the approximated shared layout, in which a candidate can influence its own verification and the next block starts right after each correction.

Q. Training cost of Learned Draft

About 1.3k (SDAR) and 0.75k (LLaDA2) H20 GPU-hours per head, for a gain of 1.03-1.04x on LLaDA2 and a slowdown on SDAR (0.92-0.97x). We agree that the cost is not justified, and all lossless results use Self-Draft, which needs no training. We keep the head as an optional component because it illustrates the cost criterion in Section 4.3: a longer accepted prefix helps only if it outweighs the added step time.

If the lossless layout address W1 and W3, we would appreciate it if you could reconsider your rating.

AuthorsOpenReview ↗

Follow-up Before the End of the Discussion Period

Dear Reviewer UsHD,

As the discussion period closeing, we would like to follow up on our response. The main change in the revision was prompted by your W1: dSpec now performs exact left-to-right verification by default. It commits the same tokens as greedy L2R decoding of the target, apart from BF16 near-ties, and remains 2.92x / 4.58x faster than LC1 at batch one without a trained head. The IFEval drop you noted came from the original shared layout and does not appear with the lossless layout. We have also added a comparison with TiDAR.

We hope this resolves your concern regarding losslessness. If so, we would be grateful if you could take it into account in your final rating.

Hi authors of submission 60266,

Thank you for the response and the additional experiments! The lossless layout addresses my concern about exactness, and the TiDAR discussion is helpful. However, given TiDAR, the novelty is limited. The Learned Draft head also adds little benefit for its training cost. In addition, the main results still use the approximate shared layout, and some per-task quality drops remain.

So I will keep my original score, and I encourage the authors to further improve this work and resubmit a stronger version in the future!

Thanks,

Reviewer UsHD

Reviewer_KggzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies speculative decoding for block-diffusion language models, focusing on draft generation and the cost of preparing the states needed to continue after verification. dSpec forms a complete proposal from the target model’s predictions on a fully masked block. Self-Draft uses these predictions directly, while Learned Draft optionally refines them with a one-layer head. Candidates are approximately verified in parallel, followed by left-to-right prefix acceptance and a correction at the first mismatch. Folded verification combines verification, candidate-block KV construction, and next-block prediction in one forward pass and fully accepted blocks will reuse these states. Rejected requests are reconstructed through Batched recovery, which supports different committed lengths within one batch.

On LLaDA2.0-mini and SDAR-8B, the authors report 1.56×1.56\times–4.27×4.27\times throughput gains over LC1 across batch sizes 1–128. At batch size 1, seven-task average scores change from 61.27 to 61.50 and from 52.18 to 51.17, respectively. A separate four-task evaluation shows substantially higher average quality than the more aggressive LC0.5 baseline at every tested batch size.

Critical strengths
  1. Competitive practical results without training a drafter.

    In Table 5, on SDAR with two H20 GPUs, batch one, and matched eager execution, Self-Draft achieves 3.25×3.25\times Native throughput, compared with 1.62×1.62\times for FreeDave. Their five-task mean quality scores are 64.80 and 64.04, respectively. This benefit does not depend on Learned Draft and cannot be explained by recovery batching across requests. It is a strong practical result.

  2. Direct evidence that state reuse helps when full acceptance is frequent.

    SDAR has approximately 76% full-block acceptance. At batch one, adding Folded verification increases throughput to 1.306×1.306\times Separate execution in Table 27. Since cross-request recovery batching is unavailable in this setting, the result provides relatively direct evidence for the value of preparing and reusing continuation states.

Critical weaknesses
  1. The scope and technical contribution of Batched recovery need clearer justification.

    Appendix G states: “Separate execution runs approximate verification first, then jointly constructs committed KV and next-block conditioning, grouping requests by committed length.” (PDF p. 23, lines 1240–1241.) Appendix H confirms: “The grouped implementation batches requests with equal committed lengths.” (p. 25, lines 1339–1340.)

    Thus, heterogeneous rejection positions can split recovery into several smaller batches. For LLaDA2 at batch 32, Folded verification alone achieves 0.944×0.944\times, while adding Batched recovery reaches 1.988×1.988\times (Table 27). This clearly demonstrates the benefit of combining those recovery batches. However, the paper does not explain whether the grouped control reflects a concrete constraint in existing implementations or a simplifying choice. It is also unclear what essential functionality Batched recovery adds beyond standard padding or variable-length batching. The control is valid for a component ablation, but its speedup alone does not establish substantial novelty or an advantage over implementations that already handle heterogeneous recovery efficiently.

  2. The added components offer limited benefits in several important settings.

    On LLaDA2, full-block acceptance is only about 2.5%, and Folded verification reduces throughput at batch 32. At batch one, Folded verification plus Batched recovery provides only 1.012×1.012\times over Separate execution, while the complete method achieves approximately 4.27×4.27\times over LC1. For SDAR’s short blocks and strong self-drafts, Learned Draft also reduces throughput to 0.954×0.954\times and 0.938×0.938\times that of Self-Draft at batches one and 32, respectively (Table 22).

    The authors clearly report these limitations, but they narrow the components’ incremental contribution. Where these components contribute little, the paper should more clearly explain what distinguishes the underlying drafting-and-verification path from prior self-drafting and parallel verification methods.

  3. Quality evidence does not cover all settings used for the speed evaluation.

    The seven-task comparison against LC1 is limited to batch size 1, with some task-level regressions despite similar average scores—for example, LLaDA2’s IFEval decreases from 82.07 to 78.62. At larger batch sizes, the four-task quality comparison uses LC0.5, so it does not establish whether quality remains comparable to LC1. Moreover, MT-Bench (first-turn prompts), Alpaca, and Arena-Hard-v2 are evaluated for throughput without quality scores (Table 6 and the accompanying evaluation description). The results therefore support useful speed–quality gains in the scored settings, while quality preservation at larger batch sizes relative to LC1 and on these three conversational/instruction workloads remains unverified.

Questions for authors
  1. Why does Separate execution group requests by committed length? Is this required by kernel/cache constraints or chosen for the ablation? Please clarify the technical obstacles that Batched recovery overcomes and when the grouped implementation is an appropriate reference.

  2. Could you provide paired dSpec/LC1 quality scores at one representative larger batch size, and any available quality results for MT-Bench, Alpaca, or Arena-Hard-v2? This concerns applicability beyond the demonstrated batch-one results, rather than their validity.

3 comments

Thank you for the specific questions, and for recognizing that Self-Draft gives strong results without a trained drafter and that state reuse helps when full acceptance is frequent.

Q1 / W1. Why separate execution groups by committed length, and what batched recovery adds

The grouping is a simplification we chose so that the ablation changes one thing at a time. It is not required by any kernel or cache constraint, and the revision now says so (Appendix K.3). The released FreeDave and S2D2 code only runs at batch one, so we had no existing batched recovery to compare with.

Batched recovery itself is padding with per-request attention visibility and position IDs. It is needed in the shared layout (the D/V verification of the submission): there a rejection starts the next block right after the corrected token, so committed lengths differ across requests. The control that you and MB4f asked for shows how much it is worth:

TargetBatchSep.+batched / Sep.Complete / Sep.+batched
LLaDA2.0-mini322.2630.905
SDAR-8B10.9981.296

On LLaDA2 at batch 32, batched recovery gives 2.26x and adding folding makes it slightly slower, because only 2.5% of LLaDA2 blocks are fully accepted. On SDAR at batch one, batched recovery has no effect and folding gives 1.30x, because 76% of blocks are fully accepted. This is what the ablation shows: where most blocks are rejected, rebuilding state is the main cost after drafting, so batching it matters more than preparing states in advance; where most blocks are accepted, the opposite holds.

W2. Limited benefit of the components; what distinguishes the drafting-and-verification path

On the learned head we agree with you. It costs about 1.3k (SDAR) and 0.75k (LLaDA2) H20 GPU-hours for a 3-5% gain on LLaDA2 and a loss on SDAR. The revision reports this cost and keeps Learned Draft as an optional component, together with the autoregressive-drafting ablation, because both show that a longer accepted draft does not pay off unless it outweighs the extra step time. All lossless results below use Self-Draft, which needs no training.

Self-drafting itself is not new, since FreeDave and S2D2 also draft from the target. The methods differ in what the draft is verified against (target passes per output token on 90 matched SDAR prompts):

MethodThe draft is verified againstExactPasses per token
FreeDaveA confidence-ordered sampler; tokens and unmasking order must both matchYes0.707
S2D2A confidence thresholdNo0.854
SimSDApproximate verification, with a separate 1.7B drafterNo0.298, plus 1.194 drafter passes
dSpec (lossless)Greedy L2R decoding of the same targetYes0.368

Compared with FreeDave, which is also exact, dSpec needs about half as many target passes, because its draft only has to match the tokens and not an unmasking order. With the order fixed, the target's fully masked pass is already a complete draft. Without any training, dSpec runs at 2.92x / 4.58x LC1 at batch one (SDAR / LLaDA2).

Q2 / W3. Paired quality against LC1 at a larger batch, and on the conversational workloads

Seven-task mean minus LC1 (points, paired bootstrap 95% CI):

ConfigurationSDAR b=1SDAR b=32LLaDA2 b=1LLaDA2 b=32
Lossless (default)-1.15 [-3.69, +1.46]-1.23 [-3.45, +1.03]+1.71 [-0.70, +4.24]+0.45 [-1.58, +2.21]
Shared layout (Learned Draft)-1.00 (Table 2)-0.73 [-3.21, +1.61]+0.23 (Table 2)-1.25 [-3.17, +0.58]

For comparison, LC1's own mean changes by -1.47 (SDAR) and +1.22 (LLaDA2) points between batch 1 and batch 32.

Win rate of lossless dSpec over LC1 on MT-Bench first turns, Alpaca and Arena-Hard-v2 (pairwise judge DeepSeek-V4.1-Flash, both presentation orders):

Targetb=1b=32
SDAR-8B0.515 [0.459, 0.569]0.524 [0.484, 0.564]
LLaDA2.0-mini0.556 [0.500, 0.613]0.529 [0.487, 0.570]

As a reference for noise, LC1 against a rerun of itself gives 0.493-0.505. None of the four intervals for dSpec lies below 0.5, so we see no sign of degradation on these workloads.

On the IFEval regression you pointed out (LLaDA2, 82.07 for LC1 and 78.62 for dSpec in Table 2), the new runs give (Table 39):

BatchLC1greedy L2R decodingLossless
181.8981.3381.70
3279.6782.2680.78

The drop does not appear with the greedy L2R decoding or the lossless layout.

If these results address your concern about applicability, we would appreciate it if you could reconsider your rating.

AuthorsOpenReview ↗

Follow-up Before the End of the Discussion Period

Dear Reviewer Kggz,

As the discussion period closing, we would like to follow up on your two questions. We have added paired quality results against LC1 at batch 32, which show no significant difference on either target, as well as pairwise-judge win rates on MT-Bench, Alpaca and Arena-Hard-v2, all between 0.52 and 0.56. We have also clarified that grouping requests by committed length was a choice made for the ablation, not a kernel constraint.

If these results address your questions, we would appreciate it if you could reconsider your rating.

Reviewer_KggzOpenReview ↗

Official Comment by Kggz

Thank you for the clarifications and additional experiments. They clarify the recovery baseline and verification guarantee, and provide further evidence on generation quality. However, my concerns about novelty and the incremental contribution remain. The added components largely build on established techniques, and their demonstrated benefits are limited or concentrated in particular settings. While I appreciate the practical speedups, I remain unconvinced that the contribution beyond prior work is sufficiently substantial. I therefore maintain my rating of 2 (weak rejection).

Reviewer_MB4fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

dSpec proposes speculative decoding for block-diffusion language models, addressing the cost of constructing continuation states after verification. The target's hidden states and logits for a fully masked block form the anchor; Self-Draft takes the logits' argmax. Learned Draft adds in-position refinement through a one-layer bidirectional Transformer that produces a residual on the anchor logits. Folded verification uses D/V for approximate L2R verification and C/N for candidate KV construction and next-block anchor prediction. Full acceptance reuses C/N; batched recovery reconstructs committed KV and the next anchor for rejected requests with unequal committed lengths.

On LLaDA2.0-mini and SDAR-8B at batch sizes 1-128, the paper reports eight-workload geometric-mean speedups of 1.56-4.27x over confidence-threshold decoding at threshold 1.0 (LC1). At batch one, seven-task mean quality changes by +0.23 and -1.00 percentage points, respectively. Comparisons with FreeDave, S2D2, and SimSD are conducted on SDAR at batch one.

Critical strengths
  1. The execution path addresses continuation-state costs directly. C/N prepares candidate KV and the next anchor during verification, and batched recovery handles requests with unequal committed lengths. Self-Draft requires no trained drafter.

  2. The experiments cover two targets, batch sizes 1-128, multiple inference block sizes, confidence-threshold baselines, and three speculative methods at batch one. The appendices add long-budget evaluation, single-GPU comparisons, CUDA Graphs, stage profiles, and numerical checks. In Table 5, Self-Draft achieves 3.25x Native throughput versus FreeDave's 1.62x in eager execution, and 3.14x versus SimSD's 1.25x with CUDA Graphs, with quality reported in both modes.

  3. The ablations identify when each component helps. Learned Draft improves LLaDA2 throughput but reduces SDAR throughput. Folding benefits SDAR's high full-acceptance rate, whereas batched recovery provides the main execution gain on LLaDA2 (Sections 4.2-4.3; Tables 22, 27-28).

Critical weaknesses
  1. The quality impact of approximate L2R verification is not isolated. Appendix H states that V1 can read V2, which can read D2 containing the candidate being checked at position 2. Candidate-dependent verification may therefore accept tokens that independently reconstructed partial blocks would reject, changing the committed tokens and potentially generation quality. Table 2 reports complete-generation scores, but the small changes in mean quality do not establish that this approximation is harmless across tasks. LC1 uses confidence-threshold decoding, while Separate execution uses the same approximate D/V verifier; neither isolates the approximation's quality impact. A fixed-state comparison of first-mismatch positions and committed tokens, supplemented where feasible by task quality with independent partial-block verification, would address this concern. The current results do not establish that the approximation causes the observed task-level decreases.

  2. The execution ablation lacks Separate execution with batched state construction. Tables 27-28 compare Separate execution, folded verification, and folded verification with batched recovery. Separate execution groups requests by committed length. Thus, Complete/Separate combines the effects of preparing C/N in advance and batching unequal committed lengths. On LLaDA2 at batch 32, Folded/Separate is 0.944x and Complete/Separate is 1.988x. Adding a Separate + batched state construction control would isolate the incremental benefit of folding. The D/V layout is related to SimSD's temporal attention mask, making a matched control for C/N state preparation and reuse particularly relevant to the system contribution.

Questions for authors

NA

2 comments

Thank you for the careful reading and for the two concrete suggestions.

W1. The quality impact of approximate L2R verification is not isolated

We did both things that you suggested: a fixed-state comparison, and task quality with independent partial-block verification. For the latter we implemented verification with independent partial blocks: each verified position gets its own copy of the block, and the copies do not attend to one another, so a candidate cannot influence its own verification. With this layout the committed tokens are exactly those of greedy L2R decoding of the same target on its own block grid . We call it the lossless layout and now recommend it as the default.

Fixed-state comparison. At each verification pass, we recomputed from the same state what the greedy L2R decoding would commit and compared it with what each layout committed. Both layouts use Self-Draft, so only the verification layout differs.

Steps whose committed tokens equal the greedy L2R decoding'sSDAR-8B (b=1 / 32)LLaDA2.0-mini (b=1 / 32)
Lossless layout99.15% / 99.08%96.97% / 96.86%
greedy L2R decoding run twice with different batch padding99.86% / 99.39%97.56% / 97.64%
Shared layout, b=197.06%60.87%

The approximation in the shared layout does change decisions, and the effect is large on LLaDA2 but small on SDAR.

To separate real decision changes from BF16 rounding, we looked at how clear the greedy L2R decoding's choice was at each disagreement, measured by the gap between its two highest logits. If the gap is small, the two tokens are nearly tied, and rounding can flip the choice. In the lossless layout this is always the case: every disagreement has a gap of at most 0.75 (median 0.125), while the median gap over all positions is 10-12. The second row shows that changing only the batch padding causes a similar number of such flips. In the shared layout on LLaDA2, by contrast, 16.8% of all steps disagree at a gap above 1, with gaps up to 19.1.

Task quality (seven tasks, paired bootstrap 95% CI, points).

ComparisonSDAR-8BLLaDA2.0-mini
Shared layout (Learned Draft) minus greedy L2R decoding, b=1+0.42 [-1.57, +2.49]-1.93 [-3.45, -0.18]
Lossless minus LC1, b=1-1.15 [-3.69, +1.46]+1.71 [-0.70, +4.24]
Lossless minus LC1, b=32-1.23 [-3.45, +1.03]+0.45 [-1.58, +2.21]

On LLaDA2 the shared layout does cost quality: the submitted configuration scores 1.93 points below L2R decoding. On SDAR the difference is small (+0.42). The lossless layout does not show this loss: on LLaDA2 it is +1.71 and +0.45 points relative to LC1 at batch 1 and 32. The lossless layout uses about as many target passes as the shared one, but wider ones, so it is as fast at batch one and slower at large batch (General Response).

W2. The execution ablation lacks Separate execution with batched state construction

We added this control on the shared layout, paired on the same GPU and inputs (Table 41):

TargetBatchSep.+batched / Sep.Complete / Sep.+batchedComplete / Sep.
LLaDA2.0-mini10.9961.0181.014
LLaDA2.0-mini322.2630.9052.048
SDAR-8B10.9981.2961.293
SDAR-8B321.0681.1431.221

The middle column is the matched control for C/N state preparation and reuse that you suggested. It shows that the 1.99x we reported on LLaDA2 at batch 32 came from batched recovery (2.26x), and that folding makes this setting 9% slower: only 2.5% of LLaDA2 blocks are fully accepted, so the prepared states are rarely used. On SDAR, where 76% of blocks are fully accepted, folding gives 1.30x at batch one and 1.14x at batch 32. Table 42 shows the same pattern across LLaDA2 inference widths. So folding pays off when most blocks are fully accepted, and batched recovery when most requests reject; the revision states folding's benefit with this condition.

We hope this addresses your soundness concerns.

AuthorsOpenReview ↗

Follow-up Before the End of the Discussion Period

Dear Reviewer MB4f,

Thank you again for suggesting verification with independent partial blocks. We have implemented it and adopted it as the default: it matches greedy L2R decoding except at BF16 near-ties and removes the quality loss we observed on LLaDA2. We have also added the separate execution with batched state construction control you requested.

As the discussion period closes, we wanted to ensure these results had come to your attention. If you find that it strengthen the paper, we would be grateful if your rating could reflect this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for the careful reading. The main change in the revision is that dSpec can now run losslessly, and in this form it is still 2.9x to 4.6x faster than the baseline at batch one, with no trained component. This post summarizes the changes; the individual replies give the details each reviewer asked for.

What we added. The submission verified drafts with a shared layout that approximates left-to-right (L2R) verification over independent partial blocks (Appendix H). UsHD and MB4f asked about this approximation, and MB4f suggested verifying with independent partial blocks. We implemented this exact form. It commits the same tokens as greedy L2R decoding of the same target, which we call the reference decoder, and we now recommend it as the default ("lossless layout"). The drafting step, the L2R acceptance rule and all submitted results are unchanged.

Why exact verification stays fast. On these models, much of the cost of speculative decoding comes from having to follow the order in which the target unmasks positions. If the order is fixed to L2R, one target pass over the fully masked block already gives a complete draft. The submission argued this in Sections 3.1 and 3.3, and the new measurements support it:

  • Verifying in confidence order needs more passes. On 90 SDAR prompts at batch one, FreeDave uses 0.707 target passes per output token, dSpec verified exactly in confidence order uses 0.461, and dSpec verified exactly in L2R order uses 0.368.
  • Fixing the order to L2R does not lower quality. The greedy L2R decoding differs from LC1 (the paper's threshold-1.0 baseline) by -1.45 [-3.91, +0.95] points on SDAR and +2.02 [-0.03, +4.17] on LLaDA2 (seven-task mean).
  • The draft does not need training. The fully masked pass already predicts the committed token at 90.3% of SDAR positions; on LLaDA2's longer blocks it does so at 82.7% of the first three positions and less often further in. A trained head adds only 3-5% throughput on LLaDA2 and is slower on SDAR, so all lossless results use Self-Draft.
  • Making verification exact widens each pass but does not add passes: 20 instead of 14 query tokens per pass, and 0.368 instead of 0.378 passes per token.

Speed. Throughput over LC1 (geometric mean over eight workloads, both layouts with Self-Draft; Table 36 in Appendix K.1):

TargetLayoutb=183264128
SDAR-8BLossless2.92x2.28x1.64x1.40x1.40x
SDAR-8BShared2.93x2.50x1.96x1.77x1.72x
LLaDA2.0-miniLossless4.58x2.40x1.41x1.22x1.06x
LLaDA2.0-miniShared4.15x3.28x2.29x2.04x1.83x

At batch one the lossless layout is as fast as the shared one. At larger batches it falls behind, because once the GPU is saturated wider passes take longer. For that setting we keep the shared layout as an option, and the paper now states clearly that it is approximate.

Exactness in BF16. Starting from the same state, the lossless layout commits the same tokens as greedy L2R decoding on 99.1-99.2% of verification passes on SDAR and 96.9-97.0% on LLaDA2. At every disagreement the two highest logits are within 0.75 of each other, while the median gap is 10-12, so these are rounding effects. Under the same test, the shared layout agrees on 97.1% (SDAR) and 60.9% (LLaDA2) of passes. The full table is in our reply to MB4f.

Quality. Lossless dSpec minus LC1, seven-task mean (paired bootstrap 95% CI, points):

Targetb=1b=32
SDAR-8B-1.15 [-3.69, +1.46]-1.23 [-3.45, +1.03]
LLaDA2.0-mini+1.71 [-0.70, +4.24]+0.45 [-1.58, +2.21]

None of these differences is statistically significant. A pairwise judge on the three conversational workloads also shows no sign of degradation (reply to Kggz).

Other additions. The execution control requested by MB4f (replies to MB4f and Kggz), a comparison with TiDAR (reply to UsHD), and the training cost of Learned Draft (reply to UsHD). New text in the PDF is in blue. All added tables are in Appendix K (Tables 36-43).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.