acceptodds
Under review as a conference paper at ICLR 2027

Factorization-Guided Training and Inference for Diffusion-Drafted Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference through parallel verification of tokens proposed by a drafter. Diffusion drafters further parallelize proposal generation, but greater proposal capacity or higher mean accepted tokens (MAT) need not yield higher throughput. Because verification accepts only a consecutive prefix, early rejections can render later correct predictions ineffective, limiting MAT gains, while even higher MAT may not improve throughput if per-round latency increases. We introduce DScout, which uses two complementary factorizations to guide diffusion drafter training and inference for higher throughput. For training, DScout factorizes position-wise loss into inclusion, weighting, and prediction loss. This guides the design and combination of mixed-horizon training, intra-block unmasking, and acceptance-weighted loss to align supervision with prefix acceptance and improve MAT. For inference, DScout factorizes throughput as the ratio of MAT to per-round latency, guiding target-conditioning depth, drafter depth, and inference block size selection to identify a high-throughput configuration for each workload. Across six benchmarks, DScout achieves and mean speedups over autoregressive decoding on Qwen3-4B and Qwen3-8B, respectively, improving mean speedup over the state-of-the-art baseline by 9.1% and 7.1%. Across models and concurrency settings, our inference configuration policy provides up to 9.75% additional mean speedup over DScout's fixed configuration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, diffusion language models, drafter training, inference-cost-aware search, throughput optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E7UGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DScout, a framework to improve the efficiency of diffusion-based speculative decoding through both training and inference optimization for diffusion drafters. The paper points out that, since speculative decoding can only accept consecutive token prefixes, higher prediction accuracy does not necessarily lead to higher MAT, while higher MAT can not guarantee the greater throughput. Based on the observation, specifically, for training, it introduces mixed-horizon training, intra-block unmasking, and acceptance-weighted loss to improve the mean accepted tokens. For inference, it optimizes target-conditioning depth, drafter depth, and block size to balance MAT and latency. Experiments across multiple models and benchmarks show that DScout consistently outperforms DFlash.

Critical strengths
  1. This paper is written very clearly and concisely. Also, the paper provides rich technical details, including how to select hidden states and drafter depth.
  2. The proposed Acceptance-Weighted Loss is very reasonable and has good motivation. As speculative decoding can only accept consecutive token prefixes, a draft token can be accepted only if all preceding draft tokens have been accepted. It is natural and intuitive to adjust the loss weights based on prefix acceptance probabilities.
Critical weaknesses
  1. The method in this paper seems to be a stack of several small improvements in training and inference based on DFlash. Although technically solid, the novelty is incremental.
  2. During inference, DScout still uses DFlash's one-pass full-mask prediction instead of multi-round unmasking. However, during training, intra-block unmasking is used. Why does this seemingly inconsistent training-inference approach work? The authors should provide analysis and explanation.
  3. The authors chose to use more layers of target features. Although this doesn't change the computational cost much, the change in memory cost can be huge, significantly increasing memory consumption for both training and inference. For offline training, it will also put a lot of pressure on storage space.
  4. Compared to the baseline DFlash, DScout's improvement in speedup seems to be relatively limited or even worse, although DScout is largely built upon the existing mechanisms of DFlash and incorporates many optimizations for both training and inference.
Questions for authors
  1. For training, DScout adopts intra-block unmasking, while during inference, DScout still uses one-pass full-mask prediction rather than multi-round unmasking. How can this training-inference mismatch work?
  2. What is the actual memory overhead for using more target hidden states, especially during training?
  3. Although DScout incorporates several optimizations for both training and inference, the additional speedup gains over DFlash are relatively limited or even worse. The more detailed ablation studies on end-to-end speedup for each component is required to figure out which contributes most to the performance gains, which have little effect, and which even negatively affect performance.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

1. Training uses intra-block unmasking while inference uses full masking, and the cost of this mismatch is not measured.

We thank the reviewer for pointing out this inconsistency. We have measured the cost of the mismatch, and it runs in our favour: a drafter trained with intra-block unmasking is better under the full masking used at inference - on mixed horizons {8,16,24,32}, adding unmasking raises acceptance length from 6.33 to 6.49 (+0.16), and the evaluation itself is carried out under full masking.

Why this works. The inference-time condition is not approximated during training but is the most frequent one: a block is kept fully masked with a fixed probability, and only the remaining blocks draw a mask rate from a Beta distribution and reveal a random subset of positions, which provide context only and do not enter the prediction loss. Full masking is therefore both the inference condition and the most common training condition, and the partially revealed blocks act as data augmentation, letting the model see prediction contexts beyond full masking, as is standard in training diffusion language models.

2. The memory cost of the extra target hidden states is not given, for training or for inference; offline training would also consume storage.

We thank the reviewer for separating the training, inference and storage sides.

On the inference side we measured two targets at four concurrency levels. For each target we take a pair of drafters with the same drafter depth that differ only in the number of target layers read, at block size 32, reading peak serving memory on an 80 GB accelerator, which includes the KV pool SGLang preallocates at startup; the percentages are computed from the unrounded peaks:

Concurrency8B / 5 layers8B / 35 layersIncrease4B / 5 layers4B / 35 layersIncrease
150.753.7+5.8%49.751.8+4.2%
851.454.8+6.6%50.452.7+4.6%
1651.955.8+7.5%50.953.5+5.1%
3253.358.8+10.2%52.255.5+6.3%

The cost is 4%-10% across concurrency levels.

On the training side two quantities grow with the layer count. One is the captured target state, at layers x hidden width x 2 bytes per token, which on Qwen3-8B is 8 KiB per token per layer, so 40 KiB at 5 layers and 280 KiB at 35. The other is the fusion projection itself: its input width is layers x hidden width, so the drafter's parameter count grows linearly in the conditioning depth, and during training this part also carries gradients and optimizer state.

On the storage side these states are read from the target online and never written to disk, so there is no offline cache; the byte counts above are what caching offline would cost.

If memory is tight, the conditioning depth can be reduced with little loss: MAT rises 1.9% from 9 to 18 layers and only 0.7% more from 18 to 35 (same recipe and drafter depth, six-task mean at concurrency 1).

3. The ablation uses acceptance length rather than end-to-end speedup; and the improvement over DFlash is limited, with some settings worse.

We thank the reviewer for pointing out that the improvement over DFlash is limited. Our claim is not only about speedup, which varies with the target and the load: what we also want to leave behind is the factorization that produced the three techniques. The position-wise loss decomposes into inclusion, weighting and prediction factors, which decides which techniques to combine and anticipates that their gains will not simply add; the "Joint design" paragraph of §3.1.2 states this with the mechanism.

Which contribution is largest. We read the same ladder end to end, as six-task mean tokens per second at concurrency 1: 529.9 at fixed 16, 531.5 at fixed 32, 541.3 with mixed horizons, 555.2 with unmasking, 595.5 with AWL, so MAT rises 16.1% and throughput 12.4% across it. AWL is the largest on both, +0.49 in acceptance length and +7.3% in throughput.

On the settings where we are worse. In the serving table this happens at Llama-3.1-8B, concurrency 32: DScout (16) reads 2.33, below DFlash's 2.50. The reason is that the suitable block size on this target is shorter than 16 to begin with - DFlash uses a block-10 drafter here, which shows they also found the optimal block size to depend on the target and the load, except that using it requires training a separate drafter for block 10. The DScout (16) row is an unadapted fixed block 16, so it compares our unadapted configuration against their adapted one; selecting block 10 from the same checkpoint gives 2.57, which is ahead. This cell is precisely where our approach is useful.

Thanks for the authors' rebuttal. The most concerns have been addressed. Although I still concern the actual novelty of the proposed method, I will raise my score to 3.

Reviewer_nUknOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

DScout aims to improve throughput. A key claim of the authors is that, MAT alone cannot characterize improvements in overall throughput. The paper therefore handles the problem from both the training and inference sides.

  1. To improve MAT, the authors introduce a position-wise loss formulation that includes three strategies: mixed-horizon training, intra-block unmasking, and acceptance-weighted loss.
  2. To improve inference efficiency, they analyze factors beyond MAT that affect throughput and consider three approaches:
    • Full target conditioning, which can also increase MAT.
    • Unimodality-guided drafter depth selection.
    • Block-size selection without retraining.

The experimental results show gains of approximately 9.1% and 7.1% on Qwen3-4B and Qwen3-8B (at concurrency 1), respectively. At higher concurrency levels, the gains are approximately 5.1%–7.3% for Qwen3-4B and 1.6%–3.4% for Qwen3-8B.

Critical strengths
  1. DScout extends DFlash and presents some experimental observations useful for industrial deployment. For example, the mixed-horizon strategy may reduce the need to maintain multiple checkpoints for different deployment scenarios, as shown in Table 2.

  2. The position-based loss, mixed-horizon training, intra-block unmasking, and AWL do improve MAT. Specifically, the sequence B16 → B32 → mixed → unmask → AWL yields a total MAT improvement of 16.1%, with successive increments of +0.171, +0.151, +0.155, and +0.491.

  3. DScout provides strategies for selecting draft block size and drafter layer depth to maximize throughput at concurrency 1. Inference block-size selection yields improvements of 2.1% and 3.8%. At concurrency levels of 8, 16, and 32, DScout achieves overall gains of approximately 1.6%–3.4% over DFlash on Qwen3-8B.

  4. Further improvements to DFlash are among the directions of interest to the speculative decoding community.

Critical weaknesses
  1. The authors point out that MAT is not the only factor affecting throughput. However, the method has two main components: position-related modifications and strategies for tuning drafter depth and inference length. The decomposition of throughput appears in Section 3.2 and includes MAT, inference length, and drafter depth. However, the position-related modifications in Section 3.1 only affect MAT. Drafter depth and inference length are selected as inference configurations, and the optimization strategy seems to enumerate all options and select the best one. The experimental gains are also limited, at approximately 2.1%–3.8%.

  2. Position-wise loss is already a topic of interest to the community and to DFlash-style methods. However, the paper introduces several position-related strategies: mixed-horizon training, intra-block unmasking, and AWL. Each provides some gains, but the individual gains are not very large. Specifically, the sequence B16 → B32 → mixed → unmask → AWL yields a 16.1% relative improvement in MAT over the initial baseline, with successive absolute increments of +0.171, +0.151, +0.155, and +0.491 (0.968/6.01×100%≈16.1%0.968 / 6.01 \times 100\% \approx 16.1\%).

  3. For intra-block unmasking, I can see gains in the results, but it is essentially a variant of position-based loss. Why are mixed-horizon training, intra-block unmasking, and AWL all needed for position-based optimization? These methods appear to address the same issue from several angles. In Figure 4, mixed-horizon training and unmasking add approximately 0.151 and 0.155 to MAT in the cumulative recipe, respectively, while AWL adds approximately 0.491. However, I do not understand why so many local strategies are needed to address the same issue. As a reader, I find it difficult to understand the core innovation compared with existing methods.

    • AWL raises the same issue. It also seems to be a variant of position-related loss, further emphasizing position-related information through the loss computation. However, its incremental gain is also limited.
  4. Besides MAT, drafter layer count and decoding depth also affect the final gains. However, the improvements in Section 3.1 do not explicitly use this information. The connection seems indirect: is the drafter first selected using the strategies in Section 3.2, and then the strategies in Section 3.1 used to determine the final gains? Is the architecture selected before applying the training recipe, or are candidate models trained before selecting the configuration? Does this require repeated training? At the same time, this approach does not seem particularly novel. Some of these descriptions already appear in DFlash[1] and D-PACE[2].

  5. For full target conditioning, the authors consider FLOPs and provide throughput measurements at a fixed acceptance length. It would be helpful if the authors could also report the increase in GPU memory usage across different concurrency levels and models. Activations from all target blocks may need to be kept in GPU memory, which would increase memory consumption.

  6. The paper provides an analysis of unimodality-guided drafter depth selection and experiments on block-size selection without retraining in Tab. 4 and Fig. 6. These appear to be inference-stage improvements, but the gains are only approximately 2.1%–3.8% for block-size selection at concurrency 1, which I find limited. The cost of searching over depth and layer counts is also not described. For example, on Qwen3-4B and Qwen3-8B, how many checkpoints with different values of dd need to be trained, how many values of bs need to be evaluated, how much GPU time is required, and how much search cost does the unimodality-guided strategy actually save? An evaluation of these costs would be useful.

  7. The overall gains over DFlash are not very substantial: approximately 7%–9% on the 4B and 8B models at concurrency 1. [1] Chen, Jian, Yesheng Liang, and Zhijian Liu. "Dflash: Block diffusion for flash speculative decoding." arXiv preprint arXiv:2602.06036 (2026).

[2] Wu, Tianyu, et al. "D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting." arXiv preprint arXiv:2605.18810 (2026).

Questions for authors
  1. Why does the change B16 → B32 increase the proportion of rounds with accepted lengths of 1–4? What causes this? Why focus on the range 1–4 rather than 1–8 or another range(Is this related to the model or other factors?)? Why does increasing B lead to more early rejections, and how does mixed-horizon training address the problem reflected in this observation?
  2. Tab. 2 is somewhat confusing. For example, why does fixed-32 training with inference block size 16 perform worse than fixed-16 training with inference block size 16? Why does fixed-32 training with inference block size 24 also perform worse than fixed-16 training with inference block size 24?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

1. Three techniques for one problem, and the gains are small; the §3.1 changes affect only MAT, while the inference side looks like enumerating all options.

We thank the reviewer for raising this.

Why three items are needed.

The three act on different factors of the position-wise loss: mixed horizons decide which draft positions are supervised; intra-block unmasking changes which are included and what context the rest are predicted from, so it shares the inclusion factor with mixed horizons and §3.1.2 designs those two jointly; AWL leaves inclusion untouched and changes only how much each supervised position counts. What we claim is the factorization: it decides which techniques to combine and predicts that their gains will not simply add - adding unmasking on the {4,8,16,32} horizon set reads 6.361 against 6.416 for unmasking on a fixed horizon. The weighting also has a single-variable comparison, gaining 4.6% (see our reply to reviewer ngCk, point 1).

On the size of the gains. The two numbers differ in what they measure. The 2.1% and 3.8% the reviewer quotes are DScout (best) against DScout (16) at concurrency 1 in Table 4, that is block-size selection alone (1.31% and 2.72% under the revised aggregation, as the per-cell table in the appendix gives them); the end-to-end gain over DFlash is +8.3% on Qwen3-4B and +5.8% on Qwen3-8B. Selection alone is indeed small and seven of the sixteen cells are zero; its value is that one checkpoint need not be retrained per block size, while buying a larger gain on a few workloads (9.92% in the largest cell).

On "§3.1 affects only MAT". That is correct, and it is why we treat the two sides separately. The argument is that MAT alone does not determine throughput, so the training side raises MAT and the inference side picks a point in the trade-off between MAT and per-round latency. The measurements: across the four training recipes ordered by MAT, throughput increases in the same order (6.18 gives 531.5 tokens per second, 6.33 gives 541.3, 6.49 gives 555.2, 6.98 gives 595.5), so at a fixed inference configuration an improvement in MAT does translate into throughput; whereas changing the inference configuration can move MAT and throughput in opposite directions (two cells of Table 4 are faster at lower MAT).

On enumerating all options. Only one of the three dimensions is a search. Conditioning depth is not searched - the FLOPs analysis and the profiling at fixed MAT set it to all available layers. Drafter depth is located by binary search using unimodality. Only the inference block size is swept, and that dimension needs no retraining at all; its cost is reported in point 2 below.

2. The search cost is unreported: how many depths must be trained, how many block sizes evaluated, how much GPU time, and how much does unimodality save.

We thank the reviewer for the reminder that search cost should be reported. We agree, and the cost is best read one dimension at a time.

Drafter depth is the only dimension that needs training per candidate, and unimodality is what makes binary search on the sign of the increment valid there. Over the 2-12 range that locates the peak in four to six trainings instead of the 11 an exhaustive sweep needs, a saving of 45% to 64% training time. We trained eight depths (2-8 and 12) to draw the curve and verify unimodality.

Inference block size needs no extra training at all: one mixed-horizon checkpoint serves every candidate. Without mixed horizons each candidate size would need its own drafter trained, which is what this dimension would otherwise cost.

Target conditioning depth is not searched. On the FLOPs analysis and the fixed-acceptance profiling already in §3.2 it is set directly to all available layers.

Evaluation cost. One cell is one drafter at one block size and one concurrency level, over the seven tasks the campaign runs, 750 requests; the sweep is 112 cells, four targets by four concurrency levels by seven sizes from 8 to 32. This is a one-off offline evaluation with no training anywhere in it, and the chosen configuration is then reused in serving.

3. The memory cost of conditioning on all target hidden states is not quantified.

We thank the reviewer for asking for this to be quantified. We measured peak inference memory on two targets at four concurrency levels: for each target we take a pair of drafters that differ only in the number of conditioning layers, and the cost of full-layer conditioning is 4%-10% across concurrency levels (+5.8% to +10.2% on Qwen3-8B, +4.2% to +6.3% on Qwen3-4B). The full table, the training and storage sides, and the trade-off between layer count and acceptance length are answered together in our reply to reviewer E7UG, point 2.

AuthorsOpenReview ↗

Official Comment by Authors

4. D-PACE is not cited.

We thank the reviewer for catching this omission, which is ours. D-PACE (arXiv:2605.18810) is indeed prior work and is now cited in the weighting paragraph of §3.1.2, item 1 of the general response. Below we set out the substantive difference between the four weightings.

The four differ in which quantity enters the per-position weight:

Weight at position kWhat is multiplied in
DFlash, fixed decayexp(-(k-1)/gamma), independent of the data and of the drafter
CAT (PARD-2)the target's probability of the ground-truth token
D-PACEthe drafter's probability of the token the target selects, smoothed as (1-alpha)q+alpha with alpha=0.5, times a continuation term
AWL (ours)the standard speculative-sampling acceptance ratio at the drafter's own greedy proposal, min(1, P(y)/Q(y)) with y = argmax Q

The three earlier weights are all proxies for the acceptance probability, evaluated along a reference path unrelated to what the drafter actually proposes: CAT multiplies the target's probability of the ground-truth token, and D-PACE the drafter's probability of the token the target selects (D-PACE itself calls this a smooth proxy for the acceptance probability). Our min(1, P/Q) is the standard speculative-sampling acceptance ratio, and it acts on the token the drafter would actually propose. The two kinds of quantity diverge exactly where it matters: the drafter can be confident about a token the target would reject, in which case the weight along a reference path need not shrink while ours does. The comparison with the weighting as the only variable is in our reply to reviewer ngCk, point 1.

5. Why does B16 to B32 increase the share of rounds with acceptance length 1-4? Why that interval? And how do mixed horizons solve it?

We thank the reviewer for pressing on the mechanism.

The quantities behind the figure. At B=16/bs=16 the 1-4 share is 52.3%, which rises to 57.3% at B=32/bs=32, while rounds above 16 rise from 0 to 8.6%. Other cut points give the same conclusion: 1-8 rises from 73.4% to 77.8%, and 1-2 from 30.5% to 34.4%. So 1-4 is not a chosen threshold; the shading in the figure only marks the two ends of the distribution.

Our reading of the cause. Once the block widens from 16 to 32, every position must be predicted inside a wider fully masked block: the available context is unchanged while the number of positions to predict at once doubles, which makes each individual position harder. At the same time the same amount of supervision is spread over more than twice as many positions, so the share reaching the early positions shrinks. The early positions therefore face a harder task with less supervision, and their error probability rises. Since verification accepts only a consecutive prefix, an error at any early position truncates the whole round into the short interval - which is where the rise in the 1-4 share comes from.

Why mixed horizons address it. Training equally often on the four horizons {8,16,24,32}, positions 1-8 are supervised under all four, positions 9-16 under three, 17-24 under two and 25-32 under one, so the early positions receive the most supervision; at the same time the model sees blocks of width 8 and 16 again. The measurements match this expectation: the 1-4 share falls from 57.3% to 55.3% while rounds above 16 hold at 8.9%; adding intra-block unmasking takes 1-4 down to 53.1% with rounds above 16 at 9.0%.

6. Table 2 is confusing: why is fixed-32 training worse than fixed-16 training at inference bs16? And the same at bs24?

We thank the reviewer for pointing out this anomaly. The observation is correct, and the re-measured numbers make it clearer: at bs16 fixed-32 reads 5.29 against 6.01 for fixed-16, and at bs24 it is 5.65 against 5.87. The two cases are two ends of the same phenomenon - the gap narrows from 0.72 at bs16 to 0.22 at bs24, so the closer the inference block size is to the 32 used in training, the smaller the gap.

The cause is the same as in the previous point. A fixed-32 drafter has only ever predicted inside 32-wide fully masked blocks; asking it to draft at block 16 asks it to fill a block it has never seen in that form, whereas fixed-16 was trained for exactly that layout. This is the cost of a mismatch between training and inference block sizes, and DFlash reports the same phenomenon: their block-16 drafter run at block 8 drops from 6.33 to 5.09 in acceptance length on MATH-500.

Why we read this as supporting evidence rather than a problem: precisely because a single fixed horizon pays this much at other block sizes, we train one model with mixed horizons so that the same checkpoint covers the whole range from 8 to 32, instead of maintaining one checkpoint per block size.

Thank you for the clarifications and additional results. They largely address my questions about the logic of the method. However, I still find the overall contribution limited, and the gains are smaller after the corrections. My concerns about attributing the ablation gains to individual changes and the memory cost of full target conditioning are also not fully resolved. I will keep my rating.

Reviewer_ngCkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DScout trains and configures a block-diffusion drafter by treating two quantities as products. For training it writes each position's loss as an inclusion indicator times a weight times a cross-entropy term, then combines equal-frequency horizons of 8, 16, 24 and 32 tokens, training blocks that reveal a random subset of their future positions, and weights formed from the running product of acceptance ratios, so that a position is supervised in proportion to how likely verification is to reach it. For inference it factorizes throughput as accepted tokens per round over round latency, and searches target-conditioning depth, drafter depth and inference block size. Across six benchmarks it reports mean speedups of 3.84x and 4.14x over autoregressive decoding on Qwen3-4B and Qwen3-8B, and 9.1% and 7.1% over DFlash.

Critical strengths
  1. The evaluation is broad and its hygiene is stated rather than implied: four target models including one outside the Qwen family, four concurrency levels, six benchmarks, two temperatures, five repeats with sample standard deviations in Table 5, and all three baselines retrained on the same 1.4M Open-PerfectBlend examples under their official recipes. The one asymmetry, PARD-2 on its own Transformers runtime with its own matched AR baseline, is declared in Appendix A.1 rather than left for a reader to discover. Several results that cut against the paper's own story are also reported as results: adding unmasking over the {4,8,16,32} horizon set lowers MAT (Table 1, 6.228 to 6.197), and Section 4.1 states that nine benchmark cells lose speedup when block size moves from 16 to 32 even as MAT rises.

  2. The estimator inside the acceptance weight is the quantity the verifier applies, not a correlate of it. With alpha_j = min(1, P_j(yhat_j) / Q_j(yhat_j)), the running product is the probability that the draft prefix survives verification under the rule actually applied at inference, whereas a target-confidence score is a proxy for that probability; deriving it at stop-gradient and folding it into a weighted mean keeps the objective a reweighting of cross-entropy rather than a heuristic scale factor. The supporting analysis is checkable in the same way: the conditioning cost comes with a FLOPs account that names what it excludes, the round-latency model is an affine fit with R^2 = 0.9975, and Algorithm 1 gives the training loop including epsilon and the per-anchor normalization.

Critical weaknesses
  1. The largest training rung re-derives an objective that one of the paper's own baselines already optimizes, and the text does not say so. PARD-2 (An et al., 2026) trains with a loss weighted by s_k := prod_{j<k} c_j with s_0 := 1, called in that paper "a stop-gradient weight" that "approximates the probability that position k is reached during verification", and adopted to shift "the focus from token prediction accuracy to the overall acceptance length". That is the same construction as w^AWL with a different estimator: target confidence c_j against the acceptance ratio min(1, P_j/Q_j). Section 3.1.4 names only DFlash's fixed decay w_k = exp(-(k-1)/gamma) as prior art, and Section 5 cites PARD solely as "parallel masked-token prediction", so a reader cannot see that AWL, at +0.491 of +0.968 the largest single rung, lands on a published objective. The estimator substitution may be an improvement, but it is neither named, motivated, nor ablated against what it replaces.

  2. The inference factorization and two of its three dimensions are already in DFlash. DFlash Eq. (1) gives L = (T_draft + T_verify) / tau with tau including the bonus token, and states that "speedup improves either by increasing the expected acceptance length tau or by reducing the drafting overhead". DFlash Table 6 sweeps drafter depth at 3, 5 and 8 layers and Table 7 sweeps target-feature count at 3 against 5, each reporting tau and speedup, and concludes that "the optimal number of layers depends on the deployment setting". DFlash Table 8 shows a checkpoint trained at block 16 transferring to block 8, proposes dynamic block-size scheduling as the use of that property, and gives the mechanism this paper rediscovers at concurrency 32: a large batch makes verification cost-bound, so a shorter block wins. Contribution 2 attributes the factorization to Liu et al. (2024) and presents all three dimensions as newly identified. What remains DScout's own is that one mixed-horizon checkpoint stays usable from block 8 to block 32 rather than degrading, and that blocks 10, 12 and 14 lie outside the trained set, and neither point is separated from DFlash.

  3. Both headline percentages are single cells, and their conditions are not in the abstract. The 9.1% and 7.1% reproduce only as a ratio of the two six-task means, only at temperature 0, only at concurrency 1, and only with DScout at block 32 against DFlash at block 16; averaging the six per-benchmark ratios instead gives 8.3% and 5.9%, and at temperature 1 the same comparison gives 5.6% and 3.4%.

  4. The 9.75% gain is a maximum over seven candidates selected on the same prompts that produce it. Appendix A.2 states that each target-concurrency setting "selects one block size by maximizing equal-weight mean speedup across all six tasks" and that "selection uses the evaluation tasks themselves". At the cell that supplies the number, Llama-3.1-8B at concurrency 32, Table 5 gives a sample standard deviation of 0.161 over five repeats while the gain is 0.23, about 1.4 of those deviations; across the sixteen model-concurrency cells the mean gain is 1.5%.

  5. Figure 4 cannot be reconciled with Tables 1 and 2, and those two tables do not agree with each other. The ladder starts at 6.0 and Section 4.2 says the ablation is measured "at a common inference block size of bs = 16"; at that size Table 2 makes the B=16 to B=32 rung -0.72 (6.01 to 5.29), where Figure 4 labels +0.171. Reading each rung at its own best block size instead gives +0.47 for that rung, and for the unmasking rung +0.055 from Table 1 (6.263 against 6.208) where the figure labels +0.155. Table 1 and Table 2 also disagree on the same nominal configuration, mixed horizons at block 32 and no unmasking, reading 6.208 and 6.38, with fixed 32 reading 5.942 and 6.18, and neither caption states its aggregation.

  6. The largest rung has no isolating arm, and the horizon rung moves two things at once. The ablation is cumulative only: there is no AWL-alone arm and no leave-one-out at the full recipe, so "AWL producing the largest increment" is measured on a recipe that already contains the other two strategies. Appendix A.3 records that fixed-16 uses positional decay gamma = 7 while fixed-32 and the mixed set use gamma = 15, so the B=16 to B=32 rung changes horizon and decay together. The ladder is run at the 9t9d configuration and block 16, not at the 35t6d configuration the paper ships for Qwen3-8B.

  7. Four gaps block any cross-check against the literature. No absolute throughput is reported for any method, although each speedup divides by a different matched AR baseline and Appendix A.1 concedes that cross-runtime speedups "do not establish an absolute throughput ranking", while Figure 1 places all four methods on one axis. The GPUs are identified only by memory and bandwidth, and no drafter parameter count, numerical precision or training time appears anywhere. The cost of the configuration search is never reported even though the search runs the full grid, while Contribution 1 advertises the recipe as adding no inference-time computation.

Revisions: state the relation to CAT in Section 3.1.4 and add one arm, the same drafter with the cumulative product of target confidence in place of the acceptance ratio at matched budget, so that the estimator substitution is measured rather than asserted; give Figure 4's base configuration and aggregation, reconcile its five rungs with Tables 1 and 2, and state in each caption how the MAT values are aggregated; select block size on a held-out split of prompts or tasks and report the sixteen per-cell gains with intervals rather than the maximum alone; report absolute tokens per second for every arm from a single runtime, with drafter parameter counts, precision, GPU model and training time, and the wall-clock cost of the configuration sweep; add a leave-one-out at the full recipe and an AWL-alone arm, and either hold gamma fixed across the horizon rungs or report the horizon-only and decay-only steps separately.

Questions for authors
  1. How does AWL relate to PARD-2's CAT objective? If s_k there and w^AWL here are the same cumulative-product stop-gradient weight differing only in the per-position estimator, what does the acceptance ratio buy over target confidence, and how do the two compare head to head on one architecture at one budget?

  2. What is Figure 4's base configuration, and at which block size is each rung measured? Can the five rungs be reproduced from Tables 1 and 2? Separately, why do the two tables give 6.208 and 6.38 for mixed horizons at block 32 with no unmasking?

  3. Was block size selected on the prompts and tasks used for reporting? What is the gain under selection on a held-out split, and what is the spread of the sixteen per-cell selection gains rather than only its maximum?

  4. Training reveals a random subset of future positions, with a Beta(2,2) masking ratio and a 0.25 lower bound, while inference masks every position; DFlash builds its blocks so that training matches inference exactly. How does MAT move as the reveal fraction is varied, and does the mismatch cost anything at block 32?

  5. At Qwen3-8B, MATH-500, PARD-2 reads 1.75x with tau = 7.99, while the other five cells of that row give tau over speedup between 2.10 and 2.79 and this one gives 4.57; at 3.75x it would give 2.13. Is this a transcription error? The three Qwen3-4B, concurrency-1 cells of Table 3 also read 0.01 higher than the same cells of Table 4, with identical tau.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

1. AWL and PARD-2's CAT are the same cumulative-product stop-gradient weight, so what does the acceptance ratio buy over target confidence?

We thank the reviewer for pointing out this attribution gap. We accept the correction: the closest existing weighting was not named, and the changes in the paper are items 1.1-1.3 of the general response. Below we answer the substantive question.

What the two weights share. Only the form: a cumulative product along the acceptance chain, taken with a stop-gradient. What enters the product differs, and that is what the weight represents. Our starting point is that each position should be supervised in proportion to the probability that verification reaches it, and min(1, P(y)/Q(y)) at y = argmax Q is the standard speculative-sampling acceptance ratio, evaluated at the token the drafter would actually propose. Its cumulative product is the probability that this draft prefix survives verification under that rule. The target's probability of the ground-truth token, which CAT multiplies, measures how predictable the data is for the target; it is only correlated with how likely this drafter's proposal is to be accepted.

What changes when the estimator changes. Two things. First, the weight depends on the drafter's distribution Q and moves as the drafter trains, whereas CAT's weight is a property of the training data and does not change with the drafter. Second, the token being scored is the one the drafter would actually propose rather than the one in the data: when the drafter confidently proposes a token the target would reject, the acceptance ratio pushes down the weights of the later positions, while the target's confidence on the ground-truth token can still be high.

The head-to-head comparison at matched budget. The two checkpoints are trained for the same number of steps, both with mixed horizons {8,16,24,32} and intra-block unmasking, and differ only in whether the position weight is CAT or AWL; the evaluation setting is the same as Figure 4 (SGLang, concurrency 1, block 32, mean over six tasks). CAT reads 6.67 and AWL 6.98, so AWL is ahead by 0.31 (+4.6%), and this pair can be read directly against the last rung of Figure 4.

AuthorsOpenReview ↗

Official Comment by Authors

2. The inference factorization and two of its three dimensions are already in DFlash, yet Contribution 2 presents all three as newly identified.

We thank the reviewer for listing the overlap with DFlash item by item, and we accept these points. We adopt DFlash's architecture and say so in §2.2, and the latency factorization of their Eq. (1) together with the sweeps in their Tables 6-8 are all theirs. The mechanism at concurrency 32 is also already stated by them: in §5.5.4 they note that this transferability "enables dynamic block-size scheduling during inference" and that reducing the block size under compute-bound settings "can therefore yield better overall speedup", leaving adaptive block-size scheduling as future work. Our text did not previously attribute this passage to them; that is now fixed (item 1.4 of the general response).

On Contribution 2's wording. It states that we use this ratio for configuration selection and does not claim that we first identified the three dimensions.

DFlash's Eq. (1)-(3) describe where latency comes from; in §3.2 we turn the same relation into a selection objective, (t*, d*, bs*) = argmax tau(t,d,bs) / T(t,d,bs), and solve it per workload. What we add is how each dimension is handled in that argmax: a binary search on drafter depth, a cost argument that takes conditioning depth out of the search, and one checkpoint across the whole block-size range.

On drafter depth, their Table 6 compares three depths and concludes that the optimal number "depends on the deployment setting", since acceptance length rises with depth while drafting latency rises too. We agree, and make that conclusion locatable: since each candidate depth needs its own drafter trained, we prove that under diminishing acceptance gains and latency linear in depth the throughput tau_d / T_d is unimodal, so the peak can be found by binary search. What we reduce here is the search cost, not the observation that depth affects throughput.

On the number of target feature layers, their Table 7 compares 3 against 5 and concludes that more features raise acceptance length while adding drafting latency, so the optimum again depends on the setting. Taking all available layers walks that direction to its endpoint; what we add is why it can be taken without searching. Raising t from 9 to 35 adds 2.75% of the target projection and MLP FLOPs, and profiling at fixed MAT shows throughput varying by less than 1% across conditioning depths (0.53% from t=1 to t=35), so this dimension leaves the search.

On block size, their Table 8 covers two values, and their conclusion is that "A model trained with a larger block size generalizes well to smaller inference-time block sizes. However, the reverse does not hold" - generalization runs downward only, and even downward it costs acceptance length. The dynamic scheduling they propose is therefore confined to downward transfer and left as future work. Our result is that a single mixed-horizon checkpoint does not degrade anywhere from 8 to 32 and is equally usable at 10, 12 and 14, which never appear among the training horizons, so scheduling is not restricted by transfer direction and no checkpoint is kept per block size - the two points the reviewer also lists as ours.

3. The headline speedup states no conditions, and the numbers shrink when the aggregation is changed.

We thank the reviewer for pointing out that the headline lacked its conditions, and for recomputing it under other aggregations on our behalf.

What the abstract now states. Greedy decoding (temperature 0), concurrency 1, and each method at its own best draft configuration.

The convention we now follow throughout is DFlash's, the geometric mean of per-task speedups and the arithmetic mean of per-task MAT. All three aggregations are reproducible from the printed values of Table 3:

SettingRatio of means (original headline)Per-task ratios, arithmetic meanPer-task ratios, geometric mean
temperature 0, Qwen3-4B9.1%8.3%8.3%
temperature 0, Qwen3-8B7.1%5.9%5.8%
temperature 1, Qwen3-4B5.6%5.1%5.0%
temperature 1, Qwen3-8B3.4%2.7%2.6%

The two sets of numbers the reviewer quotes are the second and first columns. The direction is the same under all three, and the spread is within 1.3 percentage points.

AuthorsOpenReview ↗

Official Comment by Authors

4. The 9.75% gain is the maximum over seven candidate block sizes, the selection uses the very tasks the results are reported on, and at that cell the gain is only 1.4 repeat standard deviations.

We thank the reviewer for pointing out this selection bias.

On granularity. Each cell uses one block size for all six tasks, with no per-task choice, so the selection is one choice among seven candidates - already the conservative end of what the protocol allows, since per-task selection would give a larger number.

Selection on a held-out split. We reproduce the reviewer's 1.5% mean over the sixteen cells (1.47% under the geometric convention of point 3 above). Following the suggestion, we redid the selection with a leave-one-task-out protocol: the block size is chosen on five tasks and scored only on the sixth, averaged over the six folds, giving 1.47% in-sample and 0.89% held out. The per-cell gains with 95% intervals for all sixteen cells are in the revised appendix: selection returns the default bs16 in seven cells, and of the nine where it changes the block size, eight have intervals excluding zero. These percentages use the geometric convention of point 3 above, and the block size selected in each of the sixteen cells does not change under the other aggregations.

On the 1.4 repeat standard deviations. The two block sizes are run in the same five repeats, so we read the gain within each: 11.35%, 9.62%, 9.74%, 12.37% and 6.38%, a mean of 9.89% with a 95% t-interval of [+7.06%, +12.72%]. Taken over the two six-task means, the same cell reads 9.92%, which is the definition used for every reported speedup; the submitted 9.75% is that ratio under the arithmetic aggregation of point 3 above.

5. Figure 4 cannot be reconciled with Table 1 or Table 2, and the base configuration and the block size each rung is read at are not given.

We thank the reviewer for checking the figures and tables point by point. §4.2 described the ladder as a comparison "at a common inference block size of bs=16"; that sentence is wrong and we retract it - read that way the ladder indeed cannot be reconciled with Table 2, and the -0.72 the reviewer computes follows from it.

The actual reading is that each rung is evaluated at the block size matching its training horizon: rung 1 (fixed horizon 16, gamma=7) is read at bs16, and the other four (fixed horizon 32, mixed horizons {8,16,24,32}, plus intra-block unmasking, plus AWL) are all read at bs32.

The base configuration is a nine-layer drafter reading nine target layers, trained for six epochs on a 400K subset, read with the SGLang serving evaluator at concurrency 1 as a mean over six tasks. The five rungs are 6.0124, 6.1827, 6.3338, 6.4891 and 6.9795, so the increments are +0.170, +0.151, +0.155 and +0.490, which the revised figure labels to two decimals.

The disagreement between the two tables comes from different inference engines: Table 1 was evaluated with HF Transformers, while Table 2 and the recipe ladder use SGLang. The 6.208 and 6.38 the reviewer asks about (mixed horizons, block 32, no unmasking) are the same configuration read under the two engines. We have re-measured every cell of Table 1 under SGLang and updated it, and replaced the mixed-horizon row of Table 2 with re-measured values, so in the revised version Figure 4, Table 1 and Table 2 share one convention and can be aligned cell by cell.

6. The ablation in Figure 4 is only cumulative, with no AWL-alone experiment; the B16 to B32 rung changes both the horizon and gamma; the ablation uses 9t9d.

We thank the reviewer for these comments on the ablation design.

How the increments should be read. In a cumulative ablation each increment is by construction the marginal contribution of that item on the recipe built so far, which is how the paper reports them and how AWL's rung should be read.

The weighting is also isolated directly. The AWL-versus-CAT comparison in point 1 above holds the horizon distribution, masking and optimization fixed and changes only the estimator inside the weight, gaining 4.6%.

On gamma. It follows DFlash's setting for each block size and is the companion hyperparameter of that horizon, not extra tuning.

On the 9t9d base. The ladder is fixed at 9t9d so that only the training recipe varies within one configuration. The 35t6d model the main results report is trained with the complete recipe.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal/Revision Response

We thank the reviewers for checking the paper closely, and in several places for catching errors we had to correct. Major revisions

  1. Attribution and related work
  • 1.1 Revised §3.1.2: the existing position weightings are now cited together by what they share - they score a reference token (the one in the training data, or the one the target selects) rather than the one the drafter would propose - with the citations for PARD-2 and D-PACE added here. (ngCk 1, nUkn 4)
  • 1.2 Revised §2.2: added that PARD-2 additionally weights the loss by a cumulative product of target confidences. (ngCk 1)
  • 1.3 New appendix section: a single-variable measurement of AWL against CAT at matched budget, with only the estimator inside the weight changed, replacing the previous assertion. (ngCk 1)
  • 1.4 Revised §4 and §3.2: the mechanism by which large blocks raise verification cost at high concurrency, the reduction in acceptance length under a training-inference block-size mismatch, and the cost of an exhaustive depth sweep now all cite DFlash. (ngCk 2)
  1. Aggregation and headline numbers
  • 2.1 Revised Tables 3-4, Figure 1 and all captions: speedups are now the per-task geometric mean and MAT the per-task arithmetic mean, matching the Avg column of DFlash's Table 1; cells already rounded are no longer averaged, and each caption states its aggregation. (ngCk 3, 5 and 8)
  • 2.2 Revised abstract and §4: the headline reads 3.70x / 3.97x and +8.3% / +5.8% over the baseline, with three conditions stated - greedy decoding, concurrency 1, and each method at its own best draft configuration. (ngCk 3)
  • 2.3 Revised Figure 1: the PARD-2 bar is removed, since its speedup divides by an AR baseline measured in a different runtime; the main table keeps PARD-2 with a dagger note that the row is not comparable across rows. (ngCk 7)
  • 2.4 Revised Table 3, PARD-2 row: MATH-500 and the six-task mean updated from the re-measurement, 4.15x and 3.12x. (ngCk 8)
  • 2.5 Revised Tables 1-2: every cell of Table 1 re-measured under SGLang, the same engine as Table 2 and Figure 4; the mixed-horizon row of Table 2 replaced with re-measured values. (ngCk 5)
  1. Reporting of configuration selection and setup
  • 3.1 Revised §4 and new appendix table: block-size selection is reported with a held-out protocol, giving the per-cell gains and intervals for all sixteen cells - 1.47% in-sample, 0.89% held out, seven cells at zero, 9.92% largest - rather than the maximum alone. (ngCk 4)
  • 3.2 Revised §4.2 and Figure 4: the incorrect description of a comparison "at a common inference block size of bs=16" is removed; the text now states the block size each rung is read at, the aggregation, and that the ladder is trained on a 400K subset. (ngCk 5)
  • 3.4 New appendix entries: drafter parameter count, numerical precision, and an absolute-throughput table for all four methods. (ngCk 7)

We once again thank the Area Chair and reviewers for their valuable time and constructive feedback. These revisions cite the prior work behind our position weighting and our block-size discussion, correct the inconsistencies the reviewers identified in our own tables, and state the conditions and aggregation behind every reported number. All changes are highlighted in blue in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.