- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DScout, a framework to improve the efficiency of diffusion-based speculative decoding through both training and inference optimization for diffusion drafters. The paper points out that, since speculative decoding can only accept consecutive token prefixes, higher prediction accuracy does not necessarily lead to higher MAT, while higher MAT can not guarantee the greater throughput. Based on the observation, specifically, for training, it introduces mixed-horizon training, intra-block unmasking, and acceptance-weighted loss to improve the mean accepted tokens. For inference, it optimizes target-conditioning depth, drafter depth, and block size to balance MAT and latency. Experiments across multiple models and benchmarks show that DScout consistently outperforms DFlash.
Critical strengths
- This paper is written very clearly and concisely. Also, the paper provides rich technical details, including how to select hidden states and drafter depth.
- The proposed Acceptance-Weighted Loss is very reasonable and has good motivation. As speculative decoding can only accept consecutive token prefixes, a draft token can be accepted only if all preceding draft tokens have been accepted. It is natural and intuitive to adjust the loss weights based on prefix acceptance probabilities.
Critical weaknesses
- The method in this paper seems to be a stack of several small improvements in training and inference based on DFlash. Although technically solid, the novelty is incremental.
- During inference, DScout still uses DFlash's one-pass full-mask prediction instead of multi-round unmasking. However, during training, intra-block unmasking is used. Why does this seemingly inconsistent training-inference approach work? The authors should provide analysis and explanation.
- The authors chose to use more layers of target features. Although this doesn't change the computational cost much, the change in memory cost can be huge, significantly increasing memory consumption for both training and inference. For offline training, it will also put a lot of pressure on storage space.
- Compared to the baseline DFlash, DScout's improvement in speedup seems to be relatively limited or even worse, although DScout is largely built upon the existing mechanisms of DFlash and incorporates many optimizations for both training and inference.
Questions for authors
- For training, DScout adopts intra-block unmasking, while during inference, DScout still uses one-pass full-mask prediction rather than multi-round unmasking. How can this training-inference mismatch work?
- What is the actual memory overhead for using more target hidden states, especially during training?
- Although DScout incorporates several optimizations for both training and inference, the additional speedup gains over DFlash are relatively limited or even worse. The more detailed ablation studies on end-to-end speedup for each component is required to figure out which contributes most to the performance gains, which have little effect, and which even negatively affect performance.
2 comments
Thanks for the authors' rebuttal. The most concerns have been addressed. Although I still concern the actual novelty of the proposed method, I will raise my score to 3.
Official Comment by Authors
1. Training uses intra-block unmasking while inference uses full masking, and the cost of this mismatch is not measured.
We thank the reviewer for pointing out this inconsistency. We have measured the cost of the mismatch, and it runs in our favour: a drafter trained with intra-block unmasking is better under the full masking used at inference - on mixed horizons {8,16,24,32}, adding unmasking raises acceptance length from 6.33 to 6.49 (+0.16), and the evaluation itself is carried out under full masking.
Why this works. The inference-time condition is not approximated during training but is the most frequent one: a block is kept fully masked with a fixed probability, and only the remaining blocks draw a mask rate from a Beta distribution and reveal a random subset of positions, which provide context only and do not enter the prediction loss. Full masking is therefore both the inference condition and the most common training condition, and the partially revealed blocks act as data augmentation, letting the model see prediction contexts beyond full masking, as is standard in training diffusion language models.
2. The memory cost of the extra target hidden states is not given, for training or for inference; offline training would also consume storage.
We thank the reviewer for separating the training, inference and storage sides.
On the inference side we measured two targets at four concurrency levels. For each target we take a pair of drafters with the same drafter depth that differ only in the number of target layers read, at block size 32, reading peak serving memory on an 80 GB accelerator, which includes the KV pool SGLang preallocates at startup; the percentages are computed from the unrounded peaks:
The cost is 4%-10% across concurrency levels.
On the training side two quantities grow with the layer count. One is the captured target state, at layers x hidden width x 2 bytes per token, which on Qwen3-8B is 8 KiB per token per layer, so 40 KiB at 5 layers and 280 KiB at 35. The other is the fusion projection itself: its input width is layers x hidden width, so the drafter's parameter count grows linearly in the conditioning depth, and during training this part also carries gradients and optimizer state.
On the storage side these states are read from the target online and never written to disk, so there is no offline cache; the byte counts above are what caching offline would cost.
If memory is tight, the conditioning depth can be reduced with little loss: MAT rises 1.9% from 9 to 18 layers and only 0.7% more from 18 to 35 (same recipe and drafter depth, six-task mean at concurrency 1).
3. The ablation uses acceptance length rather than end-to-end speedup; and the improvement over DFlash is limited, with some settings worse.
We thank the reviewer for pointing out that the improvement over DFlash is limited. Our claim is not only about speedup, which varies with the target and the load: what we also want to leave behind is the factorization that produced the three techniques. The position-wise loss decomposes into inclusion, weighting and prediction factors, which decides which techniques to combine and anticipates that their gains will not simply add; the "Joint design" paragraph of §3.1.2 states this with the mechanism.
Which contribution is largest. We read the same ladder end to end, as six-task mean tokens per second at concurrency 1: 529.9 at fixed 16, 531.5 at fixed 32, 541.3 with mixed horizons, 555.2 with unmasking, 595.5 with AWL, so MAT rises 16.1% and throughput 12.4% across it. AWL is the largest on both, +0.49 in acceptance length and +7.3% in throughput.
On the settings where we are worse. In the serving table this happens at Llama-3.1-8B, concurrency 32: DScout (16) reads 2.33, below DFlash's 2.50. The reason is that the suitable block size on this target is shorter than 16 to begin with - DFlash uses a block-10 drafter here, which shows they also found the optimal block size to depend on the target and the load, except that using it requires training a separate drafter for block 10. The DScout (16) row is an unadapted fixed block 16, so it compares our unadapted configuration against their adapted one; selecting block 10 from the same checkpoint gives 2.57, which is ahead. This cell is precisely where our approach is useful.