- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ProSpect, a self-speculative decoding method in which a single model drafts and verifies in one forward pass. It makes two contributions. First, Gumbel-noise conditioning represents pre-sampled noise as mask tokens and reuses it at verification, so parallel drafts approximate the joint conditional rather than per-position marginals. Second, a dynamic draft tree verifies multiple candidates per position and places a fixed number of branches at the most likely landing points, keeping the query size linear rather than quadratic in draft length. Implemented in vLLM with a FlashAttention cascade, ProSpect reports 3.9× speedup on Qwen3-8B and 3.1× on Gemma-4-31B on SpeedBench at batch size 1, outperforming DFlash, DSpark, and Gemma-4 MTP.
Critical strengths
-
Simple and effective idea for consistent parallel drafting. Encoding pre-sampled Gumbel noise as ordinary tokens and reusing it at verification lets a single forward pass draft from approximate conditionals rather than marginals. It also makes verifying multiple candidates at the same position trivial. The ablation (Table 3) shows clear acceptance gains that grow with draft length.
-
Practical, self-contained system. Combining a linear-size dynamic draft tree with a FlashAttention cascade in vLLM avoids both the quadratic query of prior self-speculation and the cost of serving a separate drafter. Training only touches LoRA adapters and mask-token embeddings, and the method delivers strong batch-size-1 speedups on two model families.
Critical weaknesses
- Lossy speedups compared against lossless baselines. The headline speedups come from ProSpect†, whose merged adapter changes the verifier and costs up to ~4 points on Qwen3-8B. The exact (gated) variant is slower than DSpark, so the claimed advantage over DFlash and DSpark depends on giving up exactness.
- Quantized sampler. The fixed set of 10 noise vectors replaces temperature sampling with a quantized sampler that can emit at most 10 tokens per position. This target is easier to draft for than the one the baselines accelerate, and its bias is never quantified.
- Repetition loops in long outputs. With thinking enabled, ProSpect falls into repetition loops several times more often than lossless methods (up to 16% of LiveCodeBench turns), which makes the accelerated model substantially less usable in exactly the long-reasoning settings it targets.
- Favorable experimental setup. Tree configurations are tuned on the evaluation sets, with a different one per benchmark suite, while baselines use default settings. The Qwen3-8B DFlash and DSpark drafters were not trained on thinking data, which inflates the thinking-mode margins.
Questions for authors
Please see the weaknesses.
2 comments
Improvements to Gated ProSpect
Since submission, we have improved the gated variant in two ways. Its verifier remains bit-identical to the base model. First, the gated adapter now computes its draft-row gate on the GPU, which removes a host synchronization at every step, and fuses the LoRA expansion with the gated addition into a single kernel. At the tree configuration of the paper, this lowers the step time from 11.4 to 9.7ms with thinking disabled, without changing the acceptance. Second, we continued training the gated adapter of the paper on its data together with non-thinking responses to the same prompts, which raises the accepted tokens per step by up to 3.5%. Under the protocol of Table~1 (Qwen3-8B, SpeedBench, H100, same tree configuration), gated ProSpect now outperforms DFlash and DSpark in both thinking modes:
Thinking disabled
| tok/step | ms/step | tok/sec | Speedup | |
|---|---|---|---|---|
| Autoregressive | 1.00 | 6.8 | 146.4 | 1.00× |
| DFlash | 3.59 | 8.9 | 397.0 | 2.71× |
| DSpark | 3.79 | 9.1 | 442.1 | 3.02× |
| Gated ProSpect, submission | 3.96 | 11.4 | 363.5 | 2.48× |
| Gated ProSpect, updated | 4.10 | 9.7 | 455.2 | 3.11× |
Thinking enabled
| tok/step | ms/step | tok/sec | Speedup | |
|---|---|---|---|---|
| Autoregressive | 1.00 | 7.0 | 143.5 | 1.00× |
| DFlash | 2.74 | 9.1 | 302.0 | 2.10× |
| DSpark | 3.30 | 9.0 | 392.3 | 2.73× |
| Gated ProSpect, submission | 3.98 | 11.1 | 402.8 | 2.81× |
| Gated ProSpect, updated | 4.01 | 10.0 | 459.6 | 3.20× |
On Qwen3-8B, distribution preservation therefore no longer requires a trade-off: the lossless gated variant is faster than every distribution-preserving baseline in both thinking modes.
We thank the reviewer for the review.
W1 (lossy speedups). Table 1 reports both variants: gated ProSpect is bit-identical to the base model, and the merged variant is marked as not exact. The merged variant reduces average accuracy by 1.26 points on Qwen3-8B and 0.32 points on Gemma-4-31B. Its largest single-benchmark difference is on GPQA-Main for Qwen3-8B, which corresponds to the ~4 points mentioned by the reviewer. The accuracy gap is caused by fine-tuning, not by speculation. Verification in ProSpect is exact with respect to the weights it runs, so every emitted token is the token the fine-tuned model itself would produce. We fine-tune on the models' own generations because the post-training data of Qwen3 and Gemma-4 are not public. If ProSpect were trained jointly with the model's post-training, the released model would itself be the verifier, and decoding with ProSpect would be bit-identical to decoding the released model. With access to the post-training data, we expect the accuracy gap to disappear.
Among methods that do not preserve the base model's outputs, ProSpect is both more accurate and faster. The I-DLM conversion of the same Qwen3-8B weights is less accurate (Table 2: 81.46 vs. 81.88) and about half as fast (Table 1: 1.88× vs. 3.88×). Nemotron-Labs-Diffusion-8B, a natively trained diffusion model, scores lower than ProSpect on every benchmark it reports and is slower in both of its decoding modes.
W2 (quantized sampler). Given the context, a position can indeed emit at most L distinct tokens. We quantified the resulting bias. We evaluated the ProSpect checkpoints on the reasoning benchmarks of Table 2 with noise sets of size L∈{1,5,10,100} and with unrestricted Gumbel noise, over the same five seeds. Eight-benchmark average accuracy (%):
*With one noise vector the seed has no effect, so the five runs are a single deterministic decode.
Every L>1 is within noise of unrestricted sampling, except for a small drop at L=5 on Gemma-4-31B. Only L=1 reduces accuracy clearly. The gap to the base model in Table 2 is already present with unrestricted noise, so it is caused by the adapter and not by the finite noise set.
W3 (repetition loops). Generations with repetition loops are removed from all speed measurements, for all methods, so they do not affect the reported throughput or acceptance. The number of removed turns per category and method is given in the Appendix (Tables 10, 14, 18 and 22). The loops occur with the Qwen3-8B adapter. Gemma-4-31B ProSpect shows no increase in repetition: across both suites and both thinking modes, about as few of its turns are removed as for its base model. The accuracies in Table 2 are computed over all generations without filtering, so the effect of the loops is included there.
W4 (experimental setup). Each model uses one tree configuration for all categories of a benchmark suite, selected from the sweep in the Appendix. The conclusion does not depend on this selection. All twelve dynamic tree configurations in the Qwen3-8B sweep (Appendix, Table 6) are faster than DFlash and DSpark in both thinking modes, as are all static configurations except K=7 with thinking disabled, which is on par with DSpark. As noted in Table 1, the released Qwen3-8B drafters were not trained on thinking data. With thinking disabled, where this does not apply, ProSpect reaches 3.88×, compared with 3.02× for DSpark.
If our responses address your concerns, we would appreciate it if you would consider raising your score. Please let us know if any concerns remain before the end of the discussion period.