- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes JetSpec, a speculative decoding framework that combines parallel drafting with tree-based verification. Building upon DFlash, JetSpec replaces bidirectional attention with causal attention in the draft head to improve the consistency of predictions across future positions. The resulting draft distributions are used to construct candidate trees through best-first expansion, which are then verified by the target model in parallel.
Critical strengths
- The proposed method is relatively simple and introduces no additional sequential drafting passes. The experimental results show consistent improvements over DDTree across multiple benchmarks, including both dense and MoE models.
- The authors provide an optimized serving implementation with tree verification kernels and evaluate throughput under different batch sizes, demonstrating the practical potential of the proposed framework.
Critical weaknesses
- The proposed method appears to be largely a combination of existing techniques, including DFlash-style parallel drafting, standard causal attention, and best-first tree construction. While this combination is reasonable, the technical novelty seems relatively limited. More importantly, the experimental results do not convincingly establish the independent contribution of causal attention. In Table 4, a properly tuned diffusion head achieves 8.36× speedup on MATH-500, compared with 8.50× for the best causal-head configuration. This relatively small difference raises questions about whether the proposed architectural modification provides a substantial advantage over existing methods with appropriate training configurations.
- Although the paper reports substantial end-to-end speedups, the main comparisons against speculative decoding baselines focus on offline decoding, where large tree budgets can be particularly beneficial. The relative advantages under larger serving batch sizes remain unclear. The vLLM experiments in Table 8 evaluate JetSpec across different batch sizes, but only compare it against standard autoregressive decoding, rather than competing speculative decoding methods such as DFlash or DDTree. This is an important limitation because the paper's own results demonstrate that large tree budgets are not always advantageous under higher serving loads. For example, at batch size 16, reducing the tree budget from 128 to 32 improves the speedup from 2.80× to 3.81×. Consequently, it is unclear whether JetSpec maintains a meaningful performance advantage over existing speculative decoding methods under realistic batched serving workloads, especially when each method is allowed to select its optimal drafting budget. Given that the main motivation of JetSpec is to improve end-to-end decoding efficiency, I consider this an important missing comparison.
Questions for authors
- Could the authors provide end-to-end throughput comparisons against DFlash and DDTree under different serving batch sizes, preferably using the same serving framework and verification kernels? Each method should be evaluated with its own optimal drafting budget rather than a fixed large budget.
- While Appendix G provides a microbenchmark of the drafting cost for DFlash, it does not directly compare the latency of JetSpec and DDTree, nor does it account for tree construction overhead. Could the authors provide a breakdown of drafting, tree construction, and verification latency for JetSpec and DDTree under the same implementation and hardware settings, ideally across different batch sizes? This would help clarify the source of the reported performance improvements and whether they remain significant under batched serving workloads.
- Could the authors provide additional comparisons with DFlash under smaller drafting budgets (e.g., 4 and 8 tokens)? I know that tree drafting is an important component of JetSpec's contribution. However, since the current comparisons also involve different drafting structures, it would be interesting to evaluate both methods under the same chain-based setting at smaller budgets. This could help separate the benefits of the proposed causal attention mechanism from those introduced by tree drafting.
My current rating is 2. Nevertheless, I am open to raising my score if the authors can provide convincing evidence that JetSpec offers a meaningful end-to-end advantage over well-tuned speculative decoding baselines, particularly under batched serving workloads.
4 comments
Rebuttal Response to Reviewer MU4c (Part 2)
Q2: Latency breakdown
JetSpec and DDTree share the same tree verification process and kernel, differing only in the draft head. We provide a per-round latency breakdown under identical settings.
Verification dominates runtime (~80%), with nearly identical verification and tree-construction costs for both methods. JetSpec's causal draft head adds only 1.04 ms (1.7% of round latency), resulting in a 1.3% increase in total round latency.
Qwen3-8B, MATH-500, T=0, budget 256, offline engine, single A100, batch size 1. Latency in ms per round; acceptance includes selecting and committing the accepted path.
| Method | Draft | Tree construction | Verification | Accept | Round |
|---|---|---|---|---|---|
| DDTree | 7.82 | 1.34 | 47.22 | 2.58 | 58.95 |
| JetSpec | 8.86 | 1.34 | 46.92 | 2.60 | 59.72 |
JetSpec's speedup therefore comes primarily from higher acceptance length, rather than lower per-round cost. At budget 256 on MATH-500, JetSpec achieves 9.7% higher accepted length (10.76 vs. 9.81) and 9.8% higher end-to-end speedup ( vs. ) in Table 2. This demonstrates that causal attention improves drafting quality with negligible additional overhead.
Q3: Chain comparison at small budgets
To isolate the contribution of causal attention from tree drafting, we additionally compare JetSpec and DFlash under identical chain-based drafting settings at budgets 4 and 8.
JetSpec consistently outperforms DFlash across all four benchmarks, achieving 15-18% higher accepted length at budget 4 and 16-21% at budget 8. At budget 16, matching the training block size, the two methods perform comparably. However, when extending to budget 32, JetSpec regains a 13-19% advantage (Table 1).
Qwen3-8B, chain drafting, T=0, average accepted length. Block size equals draft budget. Results at budgets 16 and 32 are reported in Table 1.
| Method | Block size | GSM8K | MATH-500 | HumanEval | MT-Bench |
|---|---|---|---|---|---|
| DFlash | 4 | 2.86 | 3.07 | 2.72 | 2.36 |
| JetSpec | 4 | 3.37 | 3.53 | 3.21 | 2.72 |
| DFlash | 8 | 4.20 | 4.94 | 3.72 | 3.00 |
| JetSpec | 8 | 5.02 | 5.72 | 4.51 | 3.51 |
| DFlash | 16 | 6.01 | 7.83 | 5.18 | 4.03 |
| JetSpec | 16 | 6.00 | 7.75 | 5.32 | 3.98 |
| DFlas | 32 | 5.27 | 6.89 | 4.53 | 3.61 |
| JetSpec † | 32 | 6.14 | 8.23 | 5.35 | 4.09 |
The most interesting observation is that causal attention improves generalization to block sizes both smaller and larger than the training block. At trained length = 16, DFlash is very comparable or slightly outperforms JetSpec. There is an almost U-shaped pattern in JetSpec's advantage over DFlash.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.
The new results have substantially clarified the contributions of JetSpec:
- Causal drafting: The controlled chain-based comparisons at smaller budgets provide useful evidence that causal attention can improve drafting quality independently of tree construction.
- Tree scaling: The comparisons with tuned DDTree better demonstrate JetSpec's ability to benefit from larger budgets and deeper candidate trees.
- Serving efficiency: The batched serving experiments and latency breakdown clarify the practical trade-offs. I appreciate the authors' acknowledgment that JetSpec primarily targets latency-sensitive, low-concurrency and large tree settings, while DFlash can be more throughput-efficient at higher concurrency.
I have a few minor clarification questions:
- Chain-based comparison: Could the authors confirm whether the budget-16/32 results referenced from Table 1 also use identical chain-based drafting for both methods? This would help establish whether the observed U-shaped trend can be attributed entirely to causal attention.
- Tuned DDTree: Some results for the newly tuned DDTree are lower than those in the original Table 2, particularly on MT-Bench (4.43 vs. 5.41). Could the authors briefly clarify the configuration differences and this performance variation?
These are minor clarifications and do not change my overall assessment. Although the methodological novelty remains relatively limited and the serving gains over DDTree (with 64-node tree) are modest, the additional evidence has addressed my main concerns and better established the practical value and applicability of JetSpec.
I am therefore raising my rating from 2 to 3.
Thank you Reviewer MU4c for recognizing the additional evidence and thoughtful the follow-up. We are happy to clarify both points.
1. Chain-based comparison. Yes, the budget-16/32 results in Table 1 use the same chain-based drafting procedure as our new budget-4/8 experiments for both DFlash and JetSpec. All four budgets use linear drafts without tree expansion, under identical verification and evaluation settings. Therefore, the observed U-shaped trend is independent of tree construction.
**2. Tuned DDTree performance on MT-Bench. Table 2 uses , whereas the new experiment uses , with the same tree construction algorithm. Changing affects the bidirectional head's learned draft distributions and consequently the resulting candidate trees, potentially leading to performance variations across benchmarks. This is consistent with our finding in Table 4: the bidirectional head is more sensitive to loss weighting using tree drafting, whereas JetSpec's causal head is substantially more robust to . The new full-scale experiment further illustrates this.
We appreciate the helpful discussion and will make the experimental configurations more explicit in the revised manuscript.
Rebuttal Response to Reviewer MU4c
We thank reviewer MU4c for recognizing the simplicity of the method, the consistent gains over DDTree, and the serving implementation. We address each concern below.
W1: Tuned diffusion baselines and Contributions
Carefully tuned bidirectional baselines. We additionally evaluate DDTree with τ=7 under the same settings as Table 2 (Qwen3-8B, T=0, budget 256), using the full 800K-example training dataset, whereas the ablations in Table 4 use only the math split of Nemotron Post-Training Dataset V2.
JetSpec consistently outperforms the tuned diffusion baseline in tree drafting, retaining a 3–34% advantage in average accepted length across all six evaluated benchmarks.
Comparison with DFlash / DDTree and Contributions. We would like to clarify the scope and novelty of JetSpec in comparison with DFlash and DDTree as follows:
This controlled comparison reveals JetSpec’s key advantage: causal tree drafting continues benefiting from deeper and larger trees, whereas DDTree-style drafting saturates earlier. For example, increasing the tree budget from 64 to 256 raises JetSpec’s accepted length on MATH-500 from 7.42 to 10.76, compared with 7.16 to 9.81 for DDTree. More concretely, we report the optimal tree configuration for each draft budget using best-first expansion with cumulative log-probability scoring:
As a result, under a larger budget, JetSpec can achieve higher acceptance length by having tokens accepted at deeper depths.
JetSpec continues to convert larger budgets into deeper trees, even beyond the maximum depth seen during training (training depth=16), whereas DDTree primarily allocates additional budget to width and saturates at depth 17. Consequently, JetSpec reaches candidates at deeper positions and achieves longer accepted prefixes under larger budgets.
W2, Q1: Batched serving against DFlash and DDTree
Serving at larger batch sizes. JetSpec primarily targets latency-sensitive decoding rather than peak throughput under saturated workloads. We extend our vLLM serving evaluation on B200 to batch size 32, comparing JetSpec against DDTree and DFlash.
B200, vLLM, aggregate throughput (tokens/s); speedup over AR in parentheses.
These results demonstrate that JetSpec's advantage is most pronounced in the latency-sensitive, low-load regime, while remaining competitive with DDTree at higher concurrency. At higher concurrency, DFlash and AR become more throughput-efficient as they assign a lower compute budget for each request. Nevertheless, it doesn’t compromise JetSpec’s latency benefits discussed in the paper at such moderate loads.