acceptodds
Under review as a conference paper at ICLR 2027

JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting

35 views

Abstract

Speculative decoding (SD) accelerates autoregressive Large Language Models (LLMs) by drafting multiple tokens and verifying them in parallel, but it faces a scaling limitation: increasing the draft budget improves speed only when acceptance remains high and drafting overhead stays low. This ceiling has been difficult to break because prior head-based SD methods face a causality-efficiency dilemma. Autoregressive drafters produce path-conditioned candidates that are effective for tree speculative decoding with higher acceptance length, but their drafting cost grows with tree depth. Bidirectional block-diffusion drafters generate all positions in one pass, but their branch-agnostic marginals can form individually plausible yet mutually inconsistent trees, wasting budget and reducing acceptance. We propose JetSpec, a head-based SD framework that combines one-forward drafting efficiency with causal information flow across draft representations. JetSpec trains a causal parallel draft head using fused hidden states from the frozen target model, producing causally coupled draft distributions whose scores better align with the target model's autoregressive factorization. This enables JetSpec to convert larger draft budgets into longer accepted prefixes and higher end-to-end speedup. Across math, coding, and chat benchmarks on dense and MoE Qwen3 models, JetSpec consistently outperforms bidirectional-head and tree-based SD baselines. On H100 GPUs, JetSpec achieves up to speedup on MATH-500 and on open-ended conversational workloads, with further latency gains demonstrated through vLLM integration under realistic serving loads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Speculative Decoding, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MU4cOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JetSpec, a speculative decoding framework that combines parallel drafting with tree-based verification. Building upon DFlash, JetSpec replaces bidirectional attention with causal attention in the draft head to improve the consistency of predictions across future positions. The resulting draft distributions are used to construct candidate trees through best-first expansion, which are then verified by the target model in parallel.

Critical strengths
  1. The proposed method is relatively simple and introduces no additional sequential drafting passes. The experimental results show consistent improvements over DDTree across multiple benchmarks, including both dense and MoE models.
  2. The authors provide an optimized serving implementation with tree verification kernels and evaluate throughput under different batch sizes, demonstrating the practical potential of the proposed framework.
Critical weaknesses
  1. The proposed method appears to be largely a combination of existing techniques, including DFlash-style parallel drafting, standard causal attention, and best-first tree construction. While this combination is reasonable, the technical novelty seems relatively limited. More importantly, the experimental results do not convincingly establish the independent contribution of causal attention. In Table 4, a properly tuned diffusion head achieves 8.36× speedup on MATH-500, compared with 8.50× for the best causal-head configuration. This relatively small difference raises questions about whether the proposed architectural modification provides a substantial advantage over existing methods with appropriate training configurations.
  2. Although the paper reports substantial end-to-end speedups, the main comparisons against speculative decoding baselines focus on offline decoding, where large tree budgets can be particularly beneficial. The relative advantages under larger serving batch sizes remain unclear. The vLLM experiments in Table 8 evaluate JetSpec across different batch sizes, but only compare it against standard autoregressive decoding, rather than competing speculative decoding methods such as DFlash or DDTree. This is an important limitation because the paper's own results demonstrate that large tree budgets are not always advantageous under higher serving loads. For example, at batch size 16, reducing the tree budget from 128 to 32 improves the speedup from 2.80× to 3.81×. Consequently, it is unclear whether JetSpec maintains a meaningful performance advantage over existing speculative decoding methods under realistic batched serving workloads, especially when each method is allowed to select its optimal drafting budget. Given that the main motivation of JetSpec is to improve end-to-end decoding efficiency, I consider this an important missing comparison.
Questions for authors
  1. Could the authors provide end-to-end throughput comparisons against DFlash and DDTree under different serving batch sizes, preferably using the same serving framework and verification kernels? Each method should be evaluated with its own optimal drafting budget rather than a fixed large budget.
  2. While Appendix G provides a microbenchmark of the drafting cost for DFlash, it does not directly compare the latency of JetSpec and DDTree, nor does it account for tree construction overhead. Could the authors provide a breakdown of drafting, tree construction, and verification latency for JetSpec and DDTree under the same implementation and hardware settings, ideally across different batch sizes? This would help clarify the source of the reported performance improvements and whether they remain significant under batched serving workloads.
  3. Could the authors provide additional comparisons with DFlash under smaller drafting budgets (e.g., 4 and 8 tokens)? I know that tree drafting is an important component of JetSpec's contribution. However, since the current comparisons also involve different drafting structures, it would be interesting to evaluate both methods under the same chain-based setting at smaller budgets. This could help separate the benefits of the proposed causal attention mechanism from those introduced by tree drafting.

My current rating is 2. Nevertheless, I am open to raising my score if the authors can provide convincing evidence that JetSpec offers a meaningful end-to-end advantage over well-tuned speculative decoding baselines, particularly under batched serving workloads.

4 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer MU4c

We thank reviewer MU4c for recognizing the simplicity of the method, the consistent gains over DDTree, and the serving implementation. We address each concern below.


W1: Tuned diffusion baselines and Contributions

Carefully tuned bidirectional baselines. We additionally evaluate DDTree with τ=7\tau=7 under the same settings as Table 2 (Qwen3-8B, T=0, budget 256), using the full 800K-example training dataset, whereas the ablations in Table 4 use only the math split of Nemotron Post-Training Dataset V2.

JetSpec consistently outperforms the tuned diffusion baseline in tree drafting, retaining a 3–34% advantage in average accepted length across all six evaluated benchmarks.

MethodGSM8KMATH-500AIME25HumanEvalLCBMT-Bench
DDTree, γ=7\gamma=77.5910.049.577.087.604.43
JetSpec (Table 2)8.6210.769.827.788.795.94
JetSpec gain over DDTree γ=7\gamma=7+14%+7%+3%+10%+16%+34%

Comparison with DFlash / DDTree and Contributions. We would like to clarify the scope and novelty of JetSpec in comparison with DFlash and DDTree as follows:

  • DFlash’s main contribution is one-pass parallel drafting through a block-parallel head with target-model KV injection. JetSpec introduces causal attention across draft-depth representations, improving the alignment of one-pass parallel proposals with the target's autoregressive structure. This enables more effective scaling to deeper trees and therefore higher acceptance length without requiring sequential, branch-specific drafting passes.
  • DDTree is concurrent works. We include DDTree as a baseline and retrain all heads from scratch using the same target model, dataset, training budget, and tree-construction algorithm for fair comparison.

This controlled comparison reveals JetSpec’s key advantage: causal tree drafting continues benefiting from deeper and larger trees, whereas DDTree-style drafting saturates earlier. For example, increasing the tree budget from 64 to 256 raises JetSpec’s accepted length on MATH-500 from 7.42 to 10.76, compared with 7.16 to 9.81 for DDTree. More concretely, we report the optimal tree configuration for each draft budget using best-first expansion with cumulative log-probability scoring:

Draft BudgetDDTreeJetSpec
64Depth 16, varying widths (max width 3)Depth 18, varying widths (max width 3)
128Depth 17, wider tree (max width 5)Depth 20, wider tree (max width 4)
256Depth 17, wider tree (max width 7)Depth 23, wider tree (max width 6)

As a result, under a larger budget, JetSpec can achieve higher acceptance length by having tokens accepted at deeper depths.

JetSpec continues to convert larger budgets into deeper trees, even beyond the maximum depth seen during training (training depth=16), whereas DDTree primarily allocates additional budget to width and saturates at depth 17. Consequently, JetSpec reaches candidates at deeper positions and achieves longer accepted prefixes under larger budgets.


W2, Q1: Batched serving against DFlash and DDTree

Serving at larger batch sizes. JetSpec primarily targets latency-sensitive decoding rather than peak throughput under saturated workloads. We extend our vLLM serving evaluation on B200 to batch size 32, comparing JetSpec against DDTree and DFlash.

B200, vLLM, aggregate throughput (tokens/s); speedup over AR in parentheses.

MethodBudgetBS=1BS=2BS=4BS=8BS=16BS=32
AR–2223165127751,2962,258
DFlash16606 (2.73$\times$)891 (2.82$\times$)1,442 (2.82$\times$)2,194 (2.83$\times$)3,139 (2.42$\times$)4,000 (1.77$\times$)
DDTree64760 (3.42$\times$)1,078 (3.41$\times$)1,628 (3.18$\times$)2,254 (2.91$\times$)2,884 (2.23$\times$)3,241 (1.44$\times$)
DDTree128728 (3.28$\times$)992 (3.14$\times$)1,396 (2.73$\times$)1,781 (2.30$\times$)2,102 (1.62$\times$)2,193 (0.97$\times$)
JetSpec64766 (3.45$\times$)1,087 (3.44$\times$)1,641 (3.20$\times$)2,272 (2.93$\times$)2,907 (2.24$\times$)3,267 (1.45$\times$)
JetSpec128775 (3.49$\times$)1,056 (3.34$\times$)1,486 (2.90$\times$)1,896 (2.45$\times$)2,238 (1.73$\times$)2,334 (1.03$\times$)

These results demonstrate that JetSpec's advantage is most pronounced in the latency-sensitive, low-load regime, while remaining competitive with DDTree at higher concurrency. At higher concurrency, DFlash and AR become more throughput-efficient as they assign a lower compute budget for each request. Nevertheless, it doesn’t compromise JetSpec’s latency benefits discussed in the paper at such moderate loads.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer MU4c (Part 2)

Q2: Latency breakdown

JetSpec and DDTree share the same tree verification process and kernel, differing only in the draft head. We provide a per-round latency breakdown under identical settings.

Verification dominates runtime (~80%), with nearly identical verification and tree-construction costs for both methods. JetSpec's causal draft head adds only 1.04 ms (1.7% of round latency), resulting in a 1.3% increase in total round latency.

Qwen3-8B, MATH-500, T=0, budget 256, offline engine, single A100, batch size 1. Latency in ms per round; acceptance includes selecting and committing the accepted path.

MethodDraftTree constructionVerificationAcceptRound
DDTree7.821.3447.222.5858.95
JetSpec8.861.3446.922.6059.72

JetSpec's speedup therefore comes primarily from higher acceptance length, rather than lower per-round cost. At budget 256 on MATH-500, JetSpec achieves 9.7% higher accepted length (10.76 vs. 9.81) and 9.8% higher end-to-end speedup (9.64×9.64\times vs. 8.78×8.78\times) in Table 2. This demonstrates that causal attention improves drafting quality with negligible additional overhead.


Q3: Chain comparison at small budgets

To isolate the contribution of causal attention from tree drafting, we additionally compare JetSpec and DFlash under identical chain-based drafting settings at budgets 4 and 8.

JetSpec consistently outperforms DFlash across all four benchmarks, achieving 15-18% higher accepted length at budget 4 and 16-21% at budget 8. At budget 16, matching the training block size, the two methods perform comparably. However, when extending to budget 32, JetSpec regains a 13-19% advantage (Table 1).

Qwen3-8B, chain drafting, T=0, average accepted length. Block size equals draft budget. Results at budgets 16 and 32 are reported in Table 1.

MethodBlock sizeGSM8KMATH-500HumanEvalMT-Bench
DFlash42.863.072.722.36
JetSpec43.373.533.212.72
DFlash84.204.943.723.00
JetSpec85.025.724.513.51
DFlash166.017.835.184.03
JetSpec166.007.755.323.98
DFlas325.276.894.533.61
JetSpec †326.148.235.354.09

The most interesting observation is that causal attention improves generalization to block sizes both smaller and larger than the training block. At trained length = 16, DFlash is very comparable or slightly outperforms JetSpec. There is an almost U-shaped pattern in JetSpec's advantage over DFlash.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

The new results have substantially clarified the contributions of JetSpec:

  1. Causal drafting: The controlled chain-based comparisons at smaller budgets provide useful evidence that causal attention can improve drafting quality independently of tree construction.
  2. Tree scaling: The comparisons with tuned DDTree better demonstrate JetSpec's ability to benefit from larger budgets and deeper candidate trees.
  3. Serving efficiency: The batched serving experiments and latency breakdown clarify the practical trade-offs. I appreciate the authors' acknowledgment that JetSpec primarily targets latency-sensitive, low-concurrency and large tree settings, while DFlash can be more throughput-efficient at higher concurrency.

I have a few minor clarification questions:

  • Chain-based comparison: Could the authors confirm whether the budget-16/32 results referenced from Table 1 also use identical chain-based drafting for both methods? This would help establish whether the observed U-shaped trend can be attributed entirely to causal attention.
  • Tuned DDTree: Some results for the newly tuned DDTree are lower than those in the original Table 2, particularly on MT-Bench (4.43 vs. 5.41). Could the authors briefly clarify the configuration differences and this performance variation?

These are minor clarifications and do not change my overall assessment. Although the methodological novelty remains relatively limited and the serving gains over DDTree (with 64-node tree) are modest, the additional evidence has addressed my main concerns and better established the practical value and applicability of JetSpec.

I am therefore raising my rating from 2 to 3.

Thank you Reviewer MU4c for recognizing the additional evidence and thoughtful the follow-up. We are happy to clarify both points.

1. Chain-based comparison. Yes, the budget-16/32 results in Table 1 use the same chain-based drafting procedure as our new budget-4/8 experiments for both DFlash and JetSpec. All four budgets use linear drafts without tree expansion, under identical verification and evaluation settings. Therefore, the observed U-shaped trend is independent of tree construction.

**2. Tuned DDTree performance on MT-Bench. Table 2 uses γ=0\gamma=0, whereas the new experiment uses γ=7\gamma=7, with the same tree construction algorithm. Changing γ\gamma affects the bidirectional head's learned draft distributions and consequently the resulting candidate trees, potentially leading to performance variations across benchmarks. This is consistent with our finding in Table 4: the bidirectional head is more sensitive to loss weighting using tree drafting, whereas JetSpec's causal head is substantially more robust to γ\gamma. The new full-scale experiment further illustrates this.

We appreciate the helpful discussion and will make the experimental configurations more explicit in the revised manuscript.

Reviewer_NZ93OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JetSpec introduces a causal parallel tree drafting framework for speculative decoding. By predicting multiple draft depths in parallel during a single forward pass, and employing a causal mask to ensure a left-to-right flow of information across draft positions. It generates more consistent candidate trees, thereby increasing the acceptance rate and reducing the cost of the drafting process.

Critical strengths

1.The proposed approach combines efficient parallel drafting with tree-based speculative decoding, achieving strong acceleration, particularly with large draft budgets.

2.The paper provides extensive experimental evaluations across multiple tasks, model architectures, and hardware configurations, including practical serving experiments.

Critical weaknesses
  • Although JetSpec introduces causal attention across draft positions, its predictions are not explicitly conditioned on the actual tokens along each candidate branch. As acknowledged in Appendix A.4, off-argmax branches may still inherit inconsistent conditional information. Therefore, the proposed architecture does not fully resolve the causal inconsistency problem.

  • Limited evidence for the effectiveness of the core architectural contribution. Table 4 shows that a properly tuned diffusion head achieves 8.36x speedup on MATH-500, nearly matching the causal head's 8.40x. This raises concerns about whether the improvements primarily originate from causal masking rather than baseline training configurations. More comprehensive comparisons against properly tuned baselines are needed.

Questions for authors
  1. How does JetSpec ensure branch-level causal consistency when different branches share the same depth-wise draft distributions? Can the authors further clarify the correctness of non-greedy speculative sampling under this design?

  2. Could the authors compare JetSpec against diffusion-based baselines with optimized loss weighting across all benchmarks? This would help isolate the actual contribution of causal masking.

  3. JetSpec's speedup decreases under larger batch sizes, particularly with large draft budgets. Although the paper mentions dynamic budget scheduling as future work, do the authors have any concrete ideas or potential solutions to mitigate this performance degradation in high-concurrency serving scenarios?

6 comments
AuthorsOpenReview ↗

Response to Reviewer NZ93

We thank reviewer NZ93 for recognizing the combination of parallel drafting with tree verification and the breadth of the evaluation. We address each concern below.


W1, Q1: Branch consistency and sampling correctness

We appreciate the reviewer's question and would like to distinguish causal conditioning through hidden representations from explicit conditioning on realized branch tokens.

Branch-level consistency. In JetSpec, logits are produced from hidden states rather than parent-token IDs through a separate causal conditional module. Thus, earlier draft representations causally influence later predictions in a single forward pass. We acknowledge that this does not provide explicit causal factorization on realized off-argmax branch prefixes. Nevertheless, latent-level causal conditioning improves proposal alignment even for some alternative branches. Appendix A.2 provides a representative comparison of surrogate-to-target log-probability gaps:

Branch RankJetSpec Gap (nats)Diffusion Gap (nats)
Rank 2+7.73+87.02
Rank 3+42.50−3.69
Rank 4+45.75+47.10
Rank 5+36.58+92.57
Mean Absolute Gap33.1457.60

JetSpec achieves smaller absolute gaps at 3 of 4 off-argmax ranks in this example, suggesting that latent causal coupling can improve alternative proposals despite latent-level branch conditioning.

Non-greedy sampling correctness. At temperature T>0T > 0, the target model samples its next token at each tree node from the target distribution conditioned on that node's prefix. A drafted child is accepted only if it matches this sample; otherwise, the sampled target token is emitted and verification terminates.

Consequently, every emitted token follows the target model's conditional distribution, regardless of how the draft tree was constructed. This is consistent with the distribution-preservation principle established in Lookahead Decoding (Fu et al., ICML 2024, Appendix B). Thus, off-argmax context mismatch affects acceptance efficiency, not sampling correctness.

AuthorsOpenReview ↗

Response to Reviewer NZ93 (Part 2)

W2, Q2: Tuned diffusion baselines and Contributions

Carefully tuned bidirectional baselines. We additionally evaluate DDTree with τ=7\tau=7 under the same settings as Table 2 (Qwen3-8B, T=0, budget 256), using the full 800K-example training dataset, whereas the ablations in Table 4 use only the math split of Nemotron Post-Training Dataset V2. JetSpec consistently outperforms the tuned diffusion baseline in tree drafting, retaining a 3–34% advantage in average accepted length across all six evaluated benchmarks.

MethodGSM8KMATH-500AIME25HumanEvalLCBMT-Bench
DDTree, γ=7\gamma=77.5910.049.577.087.604.43
JetSpec (Table 2)8.6210.769.827.788.795.94
JetSpec gain over DDTree γ=7\gamma=7+14%+7%+3%+10%+16%+34%

Comparison with DFlash / DDTree and Contributions. We would like to clarify the scope and novelty of JetSpec in comparison with DFlash and DDTree as follows:

  • DFlash’s main contribution is one-pass parallel drafting through a block-parallel head with target-model KV injection. JetSpec introduces causal attention across draft-depth representations, improving the alignment of one-pass parallel proposals with the target's autoregressive structure. This enables more effective scaling to deeper trees and therefore higher acceptance length without requiring sequential, branch-specific drafting passes.
  • DDTree is concurrent works. We include DDTree as a baseline and retrain all heads from scratch using the same target model, dataset, training budget, and tree-construction algorithm for fair comparison.

This controlled comparison reveals JetSpec’s key advantage: causal tree drafting continues benefiting from deeper and larger trees, whereas DDTree-style drafting saturates earlier. For example, increasing the tree budget from 64 to 256 raises JetSpec’s accepted length on MATH-500 from 7.42 to 10.76, compared with 7.16 to 9.81 for DDTree.

More concretely, we report the optimal tree configuration for each draft budget using best-first expansion with cumulative log-probability scoring:

Draft BudgetDDTreeJetSpec
64Depth 16, varying widths (max width 3)Depth 18, varying widths (max width 3)
128Depth 17, wider tree (max width 5)Depth 20, wider tree (max width 4)
256Depth 17, wider tree (max width 7)Depth 23, wider tree (max width 6)

As a result, under a larger budget, JetSpec can achieve higher acceptance length by having tokens accepted at deeper depths.

JetSpec continues to convert larger budgets into deeper trees, even beyond the maximum depth seen during training (training depth=16), whereas DDTree primarily allocates additional budget to width and saturates at depth 17. Consequently, JetSpec reaches candidates at deeper positions and achieves longer accepted prefixes under larger budgets.


Q3: High concurrency

Tree drafting trades spare GPU compute for fewer sequential target-model passes, making it particularly effective for latency-sensitive, low-load serving. As batch size increases and GPU utilization rises, the optimal tree budget decreases.

A practical mitigation is therefore load-adaptive tree budgeting, selecting the budget according to system load and hardware.

On our serving budget sweep on H100 (as provided in Table 8 in Appendix), the best budget is 128 up to batch size 4, 64 at batch size 8 and 32 at batch size 16. With this choice, JetSpec stays 3.49×3.49\times to 4.33×4.33\times faster than AR at every batch size, where a fixed budget of 128 falls to 2.80×2.80\times at batch size 16.

H100: Qwen3-8B, vLLM, MATH-500. Aggregate throughput (tokens/s); AR speedup in parentheses.

MethodBS=1BS=2BS=4BS=8BS=16
AR127.8163.3203.8246.2287.3
JetSpec, best budget553.3 (4.33×4.33\times)621.6 (3.81×3.81\times)742.9 (3.64×3.64\times)859.3 (3.49×3.49\times)1,094.6 (3.81×3.81\times)
Best budget1281281286432

Thank you for the detailed response. The explanation of sampling correctness under T>0 and the load-adaptive budgeting results address Q1 (sampling part) and Q3.

However, my main concern (W2) remains. The overall speedup of JetSpec is strong, but the contribution of the core novelty, causal masking, is still not convincingly isolated. In Table 4, a tuned diffusion head reaches 8.36x speedup on MATH-500, nearly matching the causal head’s 8.40x. The new DDTree comparison reports only average acceptance length, not wall-clock speedup, and longer acceptance does not necessarily translate into proportional speedup, especially with deeper trees. It also remains unclear why the two heads nearly tie in Table 4 but differ by 7% in acceptance length on MATH-500 in the new experiment. In addition, the rebuttal confirms that off-argmax branches are not conditioned on their realized tokens (W1), and the supporting evidence is a single example. I will therefore keep my score.

Reviewer_3BXeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces JetSpec, a speculative decoding method that applies a causal attention mask to a one-pass parallel draft head. Combined with forward-KL distillation and best-first tree search, it improves branch consistency during tree drafting without adding sequential overhead. The authors validate the method on Qwen3-8B and Qwen3-30B-A3B and integrate it into vLLM with custom attention kernels.

Critical strengths
  • Applying a causal mask to parallel drafting resolves rank-1 branch inconsistency without adding sequential forward passes.
  • The causal head removes the need to tune the position loss-decay hyperparameter gamma, making training much less sensitive than bidirectional diffusion heads.
  • The vLLM integration and custom paged tree-attention kernels are well executed and demonstrate strong wall-clock speedups on modern GPUs.
Critical weaknesses
  • The method is incremental, mainly swapping the bidirectional mask in DFlash for a causal mask alongside standard distillation and DDTree search.
  • With a single hidden state per depth, only the argmax path is properly conditioned while off-argmax branches still inherit mismatched contexts.
  • In Table 4, the diffusion baseline with gamma at 7 reaches 8.36x versus 8.40x for JetSpec, meaning Tables 1 and 2 overstate the gain by setting gamma to 0.
  • Evaluation only covers Qwen3 in non-thinking mode. Maybe test other model families and long reasoning tasks.
  • Training requires 800K target-generated sequences and large tree budgets degrade at higher batch sizes.
Questions for authors
  • In Table 4, the diffusion head with gamma set to 7 achieves 8.36x speedup compared to 8.40x for JetSpec. How much speedup gain remains across the benchmarks in Table 2 if DDTree is trained with gamma set to 7 instead of 0?
  • Because off-argmax branches still inherit hidden states anchored to the argmax token, how much does this context mismatch limit tree width scaling on high-entropy generation tasks?
  • Does JetSpec still outperform simple linear drafting when Qwen3 thinking mode is enabled or when the vLLM batch size increases to 32 or higher?
2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer 3BXe

We thank reviewer 3BXe for recognizing the causal parallel head, its robustness to the loss decay γ\gamma, and the vLLM integration. We address each concern below.


W1: Novelty and Comparison with DFlash / DDTree

We would like to clarify the scope and novelty of JetSpec in comparison with DFlash and DDTree as follows:

  • DFlash’s main contribution is one-pass parallel drafting through a block-parallel head with target-model KV injection. JetSpec introduces causal attention across draft-depth representations, improving the alignment of one-pass parallel proposals with the target's autoregressive structure. This enables more effective scaling to deeper trees and therefore higher acceptance length without requiring sequential, branch-specific drafting passes.
  • DDTree is concurrent works. We include DDTree as a baseline and retrain all heads from scratch using the same target model, dataset, training budget, and tree-construction algorithm for fair comparison.

This controlled comparison reveals JetSpec’s key advantage: causal tree drafting continues benefiting from deeper and larger trees, whereas DDTree-style drafting saturates earlier. For example, increasing the tree budget from 64 to 256 raises JetSpec’s accepted length on MATH-500 from 7.42 to 10.76, compared with 7.16 to 9.81 for DDTree. More concretely, we report the optimal tree configuration for each draft budget using best-first expansion with cumulative log-probability scoring:

Draft BudgetDDTreeJetSpec
64Depth 16, varying widths (max width 3)Depth 18, varying widths (max width 3)
128Depth 17, wider tree (max width 5)Depth 20, wider tree (max width 4)
256Depth 17, wider tree (max width 7)Depth 23, wider tree (max width 6)

As a result, under a larger budget, JetSpec can achieve higher acceptance length by having tokens accepted at deeper depths.

JetSpec continues to convert larger budgets into deeper trees, even beyond the maximum depth seen during training (training depth=16), whereas DDTree primarily allocates additional budget to width and saturates at depth 17. Consequently, JetSpec reaches candidates at deeper positions and achieves longer accepted prefixes under larger budgets.


W2, Q2: Off-argmax branches

We acknowledge that JetSpec does not achieve exact branch-wise conditioning for off-argmax paths. Nevertheless, latent-level causal conditioning improves not only dominant-branch coherence but also provides better alignment of some off-argmax branches, while preserving one-pass efficiency. Appendix A.2 provides a representative comparison of surrogate-to-target log-probability gaps:

Branch RankJetSpec Gap (nats)Diffusion Gap (nats)
Rank 2+7.73+87.02
Rank 3+42.50−3.69
Rank 4+45.75+47.10
Rank 5+36.58+92.57
Mean Absolute Gap33.1457.60

Empirically, this translates into JetSpec continuing to benefit from larger tree budgets under stochastic decoding. At temperature T=1T=1, accepted length increases consistently:

BenchmarkBudget 64Budget 128Budget 256Gain (64 → 256)
MATH-5006.608.369.01+36.5%
MT-Bench4.375.015.22+19.5%

These results demonstrate effective budget scaling despite latent-level off-argmax conditioning.


W3, Q1: DDTree with γ=7\gamma = 7

We additionally evaluated DDTree with γ=7\gamma = 7 under the same settings as Table 2 (Qwen3-8B, temperature 0, budget 256), extending training to the full 800k data scale. Whereas all ablation studies are conducted on the math split of Nemotron Post-Training Dataset V2.

The results show that JetSpec retains a consistent advantage even against the tuned diffusion baseline in the tree setting (all values are average accepted length).

MethodGSM8KMATH-500AIME25HumanEvalLCBMT-Bench
DDTree, γ=7\gamma=77.5910.049.577.087.604.43
JetSpec (Table 2)8.6210.769.827.788.795.94
JetSpec gain over DDTree γ=7\gamma=7+14%+7%+3%+10%+16%+34%

We also want to point out Table 4 primarily demonstrates JetSpec’s robustness to loss-decay tuning, rather than showcasing maximum achievable performance gap. Across γ\gamma values, JetSpec remains stable at 8.29 - 8.50$\times$ speedup, whereas the diffusion head varies substantially from 5.46×5.46\times to 8.36×8.36\times. Moreover, DFlash requires different γ\gamma values for different block sizes.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer 3BXe (Part 2)

W4, Q3: Other models, thinking mode, larger batches

Other models. We extend our evaluation to Gemma4-26B-A4B (a different model family) and Qwen3.6-35B-A3B (hybrid Gated DeltaNet). JetSpec outperforms DDTree on all five benchmarks for both targets, further demonstrating generalization beyond the original Qwen3 architectures.

Budget 256; each cell reports speedup / accepted length.

Target / MethodGSM8KMATH-500HumanEvalMBPPMT-Bench
Qwen3.6-35B-A3B
DDTree7.15 / 7.848.70 / 9.666.24 / 6.846.25 / 6.894.26 / 5.38
JetSpec7.56 / 8.279.55 / 10.726.78 / 7.426.58 / 7.244.42 / 5.66
Gemma4-26B-A4B
DDTree7.39 / 8.327.90 / 9.287.54 / 8.596.64 / 7.653.68 / 4.52
JetSpec8.04 / 9.148.76 / 10.218.35 / 9.407.55 / 8.744.09 / 5.16

Thinking mode. Despite training exclusively on non-thinking data, JetSpec retains strong performance with thinking enabled, achieving 1.58× the accepted length of DFlash linear drafting, compared with 1.42× in non-thinking mode.

Qwen3-8B, MATH-500, T = 0, one evaluation session.

MethodBudgetNon-thinking τThinking τ
DFlash (linear)167.733.58
JetSpec (tree)25610.955.64
Relative gain—1.42×1.58×

Serving at larger batch sizes. JetSpec primarily targets latency-sensitive decoding rather than peak throughput under saturated workloads. We extend our vLLM serving evaluation on B200 to batch size 32, comparing JetSpec against DDTree and DFlash.

B200, vLLM, aggregate throughput (tokens/s); speedup over AR in parentheses.

MethodBudgetBS=1BS=2BS=4BS=8BS=16BS=32
AR-2223165127751,2962,258
DDTree64760 (3.42)1,078 (3.41)1,628 (3.18)2,254 (2.91)2,884 (2.23)3,241 (1.44)
DDTree128728 (3.28)992 (3.14)1,396 (2.73)1,781 (2.30)2,102 (1.62)2,193 (0.97)
JetSpec64766 (3.45)1,087 (3.44)1,641 (3.20)2,272 (2.93)2,907 (2.24)3,267 (1.45)
JetSpec128775 (3.49)1,056 (3.34)1,486 (2.90)1,896 (2.45)2,238 (1.73)2,334 (1.03)

These results demonstrate that JetSpec's advantage is most pronounced in the latency-sensitive, low-load regime, while remaining competitive with DDTree at higher concurrency. At higher concurrency, AR is more throughput-efficient as they assign a lower compute budget for each request. Nevertheless, it doesn’t compromise JetSpec’s latency benefits discussed in the paper at such moderate loads.


W5: Training Data and Batch-Size Degradation

Training cost. The 800K target-generated sequences are produced once per target and shared across JetSpec, DFlash, and EAGLE-3. JetSpec and DFlash each require approximately 580 H100 GPU-hours for training, compared with 1,500 H100 GPU-hours for EAGLE-3.

Batch-size degradation. JetSpec primarily targets latency-sensitive, low-load decoding, where larger tree budgets are most effective. By adapting the budget to batch size (128 for batches 1-4, 64 for batch 8, and 32 for batch 16), JetSpec consistently achieves #3.49 - 4.33\times$ speedup over AR across all batch sizes in Table 8.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Discussion

We thank the reviewers and the area chair for the discussion. The experiments added during the discussion support the main claim of the paper: causal attention improves one-pass parallel tree drafting. In summary:

  • The causal head generalizes to draft lengths both shorter and longer than its training block of 16.
  • Its advantage holds against a bidirectional baseline with tuned loss decay (γ = 7). And we confirm the tree drafting advantage comes mostly from JetSpec’s capability of drafting and getting accepted deeper and thinner tree at fixed budgets compared to DDTree.
  • It holds on two further target models.

Below, we summarize the concerns and how we addressed them. The first three items are the main concerns of Reviewer 3BXe (rating 2), who has not replied yet. Some of the evidence for them is in our responses to the other reviewers.

  • Novelty and the contribution of causal attention (Reviewer 3BXe: W1; Reviewer NZ93: W2; Reviewer MU4c: W1, Q3). Causal attention improves drafting on its own, without a tree, and it lets the tree keep benefiting from larger budgets.
    • Chain drafting. JetSpec accepts 15–18% more tokens than DFlash at budget 4, 16–21% more at budget 8, and 13–19% more at budget 32. The two are tied at budget 16, the training block size.
    • Tree drafting. With the same target, data, training budget and tree construction, best-first search builds a deeper tree for JetSpec at budgets 64, 128 and 256. From budget 64 to 256, the accepted length on MATH-500 rises from 7.42 to 10.76 for JetSpec, against 7.16 to 9.81 for DDTree.
  • Off-argmax branches (Reviewer 3BXe: W2, Q2; Reviewer NZ93: W1, Q1). Off-argmax branches contribute a large share of the accepted tokens, and JetSpec extends them further than DDTree. In our follow-up reply to Reviewer NZ93, we measure these branches on all seven benchmarks. They account for up to 41% of the accepted draft tokens. Past the point where the accepted path leaves the top-1 path, JetSpec accepts 9% to 16% more tokens than DDTree (γ = 7) at T = 0, and 12% to 25% more at T = 1.
  • Tuned diffusion baseline (Reviewer 3BXe: W3, Q1; Reviewer NZ93: W2, Q2; Reviewer MU4c: W1). JetSpec retains its advantage over a diffusion baseline with tuned loss decay. We train DDTree with γ = 7 on the full 800K data and evaluate it in the setting of Table 2. At budget 256, the acceptance length of JetSpec is higher on all six benchmarks evaluated, by 3% to 34%. On MATH-500, the benchmark of Table 4, it is 7% higher. Table 4 shows the robustness of the causal head to γ: its speedup stays within 8.29× to 8.50×, while the diffusion head varies from 5.46× to 8.36×.
  • Other models and thinking mode (Reviewer 3BXe: W4, Q3). The gain generalizes beyond Qwen3 and to thinking mode. On Gemma4-26B-A4B and Qwen3.6-35B-A3B, JetSpec outperforms DDTree on all five benchmarks. With thinking mode enabled, the acceptance length of the JetSpec tree (budget 256) is 1.58× that of linear DFlash drafting (budget 16), against 1.42× without thinking in the same session.
  • Serving under load (Reviewer 3BXe: W5, Q3; Reviewer NZ93: Q3; Reviewer MU4c: W2, Q1, Q2). The advantage of JetSpec is largest in latency-sensitive serving at low load, and the causal head adds negligible cost. We add DFlash and DDTree to the vLLM evaluation on B200, up to batch size 32, and a breakdown of the round time. At high concurrency greater than a batch size of 32, DFlash is more throughput-efficient, and a tree budget that follows the load mitigates this. With the budget adapted to the batch size, JetSpec stays 3.49× to 4.33× faster than AR up to batch size 16 on H100 (Table 8).
  • Training cost (Reviewer 3BXe: W5). JetSpec trains at the cost of DFlash: about 580 H100 GPU-hours each, against 1,500 for EAGLE-3. The 800K sequences are generated once for each target and shared by the three methods.

We believe that these experiments and clarifications address the concerns of the reviewers.

Reviewers MU4c and NZ93 have replied to our responses, and each raised the rating from 2 to 3. We have not yet received a reply from Reviewer 3BXe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.