acceptodds
Under review as a conference paper at ICLR 2027

JetSpec++: Foresight Parallel Drafting with Self Refinements

43 views

Abstract

Speculative decoding's effectiveness is governed by two quantities: the acceptance rate of drafted tokens and the cost of producing them. We show the two objectives can be co-optimized with algorithm-system co-design, and present JetSpec++, which retains a strong learned draft model while hiding its cost almost entirely. JetSpec++ combines three components: intra-block self refinement, which trains the draft model to iteratively correct its own parallel predictions within a block of tokens; inter-block foresight drafting, which begins the next block before the current one is verified; and pipelined drafting and verification, which overlaps the two stages so drafting's marginal wall-clock cost approaches zero. In comparison with state-of-the-art speculative decoding methods, JetSpec++ attains a wall-clock speedup of up to 6.74× over autoregressive (AR) decoding and 1.41× over DSpark, with up to 1250 TPS single-request throughput on GB200, and an average acceptance length of 7.60 across seven benchmarks, while preserving the target model's output distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Speculative Decoding, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_hzfKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

JetSpec++ addresses the challenge of overlapping drafting and verification when a speculative drafter relies on target-model hidden states that are unavailable until verification completes. Its central contribution is foresight drafting: a learned adapter predicts the missing target representations, while a decoder estimates likely acceptance lengths and bonus tokens, enabling candidate drafts for the next round to be prepared and cached in advance. The method also trains the same causal parallel drafter to iteratively refine its predictions using clean-prefix, noisy-suffix training, allowing additional refinement passes to improve draft quality within the verification window. Target-side verification preserves the target model’s output distribution. Experiments on Qwen3-8B across seven benchmarks, together with component ablations, demonstrate the effectiveness of the design. In the two-GPU GB200 configuration, JetSpec++ achieves up to 6.74× speedup over single-GPU autoregressive decoding and 1.41× over single-GPU DSpark.

Critical strengths
  1. Predicted states retain 95–98% of the acceptance length achieved with true states, making the dependency-breaking approximation practically useful.
  2. Controlled ablations show that refinement benefits come from the proposed training, rather than simply running additional draft passes.
Critical weaknesses
  1. This work lacks comparisons with closely related methods. Comparisons with xPress, D²SD, and SwiftSpec under comparable resource budgets would help establish the advantage over existing refinement, adaptive branching, and asynchronous pipelining approaches.
  2. This work needs a more systematic sensitivity analysis of the number of refinement iterations, cache size, adapter refresh frequency, and GPU placement to clarify when additional drafting computation can be hidden behind verification and when it reduces the overall speedup.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hzfK

We thank reviewer hzfK for recognizing the predicted-state results and the controlled refinement ablations. We address each concern below. Where a concern overlaps with another reviewer's, we refer to our general response (G1 to G3). Jn-Rm means n refinement passes (Jn, as in the paper) with m adapter refreshes.


W1: Comparison with related methods

We add a direct comparison with xPress, because it is a Jacobi refinement method, a category that our baselines did not cover. D²SD and SwiftSpec fall within categories of our existing baselines, and each has one half of what JetSpec++ combines. D²SD has a parallel draft model that reads the target's states, as DFlash does, but it keeps drafting on the critical path. SwiftSpec is pipelined, as SSD is, but with an independent draft model that does not read the target's states. JetSpec++ pipelines a parallel draft model that reads the target's states. This is the hard case: these states are not ready when a pipelined draft starts, and foresight drafting predicts them (Section 4.2). The results follow: xPress directly, and D²SD and SwiftSpec through the baseline of their category.

  • xPress (Jacobi refinement): G2 of our general response gives the full comparison. In short, on one GPU each, JetSpec++ (J2) is 1.21× faster than xPress on GSM8K and 1.22× on MATH-500. xPress comes within 2% of DSpark's mean acceptance length τ. Its refiner adds 0.37 ms to every round, while JetSpec++ refines during verification.
  • D²SD (adaptive branching): D²SD (Zhang et al., 2026a) verifies five candidate blocks in each round (block size 16, four branches), on top of a DFlash-style draft model. On Qwen3-8B at temperature 0, it raises τ by 33% over its DFlash and the speedup by 19%, from 4.48× to 5.35× (its Table 5, mean of GSM8K, MATH, MBPP and MT-Bench), because its second draft and its larger verification add to every round. On the same four benchmarks, JetSpec++ raises τ by 23% over our DFlash and the speedup by 26%, from 4.48× to 5.66×, with one block of 16 tokens (Table 1, one GPU, budget 16). At budget 32 it reaches 6.26×.
  • SwiftSpec (asynchronous pipelining): SwiftSpec drafts asynchronously with an independent, off-the-shelf draft model (training-free), which cannot read the target's states. SSD in Table 2, a later method of this kind and our baseline for this category, also drafts asynchronously with an independent draft model, on its own GPU. On the same two GPUs, JetSpec++ (J2) is 3.75× faster than SSD on GSM8K and 4.32× on MATH-500 (Table 2).

W2: Sensitivity analysis

The submission studies each of these factors: the number of refinement iterations (Table 6), the cache size (Table 11), the adapter refresh frequency (Table 12), and when drafting is hidden behind verification (Eq. 3 and 4, Table 2). G1 of our general response adds GPU placement, and G3 a larger target. In short, at temperature 0, refinement saturates by three passes and the hit rate by cache size B = 16, and a second refresh still raises τ. We quote the key numbers of each study below.

  • Refinement iterations: after three passes, τ is within 1% of its value at the fixed point J∞ (Table 6, Appendix A). As served, the second pass raises the speedup from 5.22× to 5.54× on GSM8K and from 6.67× to 6.74× on MATH-500 (Table 2).
  • Cache size: at cache size B = 4, 16 and 64, the offline joint hit rate is .910, .984 and .992 on GSM8K. On MATH-500, it is .835, .967 and .995 (Table 11, Appendix C.3). A larger B also lengthens each draft pass, which leaves fewer refinement passes (Eq. 3).
  • Adapter refresh frequency: at three passes, a second refresh raises τ from 7.28 to 7.54 on GSM8K and from 9.02 to 9.22 on MATH-500 (Table 12, Appendix F). At temperature 0, it raises τ on all seven benchmarks. As served, Table 1 runs two refreshes (J3-R2) and Table 2 runs none (J2), because of the verification time budget (Eq. 3).

GPU placement. G1 of our general response compares one and two GPUs for both methods. In short, with its target and draft processes on one GPU, JetSpec++ (J2) is 1.18× faster than DSpark on both benchmarks. On two GPUs each, it is 1.37× faster on GSM8K and 1.34× on MATH-500, because drafting and verification do not share a GPU. At 8B, DSpark drafts and verifies in sequence, so a second GPU changes its throughput by at most 4%.

When additional drafting is hidden, and when it reduces the speedup. Drafting is hidden while the draft side fits in the verification time (Eq. 3). At 8B, the verification takes about 5.5 ms, and the two refinement passes of J2 are hidden: the draft side takes 4.86 ms. On the added Qwen3.8-27B target (G3), the verification takes 27.9 ms, and three passes with three refreshes are hidden: the draft side takes 19.62 ms. When the draft side takes longer than the verification, the excess adds to every round (Eq. 4). In that case, an extra pass or refresh reduces the speedup unless it raises τ by more.

AuthorsOpenReview ↗

Follow-up to Reviewer hzfK

Dear Reviewer hzfK, thank you again for your review. We have responded above to your questions on related methods and on sensitivity, and the revised PDF is now uploaded. With less than 24 hours left in the discussion period, we would appreciate it if you could let us know whether any concern remains, and we are glad to clarify. If our response addresses your concerns, we would be grateful if you could take this into account in your final assessment.

Reviewer_myKROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents JetSpec++, a speculative decoding framework that overlaps drafting the next block with verification of the current block. Its foresight drafting mechanism predicts target hidden states and possible verification outcomes, allowing candidate continuations to be prepared before verification completes. A causal parallel drafter, trained on clean-prefix and noisy-suffix inputs, further improves these candidates through Jacobi self-refinement. Experiments with Qwen3-8B report inference speedups on A100 and GB200 GPUs, supported by ablations of the drafting, refinement, and hidden-state prediction components.

Critical strengths
  1. Target-conditioned drafters typically depend on hidden states produced by the preceding verification step, which introduces a sequential dependency between verification and subsequent drafting. Predicting these states and preparing continuations in advance provides a concrete approach to reducing this dependency. The motivation is closely connected to practical inference latency.
  2. Foresight drafting creates an opportunity to overlap computation, while refinement uses the available time to improve candidate quality. The outcome-based candidate cache and fallback mechanism provide a coherent way to handle uncertainty in the predicted verification results. This integration offers a useful direction for coordinating draft quality and execution latency.
  3. Beyond acceptance length, the paper reports throughput, round latency, exposed drafting time, and prediction hit rates. It also evaluates both single-GPU and two-GPU configurations, as well as eager execution and CUDA-graph-based verification. The single-GPU A100 improvements are particularly valuable because they demonstrate benefits within the same GPU count.
  4. The comparisons involving token versus mask inputs, clean-prefix training, refinement iterations, training objectives, and predicted versus true hidden states help explain the observed improvements. In particular, the refinement experiments support the claim that the proposed training procedure enables the drafter to benefit from repeated passes. The hidden-state prediction experiments also help quantify the quality lost by drafting before the true target features become available.
Critical weaknesses
  1. In Table 2, JetSpec++ uses two GB200 GPUs, whereas DSpark and DFlash use one. For example, the reported GSM8K throughput increases from 746.9 tokens/s for DSpark to 1056.8 tokens/s for JetSpec++. This is a meaningful latency improvement, but its interpretation depends on the additional resources. Moreover, Appendix E describes an adapter containing twelve complete target transformer blocks, alongside a decoder using two target layers. Although Table 1 provides encouraging single-GPU results, additional reporting of total parameter count, peak memory, and performance under comparable resource budgets would help establish the practical tradeoff.
  2. The breadth and consistency of the improvements remain somewhat limited. The evaluation uses a single target model, Qwen3-8B, with thinking disabled and batch size one. Moreover, while the single-GPU greedy-decoding results are promising, the advantages over DSpark become small on several benchmarks at temp=1, and MT-Bench does not show an improvement. These results suggest that the benefit depends on the workload and decoding configuration. Evaluation with another target model and additional batch sizes would help clarify the conditions under which the approach is most effective.
  3. The incremental benefit over closely related refinement approaches could be isolated more clearly. Jacobi-based refinement has already been explored in related work such as xPress [1]. The distinctive contribution here appears to be its integration with predicted target features and advance preparation of possible continuations. A direct comparison with a related refinement method, together with experiments under a matched draft-side latency or compute budget, would help distinguish the benefits of this integration from those of additional refinement computation.

[1] xPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding. arXiv 2026.

Questions for authors
  1. Could you report single-GPU GB200 results, peak memory usage, and the total parameter count of the drafter, adapter, and decoder? How does the latency advantage change under comparable GPU budgets?
  2. How sensitive are the comparisons to the modified baseline recipes, particularly DSpark’s training batch size and disabled confidence scheduling, and DFlash’s removed positional loss decay?
4 comments
AuthorsOpenReview ↗

Response to Reviewer myKR (Part 1 of 2): W1, Q1, W2

We thank reviewer myKR for recognizing the motivation, the integrated design, the range of reported metrics, and the ablations. We address each concern below and in a second comment. G1 to G3 are the items of our general response. Jn-Rm means n refinement passes (Jn, as in the paper) with m adapter refreshes.


W1, Q1: Resource budget

Single-GPU GB200 results and comparable GPU budgets. G1 of our general response gives the comparison at equal GPU counts. In short, on one GB200 GPU each, JetSpec++ (J2) is 1.18× faster than DSpark on both benchmarks. On two GPUs each, it is 1.37× faster on GSM8K and 1.34× on MATH-500.

Parameter count. All counts exclude the embedding and the LM head, which the draft models share with the target. Parentheses give the share of the target.

Targettarget parametersJetSpec++DSpark
draft modeladapter, 12 layerstotaldraft model
Qwen3-8B6.95B1.05B2.32B3.36B (48%)1.13B (16%)
Qwen3.8-27B24.35B1.73B4.47B6.20B (25%)1.86B (8%)

The added parameters are 48% of the target's at 8B and 25% at 27B. The draft model has five layers and the adapter 12 layers at both sizes, while the target goes from 36 to 64 layers. The decoder reuses the target's top two layers, so it adds no parameters and needs no training. The adapter size is a setting. The paper uses 12 layers for a better acceptance length. In a short-training ablation at 27B, an adapter of 8 layers (2.98B) keeps 92% and 95% of the true-state τ on GSM8K and MATH-500, against 94% and 96% with 12 layers (our response to Reviewer KYH2, Q2).

Peak memory. Peak memory of each GPU in GiB, on B200 GPUs. 8B: J2. 27B: J3-R3 with anchor replacement.

MethodQwen3-8BQwen3.8-27B
GPU 1GPU 2GPU 1GPU 2
DSpark, one GPU35.1–71.2–
DSpark, draft on GPU 233.05.767.312.1
JetSpec++, two GPUs35.159.170.253.0

JetSpec++ runs its draft side as a second process on GPU 2. Besides its models, this process holds a key-value cache and the CUDA graphs that it captures before serving, which remove the launch overhead from every draft pass. From 8B to 27B, the peak of GPU 1 rises with the target (35.1 to 70.2 GiB), and the peak of GPU 2 stays about the same (59.1 and 53.0 GiB).

We will add the parameter counts and peak memory to the appendix.


W2: Breadth of the evaluation

Another target model and additional batch sizes. We add Qwen3.8-27B, and G3 of our general response gives the results. In short, on the same two GPUs, JetSpec++ is 1.15× faster than DSpark on GSM8K and 1.08× on MATH-500 with the original recipe, and 1.35× and 1.24× faster with anchor replacement, an additional training-free refinement option (G3). At batch sizes 2 to 4 on MATH-500, JetSpec++ (J2) is at least 1.17× faster than the best DSpark placement on two GPUs (G1). All runs keep thinking mode off, the mode of the training data (Section 5.1, Appendix F).

Temperature 1 and MT-Bench. Sampling at temperature 1 lowers τ for every method (the mean of DSpark falls from 6.89 to 5.25, Table 1), and MT-Bench has the lowest τ of every benchmark. Anchor replacement (G3) raises τ in both cases. At temperature 1, J1-R1 with anchor replacement is above DSpark on all seven benchmarks (table below). On MT-Bench at temperature 0, it reaches 4.38 against 3.89 for DSpark.

Acceptance length τ, Qwen3-8B, budget 16, temperature 1. † printed in Table 1; the means are computed from its cells.

MethodGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchMean
DSpark †6.106.134.695.325.186.053.265.25
JetSpec++ J3-R2 †6.006.284.815.445.045.843.175.23
JetSpec++ J1-R1, anchor repl.6.886.505.255.845.536.553.515.72
AuthorsOpenReview ↗

Response to Reviewer myKR (Part 2 of 2): W3, Q2

W3: Comparison with xPress

We re-implemented xPress, and G2 of our general response gives the full comparison. In short, on one GPU each, JetSpec++ (J2) is 1.21× faster than xPress on GSM8K and 1.22× on MATH-500.

Matched draft-side latency budget. A matched budget counts the refinement time of JetSpec++ as if it were on the critical path, as in a method without pipelining. Even under this condition, the refinement of JetSpec++ gains more. At about the same refinement time per round, eight refiner steps of xPress raise τ by 0.82 (12%) on GSM8K and 0.91 (11%) on MATH-500, and one refinement pass of JetSpec++ raises it by 0.89 (16%) and 1.03 (14%), each over its own first draft (table below). Seven and eight steps are settings of the xPress paper (its Table 3), and eight matches the refinement time. In this table both methods draft from the target's true states, so the comparison isolates the refinement. So the benefit of JetSpec++ does not come only from additional refinement computation. It comes from the integration, which enables pipelining: xPress adds its draft and refiner time to every round, while JetSpec++ drafts from predicted target states and refines during verification, and its round is 24% and 22% shorter (G2).

Acceptance length τ before and after refinement, from the target's true states. Qwen3-8B, budget 16, temperature 0. Gain: over the first draft of the same row. Refinement time is per round. † printed in Table 7.

Methodrefinement time (ms)GSM8KMATH-500
first draftafter refinementgainfirst draftafter refinementgain
xPress, 7 refiner steps0.646.607.41+0.81 (+12%)8.199.07+0.88 (+11%)
xPress, 8 refiner steps0.736.607.42+0.82 (+12%)8.199.10+0.91 (+11%)
JetSpec++, 1 refinement pass0.745.70 †6.59+0.89 (+16%)7.52 †8.55+1.03 (+14%)

Q2: Baseline recipes

We train all draft models of the submission except SSD's (SSD is training-free: it drafts with the off-the-shelf Qwen3-0.6B) under one standardized condition, so that they differ by method and not by training budget: the same 800k sequences, six epochs, and a global batch of the same scale (Appendix F). Each baseline keeps its published architecture and loss, except for the loss decay of DFlash (below).

  • Disabled confidence scheduling: confidence scheduling prunes the verified block, which cannot raise τ. Without it, every method verifies a block of fixed size (Section 5.1). The DSpark paper also disables it for its τ results (its Section 4.2).
  • DFlash's removed loss decay: this setting touches only the DFlash rows. DSpark, the baseline of our main comparisons, keeps its own loss with its decay (Appendix F). Restoring the decay at DFlash's published rate of 7 raises the τ of DFlash by about 7% on GSM8K and 3% on MATH-500 (table below). In Table 1, at temperature 0 and budget 16, the τ of DSpark and of JetSpec++ is at least 18% above DFlash's on these two benchmarks, more than twice that gain.

Acceptance length τ, Qwen3-8B, budget 16, temperature 0. DFlash with and without its loss decay, two-epoch runs in the ablation setting (Appendix F).

Loss decay rateGSM8KMATH-500
none (as in our recipe)4.426.31
34.796.45
74.726.52
154.576.33

Thanks for the authors' detailed reponse. I have no more concerns and increased my score.

AuthorsOpenReview ↗

Response to Reviewer myKR's Follow-up

We sincerely thank reviewer myKR for the reply and for raising the score. The revised PDF is now uploaded with the results that you asked for: equal GPU counts, more batch sizes, peak memory, and the comparison with xPress.

Reviewer_KYH2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JetSpec++ proposes to simultaneously optimize drafting cost and acceptance rate through algorithm-system co-design. Its core consists of three main components: inter-block foresight drafting to anticipate future verification outcomes; intra-block self-refinement using idle compute within the verification window to enhance causal dependency; and a pipelined design that overlaps drafting and verification stages. Powered by NVIDIA GB200 GPUs and CUDA graph optimizations, the method achieves up to a 6.74x end-to-end wall-clock speedup while preserving the target model's output distribution.

Critical strengths

Outstanding Algorithm-System Co-Design: The paper innovatively combines "Foresight Drafting" with "Intra-block Self-refinement," successfully hiding the drafting cost—which originally resided on the critical path—within the target model's verification window, thereby significantly improving system throughput.

Significant Speedup: Powered by modern hardware such as GB200 and CUDA graph optimizations, JetSpec++ achieves end-to-end wall-clock speedups of up to 6.74x (over autoregressive AR) and 1.41x (over DSpark), while maintaining a high average acceptance length across seven mainstream benchmarks.

Rigorous Theory and Ablations: The paper not only demonstrates exceptional performance on complex reasoning benchmarks such as GSM8K, MATH-500, and AIME25 through extensive experiments, but also provides in-depth theoretical analysis and ablation studies in the appendix regarding output distribution accuracy, the relationship with existing geometric fan-out strategies, and training loss/attention mechanisms.

Critical weaknesses

1.Lack of Generalization Validation on Larger-Scale Target Models: The current experiments are primarily based on an 8B target model (e.g., Qwen3-8B). Although the authors speculate that larger models (such as 32B or 70B) would offer a larger verification window and better performance, there is a lack of direct experimental support on ultra-large-scale models.
2.Insufficient Empirical Evaluation on Scaling to Larger Models: The evaluation is currently limited to the 8B scale. While it is hypothesized that the method scales well to larger model sizes (e.g., 32B/70B) due to extended verification windows, validating this hypothesis with empirical results on larger models remains necessary. Tedious 3.Hyperparameter Tuning and Threshold Calibration: The system introduces complex cache budget selection, calibration thresholds (split-conformal thresholds), and dynamic Jacobi iteration step limits. However, the paper lacks an explanation of how these hyperparameters generalize across different tasks and datasets, and there is no automated adaptation mechanism provided.

Questions for authors

1.Validation on Larger Models: Since evaluations are limited to an 8B model, can the authors provide empirical results for larger models (e.g., 32B/70B) and clarify how increased verification latency affects the trade-off between Jacobi passes and throughput?

2.Empirical Evidence for Scaling: Do the authors have experimental data supporting the hypothesis that larger models yield better performance, or are there theoretical bottlenecks (e.g., adapter accuracy degradation) when scaling up?

3.Hyperparameter Generalization: How sensitive are manual parameters (cache budgets, calibration thresholds, Jacobi limits) to different tasks or domain shifts, and is there an automated adaptation strategy?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KYH2 (Part 1 of 2): W1, W2, Q1, Q2

We thank reviewer KYH2 for recognizing the algorithm-system co-design, the speedups, and the theory and ablations. We address each concern below and in a second comment. Where a concern overlaps with another reviewer's, we refer to our general response (G1 to G3). Jn-Rm means n refinement passes (Jn, as in the paper) with m adapter refreshes.


W1, W2, Q1: Larger target models

Results on a larger target. We add Qwen3.8-27B, a target 3.5× larger than Qwen3-8B (G3). On the same two GPUs, JetSpec++ with the original recipe (J3-R3) is 1.15× faster than DSpark on GSM8K and 1.08× on MATH-500, and 1.35× and 1.24× faster with anchor replacement, an additional training-free refinement option (G3).

Qwen3.8-27B, batch 1, budget 16, temperature 0. Round in ms.

MethodGPUsGSM8KMATH-500
tok/s (speedup)τroundcache hittok/s (speedup)τroundcache hit
AR160.3 (1.00×)1.0016.58N/A59.3 (1.00×)1.0016.88N/A
DSpark1234.2 (3.88×)7.8632.00N/A277.9 (4.69×)9.0031.97N/A
DSpark, draft on GPU 22248.4 (4.12×)7.8730.13N/A289.4 (4.88×)8.8830.17N/A
JetSpec++ J3-R32284.6 (4.72×)8.2828.3993.2%312.9 (5.28×)8.9928.3993.5%
JetSpec++ J3-R3, anchor repl.2336.2 (5.58×)9.7528.4388.6%358.9 (6.05×)10.4028.4490.1%

Verification latency, Jacobi passes and throughput. The verification takes about 5.5 ms at 8B (Section 5.3) and 27.9 ms at 27B (table below). A pass or a refresh raises throughput while it is hidden under the verification: it then raises τ at no cost in round time (Eq. 4). Beyond that budget, it adds to every round, so it pays only if it raises τ by more. A longer verification raises the budget: the 8B verification holds two refinement passes, and the 27B verification holds three passes and three refreshes.

Verification, draft-side and round time, GSM8K, two GPUs, ms per round. † printed in the submission (Section 5.3, Table 2).

TargetRecipeverificationdraft sideround
Qwen3-8BJ25.5 †4.866.24 †
Qwen3.8-27BJ3-R327.9119.6228.39
Qwen3.8-27BJ3-R3, anchor repl.27.9424.3528.43

Q2: Evidence for scaling, and adapter accuracy

Evidence for scaling. We present the evidence on the 27B target here, and G3 of our general response gives the full results. The longer verification gives JetSpec++ a higher τ. With the original recipe, the served τ of JetSpec++ is 6.68 on GSM8K and 8.44 on MATH-500 at 8B, against 7.26 and 9.05 for DSpark (Table 2). At 27B it is 8.28 and 8.99, against 7.86 and 9.00. With anchor replacement, it is 24% above DSpark's on GSM8K and 16% on MATH-500 (G3).

Bottlenecks when scaling up.

  • Adapter accuracy: the adapter accuracy holds at 27B. With a third of the adapter training (two epochs against six), the 27B adapter stays within 1.3 points of the 8B adapter in decoder agreement and within 3 points in the τ that it keeps (first table below).
  • Adapter size: under the current design, the adapter grows with the target's width, not with its depth. From 8B to 27B the target goes from 36 to 64 layers, and we keep the adapter at 12 layers, where it retains high accuracy. Its parameters grow 1.9×, against 3.5× for the target. The number of adapter layers has little effect on the adapter's predictions: in a short-training ablation at 27B (second table below), 16 layers raise the τ kept by at most 1 point at 1.33× the cost of an adapter call, and 8 layers lose 1 to 2 points. The adapter size is therefore flexible: when the draft side must be lighter, a smaller adapter costs little τ.

Adapter training. Decoder agreement as in Table 9, on served rounds. τ kept: τ from predicted states over τ from true states (the ratio of Table 7), here at J0.

Targetadapter epochsdecoder agreementτ kept, GSM8Kτ kept, MATH-500
Qwen3-8B695.7%98%97%
Qwen3.8-27B294.4%95%96%

Adapter layers at Qwen3.8-27B, half an epoch of adapter training. τ kept at J0, read with one fixed draft model for all three rows. Cost: time of one adapter call, relative to 12 layers.

Adapterparametersτ kept, GSM8Kτ kept, MATH-500cost per call
8 layers2.98B92%95%0.73×
12 layers (as served)4.47B94%96%1.00×
16 layers5.96B94%97%1.33×
AuthorsOpenReview ↗

Response to Reviewer KYH2 (Part 2 of 2): W3, Q3

W3, Q3: Hyperparameters

Sensitivity to tasks and domain shifts. The sensitivity is low. We use one setting for all seven benchmarks: the same calibration, the same cache size B and the same J and R.

  • Calibration: one calibration serves all seven benchmarks. It is fitted once on 56 held-out prompts, 8 for each benchmark, which is about 7% of the 750 evaluation prompts (Appendix C.1). With it, the hit rate stays between .887 and .964 (first table below).
  • Domain shift: MT-Bench and AIME25 use all their prompts for evaluation, so they have no calibration prompts of their own. Prompts from UltraChat and from the GSM8K training split are used in their place (Appendix C.1), and their hit rates stay inside the range of the others.

Excerpt of Table 13: hit rate at temperature 0 with the one calibration. † printed in Table 13.

GSM8KMATH-500AIME25HumanEvalMBPPLCBMT-Bench
hit rate †.964.962.934.941.933.887.891
calibration promptsownownsubstituteownownownsubstitute
  • Cache budget: B is an upper limit: 16 in Table 1 and 4 in Table 2 (Appendix E). The served hit rate is .964 on GSM8K and .962 on MATH-500 at B = 16 (Table 13), and .942 and .924 at B = 4 (Table 2). At a fixed B, the calibration matters most when B is small (second table below). The trade-off is in Eq. 3: a larger B raises the hit rate and the time of each draft pass, which leaves fewer refinement passes.

Excerpt of Table 11, Appendix C.3: joint hit rate at a fixed cache size B, offline. † printed in Table 11.

Selection ruleGSM8KMATH-500
B = 4B = 16B = 64B = 4B = 16B = 64
calibrated score, served †.910.984.992.835.967.995
decoder score, uncalibrated †.878.970.987.719.888.962
  • Jacobi limits: they depend on the hardware and the target, not on the task. We set J and R once per target and hardware, by measurement, so that the draft side stays hidden under the verification (Eq. 3).

Automated adaptation. The total budget of the draft side is profiled once, from the roofline of the hardware: the verification time of the target (Eq. 3). Every setting below follows from this budget or from the one calibration.

  • Thresholds: they are not set by hand. Split-conformal calibration computes them from one coverage level: the verified outcome is cached in 93% of the calibration rounds (Appendix C.1).
  • Cache: in each round, the calibrated score ranks the candidate outcomes, and the threshold and the cache size B decide how many of them are cached (Section 4.2).
  • New domain: the calibration is one offline pass over a few dozen held-out prompts, so it can be repeated cheaply for a new domain.
  • Passes: J and R are fixed for a target and hardware. Within a round, the engine publishes the last pass that has finished when verification is ready (Appendix F).

We will add this explanation to the appendix.

AuthorsOpenReview ↗

Follow-up to Reviewer KYH2

Dear Reviewer KYH2, thank you again for your review. We have responded above to your questions on larger target models and on the hyperparameters, and the revised PDF is now uploaded. As the discussion period ends in less than 24 hours, we would appreciate it if you could let us know whether our response addresses your concerns. If it does, we would be grateful if you could take this into account in your final assessment. If any question remains, we are glad to clarify.

AuthorsOpenReview ↗

Question on the Ethics Flag

Dear Reviewer KYH2, we noticed that your review flags the submission for a potential Code of Ethics concern. We could not find the concern in the text of the review. If you can share it, we will address it before the discussion closes.

Reviewer_ev5vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JetSpec++ builds upon JetSpec to accelerate speculative decoding by overlapping the drafting of the next block with the target's verification of the current block. It aims to address the challenge of target-conditioned drafting, in which the drafter requires information unavailable until the verification finishes, including the accepted prefix, bonus token, and even target's hidden states the for newly verified context.

The method combines three components: 1) Foresight drafting, where an adapter predicts missing target hidden states and a decoder reuses upper target layers to estimate verification outcomes, alllowing next-block drafts for selected outcomes to be prepared and cached. 2) Self-refinement, in which a causal parallel drafter trained on clean prefix or noisy suffix inputs repeatedly revises its predictions. 3) Finally, the execution is pipelined in which drafting and verification overlap, with fallback to ordinary drafting when the realized outcome is absent from the cache.

Critical strengths
  1. In addition to overlapping drafting and verification, JetSpec++ identifies the information needed to enable the overlap and provides an adapter for approximating the missing target hidden states. The adapter's effectiveness is supported by comparisons against copying the last verified state and using the target's true states.

  2. The paper's refinement ablations showcase the importance of the training recipe. Table 3 compares the vocabulary token inputs with masked token inputs, using either clean prefixes training or fully corrupted blocks. With fully noisy training blocks, token-based inputs yield almost similar acceptance lengths across multiple refinements, while mask-token inputs lead to substantial deterioration. The proposed combination of vocalbulary token inputs and clean prefix training improves acceptance length over subsequent refinements.

Critical weaknesses
  1. The headline speedup uses unequal hardware resources. Table 2 shows JetSpec++ using 2 GPUs comparing with DSpark using 1 GPU. This is valid for latency oriented deployment but does not establish better efficiency under equal resources. For example, normalizing the GSM8K throughput by number of GPUs, JetSpec++ reaches achieves 528 tokens per sec in comparison ot DSpark's 747. Though Table 1 already provides one-GPU evidence but under a different GPU. The authors should make the comparison explicit with 1-GPU baselines under same GPU with throughput latency evaluation under multiple requests. And, also show results on batch workloads with multiple inputs, where dedicated resources for draft can get expensive.

  2. xPress also uses Jacobi refinement and evaluates Qwen3-8B on the same seven benchmark families. A comparison wtih xPress under matched hardware, training conditions, and drafting time budgets should be included. This would help establish whether refinement using the full drafter is better than using a lighweight refiner within the same time budget.

  3. The adapter's design for foresight drafting depends on the architecture of the target hidden layers. This begs the question about generalizability across target architectures. The paper only evaluates Qwen3-8B and should therefore include evaluation on other target archictures. This would help establish how much architectural adaptation and retraining would be needed for other architectures and whether performance benefits generalize.

Questions for authors

Refer to weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ev5v (Part 1 of 2): W1

We thank reviewer ev5v for recognizing the adapter design and the refinement ablations. We address each concern below and in a second comment. Where a concern overlaps with another reviewer's, we refer to our general response (G1 to G3). Jn-Rm means n refinement passes (Jn, as in the paper) with m adapter refreshes.


W1: Equal hardware resources

We add the comparison at equal GPU counts, and JetSpec++ is faster than DSpark in both cases. Your normalization (528 against 747 tok/s per GPU on GSM8K) compares JetSpec++ on two GPUs with DSpark on one. The same normalization gives 884 against 747 tok/s per GPU with one GPU each, and 528 against 387 with two GPUs each.

  • One GPU each: we run the target process and the draft process of JetSpec++ on one GPU, which they share through NVIDIA MPS (Appendix E). JetSpec++ (J2) gives 884 tok/s against 747 for DSpark on GSM8K, and 1085 against 920 on MATH-500. It is 1.18× faster on both.
  • Two GPUs each: we give DSpark the second GPU in two ways: its draft model on GPU 2, or the target tensor-parallel over both GPUs. Neither changes its one-GPU throughput by more than 4%. DSpark drafts and verifies in sequence, so a second device cannot overlap the two stages, and tensor parallelism adds communication to every forward pass and thus performs worse (G1). JetSpec++ (J2) is 1.37× faster than the better DSpark placement on GSM8K and 1.34× on MATH-500.

Qwen3-8B, batch 1, setting of Table 2. Round in ms. † printed in Table 2, with its speedup.

MethodGPUsGSM8KMATH-500
tok/s (speedup)τroundcache hittok/s (speedup)τroundcache hit
AR †1190.9 (1.00×)1.005.20N/A186.1 (1.00×)1.005.32N/A
DSpark †1746.9 (3.91×)7.269.68N/A920.4 (4.95×)9.059.76N/A
JetSpec++ J21884.4 (4.63×)6.787.6794.1%1084.5 (5.83×)8.457.7992.5%
DSpark, draft on GPU 22774.0 (4.05×)7.429.42N/A929.3 (4.99×)9.039.59N/A
DSpark, tensor parallel2726.5 (3.81×)7.339.91N/A888.7 (4.78×)9.0710.08N/A
JetSpec++ J2 †21056.8 (5.54×)6.686.2494.2%1249.7 (6.74×)8.446.4992.4%

Multiple requests, batch workloads and the cost of a dedicated draft GPU. We decode 2, 3 and 4 requests together on two GPUs, on MATH-500. With several requests, DSpark can use the second GPU for a second replica, one full DSpark on each GPU. One target GPU and one draft GPU of JetSpec++ (J2) still give at least 1.17× the throughput of the two DSpark replicas at each of these batch sizes. The per-user columns give the speed of each request, which is the latency measure.

Two GPUs, Qwen3-8B, MATH-500, tok/s. Per user is the total divided by the batch size.

MethodBatch 2Batch 3Batch 4
TotalPer userTotalPer userTotalPer user
DSpark, 2 replicas187793926818943482871
DSpark, draft on GPU 2175987923097702952738
DSpark, tensor parallel175487723697902982745
JetSpec++ J2219610983208106941331033

We will add the rows of the first table to Table 2, and the second table to the appendix.

AuthorsOpenReview ↗

Response to Reviewer ev5v (Part 2 of 2): W2, W3

W2: Comparison with xPress

We re-implemented xPress following its paper. On one GPU each, JetSpec++ (J2) is 1.21× faster than xPress on GSM8K and 1.22× on MATH-500. G2 of our general response gives the full comparison. Below are our results under your three conditions.

  • Matched hardware: the two ratios above are on one GPU each. On two GPUs each, JetSpec++ (J2) is 1.44× and 1.40× faster than the better xPress result. A second GPU changes the throughput of xPress by less than 1%: like DSpark, it runs its stages in sequence.
  • Training conditions: the target and the data are matched. As its paper describes, xPress adds a refiner to a trained DFlash draft model and co-trains the two from this warm start. Our xPress therefore starts from our six-epoch DFlash draft model and co-trains for two more epochs. JetSpec++ refines with the draft model itself, so its refinement adds no module and no training stage.
  • Time budget: a matched budget counts the refinement time of JetSpec++ as if it were on the critical path, as in a method without pipelining. Even under this condition, the refinement of JetSpec++ gains more (table below, from the target's true states). At about the same time, one refinement pass of the full draft model (0.74 ms) gains more than K = 8 steps of xPress (0.73 ms): +0.89 (16%) against +0.82 (12%) on GSM8K, and +1.03 (14%) against +0.91 (11%) on MATH-500, each over its own first draft.

Additionally, the critical difference is that the two methods have different time budgets, because our method is pipelined. xPress adds its refinement time to every round. JetSpec++ refines during verification, so the time budget of its draft side is the whole verification time (Eq. 3), and refinement within this budget adds no time to the round. A matched drafting time budget therefore does not decide the speed of a pipelined method: the round time does. On two GPUs each, J2 spends four times the refinement time of xPress at K = 4 (1.48 ms against 0.37 ms), and its round is still 38% and 36% shorter: 6.24 ms against 10.01 ms on GSM8K, and 6.49 ms against 10.12 ms on MATH-500 (G2).

The two methods start from different first drafts. DFlash, the initialization of xPress, has a τ of 5.91 and 7.59 (Table 1), close to the first draft of JetSpec++ (5.70 and 7.52). xPress has a higher acceptance length at the first draft mainly because it trains for two more epochs. With the recipe of Table 1 (J3-R2), JetSpec++ reaches a τ of 7.49 and 9.10, against 7.41 and 9.07 for xPress at K = 7. G2 gives this comparison on all seven benchmarks, and the serving results.

Acceptance length τ from the target's true states, Qwen3-8B, budget 16, temperature 0. First draft: K = 0 or J0. Gain: over the first draft of the same row. K = 4, 7 and 8 are settings of the xPress paper (its Table 3), and 9 is added. Refinement time is per round on GB200. † printed in Table 7.

Methodrefinement time (ms)GSM8KMATH-500
first draftafter refinementgainfirst draftafter refinementgain
xPress K=40.376.607.29+0.69 (+10%)8.198.94+0.75 (+9%)
xPress K=70.646.607.41+0.81 (+12%)8.199.07+0.88 (+11%)
xPress K=80.736.607.42+0.82 (+12%)8.199.10+0.91 (+11%)
xPress K=90.826.607.44+0.84 (+13%)8.199.12+0.93 (+11%)
JetSpec++ J10.745.70 †6.59+0.89 (+16%)7.52 †8.55+1.03 (+14%)
JetSpec++ J21.485.70 †6.88 †+1.18 (+21%)7.52 †8.86 †+1.34 (+18%)

W3: Other target architectures

Another target architecture, and do the benefits generalize. We add one target with another layer architecture: Qwen3.8-27B, a hybrid in which 48 of the 64 layers are Gated DeltaNet layers. G3 of our general response gives the results. In short, with the original recipe, JetSpec++ leads DSpark in τ on six of the seven benchmarks. On the same two GPUs, it is 1.15× faster on GSM8K and 1.08× on MATH-500. With anchor replacement, an additional training-free refinement option (G3), it leads on all seven and is 1.35× and 1.24× faster.

Architectural adaptation and retraining. You are right that the adapter design depends on the target's layers. The adaptation effort needed is small: the draft model requires no change in its architecture, and the adapter keeps 12 layers, each initialized from its target layer's MLP and from the attention of the nearest full-attention layer. The decoder needs no training. Every draft model that reads the target's features, such as DFlash, is trained again for a new target, and ours is too (six epochs). The only extra training of JetSpec++ is the adapter: two epochs, 172 GPU-hours. Two adapter epochs are enough: predicted states already keep 95% to 96% of the true-state τ at J0 on GSM8K and MATH-500, against 97% to 98% at 8B after six epochs.

Thank you for the detailed response and additional experiments. The new results substantially address my concerns.

For W1, the one-GPU GB200 comparison establishes that the improvement is not solely attributable to allocating an additional GPU. The two-GPU comparison against independent DSpark replicas also directly addresses my concern about alternative uses of the dedicated draft device. I consider this concern largely resolved for the evaluated workloads and concurrency levels. Please include these results in the revision and clarify whether the reported per-user rates are measured individually or calculated only by dividing aggregate throughput by batch size.

For W2, the matched xPress comparison is valuable and supports the end-to-end advantage of the proposed pipeline. At approximately equal refinement time, JetSpec++ obtains a larger improvement over its initial draft, while xPress still achieves higher final acceptance length. Thus, the evidence supports the system level advantage more clearly than an intrinsic superiority of full drafter refinement. The explanation that additional epochs account for xPress’s stronger initial draft should also be presented as a hypothesis unless supported by a controlled comparison.

For W3, the hybrid 27B target provides meaningful evidence of architectural generalization, despite remaining within the Qwen family. The description of adapter adaptation and additional training cost is helpful. The gains obtained with the original method address my main concern. Please distinguish these results clearly from those using the newly introduced anchor replacement option and document that option sufficiently for reproduction.

I'm raising my rating from 2 to 3 (Weak Acceptance) and interested in responses of other reviewers.

AuthorsOpenReview ↗

Response to Reviewer ev5v's Follow-up

We sincerely thank reviewer ev5v for the careful reply and for raising the rating. We have revised the paper according to your three suggestions, and the revised PDF is now uploaded.

  • W1, per-user rates: They are computed as the total throughput divided by the batch size, and they are not measured for each request. The caption of Table 14 in the revision states this.
  • W2, first draft of xPress: We agree that a controlled comparison would be needed to establish this cause. The revision presents the two additional training epochs as one possible cause instead of as a finding (Appendix L).
  • W3, anchor replacement: The revision reports the original recipe and anchor replacement in separate rows and paragraphs, and it documents anchor replacement step by step for reproduction (Appendix I).

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response: Overview

We thank all reviewers for the constructive feedback. Because some of the reviewers' questions overlap, we have formed the overlapping parts into three general items, G1 to G3. Each is posted as a reply to this comment:

  • G1. Comparison at equal GPU counts: two GPUs and one at batch 1, then batch sizes 2 to 4 on two GPUs (ev5v W1; myKR W1, W2, Q1; hzfK W2).
  • G2. Comparison with one more (concurrent) baseline, xPress (ev5v W2; myKR W3; hzfK W1).
  • G3. Scaling to a larger target, Qwen3.8-27B (KYH2 W1, W2, Q1, Q2; ev5v W3; myKR W2).

One term is used below: Jn-Rm. As in the submission, Jn is n refinement passes after the first draft (Section 4.3). Rm adds m adapter refreshes per round.

† marks a row or a number printed in the original submission, and results without the mark are new. All references to tables, sections and equations point to the originally submitted PDF. We will upload one revised PDF with the new results, and we will post a note in this thread when it is up. It keeps the numbering of the original submission, so these references stay valid.

AuthorsOpenReview ↗

General Response G1: Comparison at Equal GPU Counts

JetSpec++ targets per-request latency, so its best setup gives drafting its own device. We therefore take two GPUs for each method as the main comparison, and we also report one GPU for each. JetSpec++ is faster than DSpark at both GPU counts.

Two GPUs each. We give DSpark the second GPU in two ways: its draft model on GPU 2, or the target tensor-parallel over both GPUs. At 8B, neither changes its one-GPU throughput by more than 4%. DSpark drafts and verifies in sequence, so a second device cannot overlap the two stages, and tensor parallelism adds communication to every forward pass. JetSpec++ (J2) is 1.37× faster than the better DSpark placement on GSM8K and 1.34× on MATH-500.

Two GPUs, Qwen3-8B, batch 1, setting of Table 2. Round in ms. † printed in Table 2, with its speedup.

MethodGSM8KMATH-500
tok/s (speedup)τroundcache hittok/s (speedup)τroundcache hit
AR †190.9 (1.00×)1.005.20N/A186.1 (1.00×)1.005.32N/A
DSpark, draft on GPU 2774.0 (4.05×)7.429.42N/A929.3 (4.99×)9.039.59N/A
DSpark, tensor parallel726.5 (3.81×)7.339.91N/A888.7 (4.78×)9.0710.08N/A
JetSpec++ J2 †1056.8 (5.54×)6.686.2494.2%1249.7 (6.74×)8.446.4992.4%

One GPU each. The target process and the draft process of JetSpec++ share the GPU through NVIDIA MPS (Appendix E). JetSpec++ (J2) is 1.18× faster than DSpark on both benchmarks.

One GPU, same setting. Round in ms. † printed in Table 2.

MethodGSM8KMATH-500
tok/s (speedup)τroundcache hittok/s (speedup)τroundcache hit
AR †190.9 (1.00×)1.005.20N/A186.1 (1.00×)1.005.32N/A
DSpark †746.9 (3.91×)7.269.68N/A920.4 (4.95×)9.059.76N/A
JetSpec++ J2884.4 (4.63×)6.787.6794.1%1084.5 (5.83×)8.457.7992.5%

Batch sizes 2 to 4, two GPUs. With several requests, DSpark has a third way to use the second GPU: two replicas, one full DSpark on each GPU. On MATH-500, JetSpec++ (J2) leads the best DSpark placement at every batch size, by at least 1.17×.

Two GPUs, MATH-500, tok/s. Per user is the total divided by the batch size.

MethodBatch 2Batch 3Batch 4
TotalPer userTotalPer userTotalPer user
DSpark, 2 replicas187793926818943482871
DSpark, draft on GPU 2175987923097702952738
DSpark, tensor parallel175487723697902982745
JetSpec++ J2219610983208106941331033
AuthorsOpenReview ↗

General Response G2: Comparison with One More (Concurrent) Baseline, xPress

xPress comes within 2% of DSpark's mean acceptance length, and JetSpec++ (J2) serves 1.21× and 1.22× faster than xPress on one GPU each, and 1.44× and 1.40× faster on two GPUs each, against the better xPress result.

xPress is concurrent work (August 2026). Its released checkpoint is trained on its own data, so to match the training conditions we trained xPress on our 800k sequences, following its paper: a warm start from our six-epoch DFlash draft model, then co-training of the draft model and the refiner. We co-train for two epochs. We evaluate it at K = 4 and K = 7 refiner steps, two of the settings of its paper (its Table 3). As a check of the re-implementation, our xPress raises the mean τ of its initialization, our DFlash draft model, by 21% at K = 4 and 23% at K = 7. The xPress paper reports 28% over its initialization, the released DFlash draft model, on its own data.

Acceptance length. xPress reaches a mean τ of 6.79 at K = 4 and 6.88 at K = 7, against 6.89 for DSpark and 6.88 for JetSpec++ (J3-R2, Table 1).

Acceptance length τ, Qwen3-8B, budget 16, temperature 0. † printed in Table 1; the means are computed from its cells.

MethodGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchMean
DFlash †5.917.596.895.164.915.453.245.59
xPress K=47.298.948.296.316.096.693.906.79
xPress K=77.419.078.446.386.176.753.936.88
DSpark †7.319.018.356.576.176.923.896.89
JetSpec++ J3-R2 †7.499.108.056.616.156.933.816.88

Serving. The refiner of xPress adds 0.37 ms to every round, while JetSpec++ refines during verification. A second GPU changes the throughput of xPress by less than 1%, because, like DSpark, it drafts and refines in sequence with verification. On two GPUs each, JetSpec++ (J2) is 1.44× faster than the better xPress result on GSM8K and 1.40× on MATH-500. On one GPU each, it is 1.21× and 1.22× faster. As served on one GPU, the round of J2 is 24% and 22% shorter than that of xPress, so the speed comes from hiding drafting and refinement under verification, not from a longer acceptance length. With K = 7, xPress gains 2% in τ and no throughput. On one GPU, the throughput of xPress is within 5% of DSpark's.

Qwen3-8B, batch 1, setting of Table 2. Round in ms. † printed in Table 2, with its speedup.

MethodGPUsGSM8KMATH-500
tok/s (speedup)τroundtok/s (speedup)τround
AR †1190.9 (1.00×)1.005.20186.1 (1.00×)1.005.32
DSpark †1746.9 (3.91×)7.269.68920.4 (4.95×)9.059.76
xPress K=41728.7 (3.82×)7.3310.05890.0 (4.78×)8.9310.04
xPress K=71728.2 (3.81×)7.4910.29887.6 (4.77×)9.0810.23
JetSpec++ J21884.4 (4.63×)6.787.671084.5 (5.83×)8.457.79
xPress K=4, draft on GPU 22731.8 (3.83×)7.3310.01882.8 (4.74×)8.9310.12
JetSpec++ J2 †21056.8 (5.54×)6.686.241249.7 (6.74×)8.446.49

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.