- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
JetSpec++ addresses the challenge of overlapping drafting and verification when a speculative drafter relies on target-model hidden states that are unavailable until verification completes. Its central contribution is foresight drafting: a learned adapter predicts the missing target representations, while a decoder estimates likely acceptance lengths and bonus tokens, enabling candidate drafts for the next round to be prepared and cached in advance. The method also trains the same causal parallel drafter to iteratively refine its predictions using clean-prefix, noisy-suffix training, allowing additional refinement passes to improve draft quality within the verification window. Target-side verification preserves the target model’s output distribution. Experiments on Qwen3-8B across seven benchmarks, together with component ablations, demonstrate the effectiveness of the design. In the two-GPU GB200 configuration, JetSpec++ achieves up to 6.74× speedup over single-GPU autoregressive decoding and 1.41× over single-GPU DSpark.
Critical strengths
- Predicted states retain 95–98% of the acceptance length achieved with true states, making the dependency-breaking approximation practically useful.
- Controlled ablations show that refinement benefits come from the proposed training, rather than simply running additional draft passes.
Critical weaknesses
- This work lacks comparisons with closely related methods. Comparisons with xPress, D²SD, and SwiftSpec under comparable resource budgets would help establish the advantage over existing refinement, adaptive branching, and asynchronous pipelining approaches.
- This work needs a more systematic sensitivity analysis of the number of refinement iterations, cache size, adapter refresh frequency, and GPU placement to clarify when additional drafting computation can be hidden behind verification and when it reduces the overall speedup.
Questions for authors
See weaknesses.
2 comments
Follow-up to Reviewer hzfK
Dear Reviewer hzfK, thank you again for your review. We have responded above to your questions on related methods and on sensitivity, and the revised PDF is now uploaded. With less than 24 hours left in the discussion period, we would appreciate it if you could let us know whether any concern remains, and we are glad to clarify. If our response addresses your concerns, we would be grateful if you could take this into account in your final assessment.
Response to Reviewer hzfK
We thank reviewer hzfK for recognizing the predicted-state results and the controlled refinement ablations. We address each concern below. Where a concern overlaps with another reviewer's, we refer to our general response (G1 to G3). Jn-Rm means n refinement passes (Jn, as in the paper) with m adapter refreshes.
W1: Comparison with related methods
We add a direct comparison with xPress, because it is a Jacobi refinement method, a category that our baselines did not cover. D²SD and SwiftSpec fall within categories of our existing baselines, and each has one half of what JetSpec++ combines. D²SD has a parallel draft model that reads the target's states, as DFlash does, but it keeps drafting on the critical path. SwiftSpec is pipelined, as SSD is, but with an independent draft model that does not read the target's states. JetSpec++ pipelines a parallel draft model that reads the target's states. This is the hard case: these states are not ready when a pipelined draft starts, and foresight drafting predicts them (Section 4.2). The results follow: xPress directly, and D²SD and SwiftSpec through the baseline of their category.
W2: Sensitivity analysis
The submission studies each of these factors: the number of refinement iterations (Table 6), the cache size (Table 11), the adapter refresh frequency (Table 12), and when drafting is hidden behind verification (Eq. 3 and 4, Table 2). G1 of our general response adds GPU placement, and G3 a larger target. In short, at temperature 0, refinement saturates by three passes and the hit rate by cache size B = 16, and a second refresh still raises τ. We quote the key numbers of each study below.
GPU placement. G1 of our general response compares one and two GPUs for both methods. In short, with its target and draft processes on one GPU, JetSpec++ (J2) is 1.18× faster than DSpark on both benchmarks. On two GPUs each, it is 1.37× faster on GSM8K and 1.34× on MATH-500, because drafting and verification do not share a GPU. At 8B, DSpark drafts and verifies in sequence, so a second GPU changes its throughput by at most 4%.
When additional drafting is hidden, and when it reduces the speedup. Drafting is hidden while the draft side fits in the verification time (Eq. 3). At 8B, the verification takes about 5.5 ms, and the two refinement passes of J2 are hidden: the draft side takes 4.86 ms. On the added Qwen3.8-27B target (G3), the verification takes 27.9 ms, and three passes with three refreshes are hidden: the draft side takes 19.62 ms. When the draft side takes longer than the verification, the excess adds to every round (Eq. 4). In that case, an extra pass or refresh reduces the speedup unless it raises τ by more.