acceptodds
Under review as a conference paper at ICLR 2027

Stitch-and-Repair Speculative Decoding

Abstract

Speculative decoding accelerates language model inference by having a small draft model propose a block of tokens that a large target model verifies in parallel. A rejection discards the rest of the draft. Preparing continuations in advance helps, but only when they anticipate the target's sampled correction. We introduce Stitch-and-Repair, a training-free method that makes these misses both rarer and cheaper, recycling rejected computation instead of restarting from scratch. In addition to deterministic candidates, Stitch-and-Repair prepares a stochastic wildcard root which, unlike a deterministic candidate, can be coupled to the target's correction. After a rejection, it maximally couples speculative decoding's residual correction distribution with the wildcard's distribution, so the correction lands on prepared work more often while both marginals stay unchanged. On a match, the prepared continuation becomes the next proposal; on a miss, Stitch-and-Repair bridges the correction to a retained donor continuation through a single fresh draft token. All reused tokens are re-verified within the target's regular verification pass, exactly preserving the target distribution with no additional target forward passes. Across Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, and Qwen3-8B targets on HumanEval, UltraFeedback, AlpacaEval, and GSM8K, Stitch-and-Repair improves end-to-end throughput over speculative decoding with continuations prepared in advance by 4.7–8.4% on pooled workloads and by up to 12.2% on individual datasets. It is also 20.2–37.3% faster than a tuned EAGLE-3 on the two targets with official EAGLE-3 checkpoints. Against a control that prepares identical work but samples the correction independently, coupling alone improves throughput by 2.2% on average and by up to 5.3%. Rejected speculation need not be wasted: coupling routes corrections to prepared work, and bridging reconnects the rest to retained work.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Speculative Decoding, Efficient Inference, Probabilistic Inference, Coupling, Rejection Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FW7jOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Stitch-and-Repair improves speculative speculative decoding by reusing draft work after a rejection. It prepares likely candidate roots plus a stochastic wildcard root. When the target produces a correction token, maximal coupling with the wildcard increases the chance that prepared work can be reused. If the correction matches no prepared root, the method drafts one fresh bridge token and then reuses a prepared donor continuation, with the reused tokens still checked by the target.

Across three 8B target models and four datasets, the method improves pooled throughput by 4.7--8.4% over the stronger baseline, with gains up to 12.2% on an individual dataset. One refinement to your notes: the wildcard is sampled from the draft distribution’s remaining mass, rather than simply sampled at random; the paper describes the other prepared roots as deterministic candidate roots.

Critical strengths

S1. Their coupling technique adapts the maximal-coupling construction to couple the correction token's residual distribution with the wildcard distribution. The authors show that this preserves the original residual distribution and achieves maximal agreement between the two distributions, supporting the method's correctness.

S2. Their proposed method does not require any training. It changes how correction tokens are sampled to make prepared work more likely to match and be reused. Since it does not depend on improving the draft model itself, it can be combined with those techniques.

Critical weaknesses

W1. Small performance gain. The method improves pooled throughput by 4.7–8.4% over the stronger baseline. While the gain is consistent across the three target models, it is very modest in magnitude.

W2. Lack of system-level analysis. The paper reports end-to-end throughput but does not break down latency for draft-model inference, target-model verification, or bridging. This makes it difficult to assess when bridging provides the greatest benefit or how much of the overall gain comes from reducing draft-model latency.

W3. Narrow applicability. The method assumes that draft-model inference can run on a separate GPU alongside the target model. This limits its applicability in settings where a dedicated GPU for drafting is unavailable.

W4. Limited evaluation. The experiments cover only 8B-scale target models, batch size eight, and one hardware configuration. It is unclear how well the results generalize to other model sizes, workloads, and hardware setups.

Questions for authors

Q1. Table 3 suggests that coupling alone provides a relatively small throughput gain. Does this mean that most of Stitch-and-Repair’s overall improvement comes from bridging? Could you quantify the contributions of coupling and bridging to the total gain?

Q2. Combination with EAGLE-3. The paper describes EAGLE-3 and Stitch-and-Repair as complementary. Have you evaluated combining Stitch-and-Repair with EAGLE-3? If so, how does it perform? Have you also evaluated combining EAGLE-3 with SSD? What was the GPU configuration for the EAGLE-3 experiments? Did EAGLE-3 use one GPU or two? If two, how were the models placed across them?

Q3. Were the prepared-root budgets the same for SSD and Stitch-and-Repair? Specifically, was SSD evaluated with an average fan-out of three, matching Stitch-and-Repair's budget?

3 comments

We thank the reviewer for the detailed review. All new experiments use chat-formatted prompts and report paired differences over 12–24 prompt/seed pairs with 95% confidence intervals; full tables are in the General Response (GR).

W1. Small gain.

The gain is smallest for short outputs and grows with length. At 2,048 tokens it reaches +14.5% for DeepSeek-8B on A6000, a reasoning model that naturally writes this long. For Llama-8B it is +20% to +26% across GPUs, partly past Llama's natural end (GR B). Against EAGLE-3 in the same engine, we are +13.6% to +42.7% faster on 8B targets on H100 (up to +50.8% at 1,024 tokens). On larger targets, we are +42.1% faster on Qwen3-14B and +26.6% on Qwen3-32B (H100), and +32% on Llama-3.3-70B with one extra GPU for our draft (GR G). None of this needs a draft model trained per target.

W2. System-level analysis.

Per round at batch 8 on A6000 (GR F), target verification costs the same in both methods. The target waits at most 0.15 ms for the draft, so drafting is hidden. Our extra cost is the bridge forward pass on rounds with a miss. It overlaps the roughly 8 ms per round the target spends outside verification (sampling, bookkeeping, communication). Our round is therefore only 0.8 / 0.6 / 1.5 ms longer than SSD's (Llama / DeepSeek / Qwen3-8B). In exchange, tokens per row per round rise by +9.5% / +12.5% / +6.4%: a miss yields 1.7–2.0 tokens instead of 1.0, and coupling raises the hit rate by 1.2–3.1 points. None of the gain comes from faster drafting.

W3. Separate draft GPU.

We inherit this from SSD-style asynchronous speculation. On one GPU, synchronous speculative decoding is faster than both SSD and our method, and we will state this as a limitation. At equal hardware with two GPUs, two data-parallel copies of synchronous SD match our method. The asynchronous design can, however, use hardware that tensor parallelism cannot. With one extra GPU on A6000, we are +17% / +47% / +19% faster than EAGLE-3 on two GPUs (Llama / DeepSeek / Qwen; GR G3).

W4. Limited evaluation.

The revision adds A100 and H100, 8B–70B targets, batch sizes 4–32, outputs of 256–2,048 tokens and T = 0.6–1.0 (GR A–D). The gain over SSD is positive at batch 4–32 on all three GPUs. It is significant everywhere except Llama-8B at batch 4 on A6000 (+1.7%). On larger targets it is +2–4% at 256 tokens, rising to +5–7% at 1–2k tokens.

Q1. Coupling vs. bridging.

A factorial ablation (A6000, batch 8, 24 pairs; 12 for Qwen3-8B) answers this directly:

Arm (vs SSD)Llama-8BDeepSeek-8BQwen3-8B
Wildcard root only+0.6+0.1−0.7
Wildcard root + coupling+2.5+1.9+0.1
Wildcard root + bridge+5.2+9.4+1.1
Full method (wildcard root + coupling + bridge)+6.6+9.9+1.9

All arms keep SSD's three prepared roots per position; the wildcard replaces SSD's third fixed root. Confidence intervals are in GR E1. Most of the gain comes from bridging. Coupling adds +1.9% / +1.8% (Llama / DeepSeek) without the bridge and +1.4% / +0.4% with it, because the bridge already recovers most of a miss. Qwen3-8B gains little on A6000; on H100, the full method gains +4.8% and the coupled wildcard adds +1.6% over SSD + bridge (GR E3).

Q2. EAGLE-3 combination and placement.

We combined our components with SSD's own EAGLE-3 integration (A6000, 12 pairs). Coupling ports over and stays exact but adds little: +0.6% [−0.5, +1.8] (Llama) and −0.3% [−0.9, +0.3] (DeepSeek). With a one-layer head, a miss is cheap, so preparing for misses is worth little. SSD with EAGLE-3 is also 15–35% slower than SSD with the 1B draft, and our method with the 1B draft is +27% / +70% faster than SSD with EAGLE-3. On placement, EAGLE-3 now gets whichever of 1 or 2 GPUs is faster (1 on H100, 2 on A100 and A6000). On large targets we also report it on 4 GPUs (GR G4). With 3 GPUs against its 4, ours is +12.8% faster on Qwen3-32B (+16.5% at 1,024 tokens) but 3% slower on Llama-3.3-70B. With 5 vs 4, ours is +18% to +31% faster.

Q3. Matched root budgets.

Yes. Both methods use K = 4 and fan-out 3 at each of the 5 positions, i.e. 15 prepared roots per sequence. SSD prepares the draft's top-3 tokens; ours prepares the top-2 plus the wildcard.

Thank you for the additional evaluation. I still find that the gain is modest and the technique’s applicability is limited, but I will take these results into account when reconsidering my score. That said, one of the concerns I raised (W2) has not been fully addressed.

We thank the reviewer for engaging with us and reconsidering their score. Regarding (W2) we give the full per-round breakdown here. Please let us know if we misunderstood and the reviewer had a different concern.

Per-round latency breakdown for Llama-8B at batch 8 (ms per decoding round, mean over 12 runs, same steady-state window as GR Section F).

ms per round (Llama-8B, batch 8)A6000: SSDA6000: oursH100: SSDH100: ours
Target verification27.828.37.98.4
Rest of the target's round (sampling, bookkeeping, communication)7.67.96.66.3
Round time35.436.214.514.7
Draft preparation (in parallel with the target)29.830.512.813.2
Bridge pass (GPU time, overlapped with the target)–5.2–1.8
Target waits for the draft–0.12–0.08
Tokens per row per round3.043.333.033.34
Tokens per row on a hit / on a miss3.66 / 1.03.69 / 2.003.67 / 1.03.69 / 2.00
Hit rate76%78%76%80%

None of the gain comes from reducing draft latency: our draft preparation is slightly slower than SSD's, and the target almost never waits for it. Instead, the gain comes from more tokens per round: a miss yields 2.00 tokens instead of SSD's 1.0, and coupling slightly raises the hit rate, while the bridge pass overlaps the target's own work and adds only 0.2–0.8 ms to the round. Bridging therefore helps most when misses are frequent and the target has time outside verification to hide the bridge pass. At batch 8 on A6000, DeepSeek-8B misses more often than Llama-8B (hit rate 68% vs 76%) and gains more (+10.4% vs +6.7%), and the gain grows with batch size, output length and temperature (GR, Sections A–C). It is smallest for Qwen3-8B, where the target has only 2.8 ms per round outside verification (vs 7.6 ms for Llama-8B), so the bridge pass cannot be fully hidden (+2.1%).

On the size of the gain and applicability: the gain is smallest at short outputs and grows where misses accumulate, reaching +18% to +26% for Llama-8B and +11% to +15% for DeepSeek-8B at 1,024–2,048 tokens across GPUs (GR, Section B). Our method improves on SSD in every setting we evaluated, significantly in almost all of them, and since all throughput is wall-clock, these gains are net of its costs. A dedicated draft GPU is shared by all asynchronous speculation methods, including SSD, which is widely cited, reflecting the community's interest in this design; our method adds no hardware beyond SSD's. Also, against EAGLE-3 in its best layout on up to the same two GPUs, our method is ahead in 21 of 27 settings at batch 8, by up to 51%, without training a draft head (GR, Section G1).

Reviewer_zJw7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Stitch-and-Repair reuses prepared draft work after a rejection. It changes how correction tokens are sampled and adds a short bridge when needed, making reuse more likely while preserving the target distribution.

Critical strengths
  1. The method needs no training and preserves the target distribution under stochastic sampling. The reuse procedure is supported by theory.

  2. The controlled comparison tests the effect of coupling. The paper also shows where reuse helps little, making its practical value easier to judge.

Critical weaknesses
  1. I am not sure the baseline comparisons are fair. The SSD settings and preparation budget are unclear, and EAGLE-3 may be affected by the prompt format and inference settings.

  2. Some gains are small, and the tests cover fairly limited settings. More runs would help show whether the improvements are consistent.

  3. I still cannot tell how much comes from coupling, bridging, or the number of prepared candidates. The comparison with Saguaro also needs to be clearer.

Questions for authors
  1. Which SSD features are enabled, and do all methods prepare the same number of candidates?

  2. Do the smaller gains hold across more runs and prompts? What changes in accepted tokens, candidate hits, and extra drafting time explain them?

  3. How does coupling compare with Saguaro? Could you test bridging and candidate count separately, and check EAGLE-3 with its intended prompt format and settings?

1 comment

We thank the reviewer for their review. All new experiments use chat-formatted prompts and report paired differences over 12–24 prompt/seed pairs with 95% confidence intervals; full tables are in the General Response (GR).

Narrow setup.

We now cover A6000, A100 and H100, 8B–70B targets, batch sizes 4–32, outputs up to 2,048 tokens, and T = 0.6–1.0 (GR A–D). On A6000 at batch 8, the gain over SSD is +6.7% / +10.4% / +2.1% [+0.6, +3.6] (Llama-8B / DeepSeek-8B / Qwen3-8B). It grows with output length: at 2,048 tokens it is +14.5% for DeepSeek-8B and +4.8% for Qwen3-8B. Llama-8B reaches +20.4%, partly past its natural end. On larger targets it is +2–4% at 256 tokens and +5–7% at 1–2k tokens.

Q1. SSD configuration.

Both methods run in the same engine with the same placement: target on one GPU, draft on the other. Both use K = 4 and F = 3 at each of the 5 positions, i.e. 15 prepared roots per sequence. SSD prepares the draft's top-3 tokens; ours prepares the top-2 plus a wildcard. From batch 4 up, SSD uses its fast fallback on a miss, keeping only the correction token rather than redrafting just in time. Saguaro sampling is off.

Q2. Mechanism.

The gain comes from more tokens per round, not from a faster stage (GR F). At batch 8 on A6000, tokens per row per round rise by +9.5% / +12.5% / +6.4%, while rounds are only 0.8 / 0.6 / 1.5 ms longer. This has two sources. Coupling raises the root-hit rate by 1.2–3.1 points. And on a miss, our bridge yields 1.7–2.0 tokens where SSD keeps only the correction token (1.0). In both methods the target waits at most 0.15 ms per round for the draft.

Q3. Matched controls.

Both are in GR E (A6000, batch 8; Qwen3-8B on H100 in E3).

  • "SSD + three fixed roots" is SSD itself. Swapping one fixed root for an uncoupled wildcard changes throughput by +0.6% / +0.1% (Llama / DeepSeek, 24 pairs; E1), so the extra root alone adds nothing.
  • "SSD + bridging alone" gains +5.5% / +8.5% over SSD (12 pairs; E2). Replacing its third fixed root with the coupled wildcard gives the full method and adds +1.5% [+0.4, +2.6] / +1.8% [+0.8, +2.8]. On Qwen3-8B, this step adds +1.6% [+0.6, +2.7] on H100 (E3) but nothing on A6000 (−0.3%). The recycled donor tail alone adds +3.1% / +2.5% at no extra round time, and +2.4% to +3.8% at batch 16–32 and 2,048 tokens (GR E2).
  • Within the factorial (E1), coupling adds +1.9% / +1.8% without the bridge and +1.4% / +0.4% with it.

Bridging is therefore the larger contributor. Coupling adds a smaller gain, clearest when misses are costly. These coupling gains are below the submission's 2.2% average, which came from fewer runs, raw prompts and end-to-end timing; we will update that figure.

Saguaro.

Saguaro sampling raises the cache-hit rate by reshaping the draft distribution, which lowers acceptance. Coupling raises the hit rate without changing the draft or the acceptance rate: the correction keeps exactly its residual distribution, and only its joint draw with the prepared wildcard changes. The two act on different objects and could be combined. In the same engine (batch 8, Llama and DeepSeek, 12 pairs each), Saguaro shaping alone (C = 0.6 vs 1.0) changes SSD throughput by −0.3% [−0.9, +0.4]. In the same cells, our method is +8.7% [+6.9, +10.5] over SSD. Saguaro's fast fallback emits random tokens that are almost always rejected, so it effectively keeps only the correction token, as our SSD control does; our bridge replaces it with a fresh token plus a donor tail. We will add this to the related work.

Novelty.

Maximal coupling and deterministic-proposal verification are known tools. Our contribution is to show that, in asynchronous speculation, the correction after a rejection can be coupled to work prepared before the target's outcome was known, without changing the output distribution. A miss can then be bridged exactly onto prepared donor tails. Our ablations attribute the measured end-to-end gain to each part, which the offline donor-tail analysis could not.

EAGLE-3 with its intended setup.

All EAGLE-3 runs now use the chat template its heads were trained with, in our engine, on whichever of 1 or 2 GPUs is faster (GR G).

  • 8B targets. H100: +13.6% to +42.7% (up to +50.8% at 1,024 tokens). A100: +11.7% to +20.3% for Qwen and DeepSeek, −3.2% for Llama. A6000: −3.5% (Llama) to +24.0% (DeepSeek).
  • Larger targets, H100. +42.1% on Qwen3-14B (our 2 GPUs vs EAGLE-3's best layout, 1 GPU) and +26.6% on Qwen3-32B (2 vs 2).
  • With one extra GPU for our draft. +43% to +47% on 14B–32B and +32% on Llama-3.3-70B.
  • With 3 GPUs against EAGLE-3's 4. Ours is +12.8% faster on Qwen3-32B but 3% slower on Llama-3.3-70B.
Reviewer_dpYSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free speculative-decoding method that prepares deterministic alternative roots and a stochastic wildcard, then maximally couples the rejection correction with the wildcard to increase reuse of prepared continuations. When no prepared root matches, the method generates one fresh bridge token and reuses a donor tail as point-mass proposals, which are reverified by the target.

Critical strengths
  1. The paper carefully distinguishes fresh stochastic proposals from copied deterministic tokens, providing an explicit verification rule and an exactness argument for the combined procedure.
  2. The independent-sampling control isolates coupling while holding prepared work and bridging fixed, and the timing includes preparation, inter-GPU communication, and pipeline drain.
Critical weaknesses
  1. The deployment conditions supporting the throughput gains are narrow. The method relies on spare drafting capacity, and all experiments use batch size eight with draft and target on separate RTX A6000 GPUs, leaving its benefit under single-device deployment or higher serving load uncertain.
  2. The additional draft-side resource cost is not captured by the claim of no extra target passes. The method prepares successor blocks for three alternative roots at every possible rejection position, and the authors acknowledge additional draft compute, so the throughput gains need to be assessed together with preparation cost and memory consumption.
  3. The incremental coupling gain is small enough in some settings that stronger statistical evidence would help. Although the average gain is 2.2%, the DeepSeek gains are only 0.2% on UltraFeedback and AlpacaEval, while the main throughput tables provide point estimates from two random seeds.
Questions for authors
  1. Could the authors report throughput and latency as draft-side capacity is reduced or request load increases, identifying when preparation can no longer be hidden behind target verification?

  2. Could the authors provide an end-to-end factorial ablation separating wildcard preparation, maximal coupling, and bridging, including a preparation-only control without bridging?

  3. How sensitive are the results to temperature and prompt formatting, particularly given that the EAGLE-3 comparison uses temperature 1.0 and raw prompts rather than its original chat-formatted setting?

3 comments

We thank the reviewer for their thorough review. All new results are in the General Response (GR); we summarize the answers here.

W1. Batch size.

Our gain over SSD is positive at every batch size from 4 to 32 on A6000, A100 and H100. It is significant everywhere except Llama-8B at batch 4 on A6000 (GR A). It also grows with output length, holds at lower temperature, and extends to 32B–70B targets (GR B–D). The gain comes from how each method handles a miss. At larger batches almost every round has one. SSD's fast fallback then keeps only the correction token (1.0 token per row), while our bridge yields 1.7–2.0 tokens per miss at a small cost in round time. The gain grows with load on A6000 and is roughly flat at +4% to +11% on A100 and H100.

W1 (cont.). Drafting device.

A separate drafting device is inherent to asynchronous speculation. We need none beyond SSD's, and we prepare the same 15 roots per sequence (3 at each of 5 positions). Our one addition is a second copy of the draft model that computes the bridge token. Placing it on the draft GPU instead still gives +6.4% / +8.5% / +2.1% (batch 8, A6000). On a single device, synchronous speculative decoding beats both SSD and our method. We will state this as a limitation, which we share with asynchronous speculation.

W2. Draft-side cost.

All throughput numbers are wall-clock over the steady decoding window, so preparation, bridge and communication costs are already included (GR F). Compute: preparation overlaps verification, and for Llama-8B draft time stays within 5% of SSD's. The cost on the critical path is 0.6–1.5 ms per round at batch 8, repaid by 6–13% more accepted tokens per round. Memory: the draft copy and bridge cache take about 2.5–3 GB, which shrinks the target's KV cache from 707 to 631 blocks (620 at batch 32). Batch 8 needs at most 16 blocks (about 70 at 2,048-token outputs; at most 64 at batch 32), so the smaller cache never limits capacity.

W3. Coupling significance.

With more runs, the coupling gain is smaller than Table 3's two-seed estimate (2.2% on average), and we will replace Table 3. In the factorial (GR E1), coupling without the bridge adds +1.9% [+0.8, +2.9] on Llama-8B, +1.8% [+1.4, +2.2] on DeepSeek-8B and +0.8% [+0.1, +1.5] on Qwen3-8B. With the bridge, it adds +1.4% [+0.7, +2.0], +0.4% [−0.1, +1.0] and +0.7% [−0.3, +1.7]. Coupling raises the cache-hit rate by 1.2–3.1 points on every model. Its throughput effect shrinks once the bridge recovers most of a miss. It costs nothing: no extra draft passes or memory, and the output distribution is unchanged.

Q1. When does preparation stop being hidden?

At every batch size we tested, the target waits at most 0.15 ms per round for the draft (GR F). Draft utilization reaches 91–95% at batch 32, however, so on these GPUs preparation would stop being hidden soon beyond that. We add at most 2.5 ms of draft work per round, so the ceiling is essentially the same as SSD's.

Q2. Factorial ablation.

See GR E1 and E2. Bridging accounts for most of the gain. The preparation-only control gains nothing (+0.6% / +0.1%, Llama / DeepSeek), because an uncoupled wildcard is just another root. Coupling adds +1.9% / +1.8% without the bridge and +1.4% / +0.4% with it.

Q3. Temperature and prompt format.

All rebuttal experiments use chat-formatted prompts. On A6000 at batch 8, our gain over SSD is +6.7% / +10.4% / +2.1% (Llama / DeepSeek / Qwen3-8B). It holds at T = 0.6–1.0, though Llama's gain shrinks at lower temperature, where the draft misses less (GR C). Against EAGLE-3 with its chat template, lowering T from 1.0 to 0.6 barely changes the comparison (GR G). One correction: with chat prompts and equal hardware on A6000, EAGLE-3 is 3.5% ahead on Llama, while we remain ahead on DeepSeek and Qwen3. We will revise the submission's comparison.

Thanks for the responses. Considering the additional draft compute, I keep my origianl rating.

We would like to thank the reviewer for their prompt response. We would like to clarify "additional draft compute".

In the paper, "additional" is relative to synchronous speculative decoding. The preparation work is SSD's design, and our method keeps it unchanged. What our method adds on top of SSD is one bridge step on rounds with a miss. This adds at most 1.5 ms to SSD (only 0.2 ms on H100s) but our time per output token is lower than SSD's in every row (GR Section F), so the extra work always pays off.

If the reviewer's concern is regarding having a dedicated draft GPU, this is something that all asynchronous methods share. SSD is already widely cited, clearly showing the community's interest in such methods. We also show that, at the same GPU budget for moderate batch size, we outperform EAGLE-3 across most settigs, without having to train an extra speculative head.

We hope that addresses the reviewer's concerns, and if not, we would appreciate the opportunity to clarify if we misunderstood.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

General Response

We thank the reviewers for their detailed and constructive reviews. Since many requests overlap, we collect the new experiments here; the reviewer responses give the details.

Setup. All new runs use chat-formatted prompts and report steady-state decoding throughput so they are robust to compilation, and are therefore not directly comparable with Table 1. Each cell is a paired comparison over 12–24 runs (4 datasets × 3–6 seeds), with 95% t-intervals. SSD is the same control as in the submission: same engine, placement and 15-root budget (its top-3 draft tokens vs our top-2 plus the coupled wildcard at each of 5 positions), with its fast fallback. We tuned SSD (Saguaro sampling never helps) and the tree separately for each method.

Summary. Gains over SSD hold at batch 4–32 on three GPUs, grow with output length (+14% for DeepSeek at 2,048 tokens), and hold at T = 0.6–1.0 and on 32B–70B targets. Bridging gives most of the gain; coupling and recycling the donor tail add +4.3–4.6% over a fresh bridge token on Llama and DeepSeek. Coupling alone adds +0.4% to +1.8% on top of the bridge. Costs are small: the target waits at most 0.15 ms per round for the draft.

Changes to the paper: we will replace Table 3 with these coupling results, revise the EAGLE-3 comparison to use chat-formatted prompts and matched hardware, add all the extra experiments reported here, and add single-device deployment to the limitations.

A. Batch size

% throughput gain over SSD (256 tokens, T = 1.0); all cells significant except those denoted by *.

Llama / DeepSeek / Qwen3B4B8B16B32
A6000 +1.7* / +4.4 / +2.0+6.7 / +10.4 / +2.1+8.2 / +10.2 / +5.4+10.3 / +11.2 / +5.0
A100+10.2 / +10.9 / +5.3+7.8 / +9.6 / +5.6+10.0 / +11.0 / +4.6+9.6 / +11.0 / +5.5
H100+9.7 / +11.0 / +6.6+8.4 / +9.6 / +5.1+6.3 / +9.0 / +4.5+8.3 / +9.8 / +5.1

B. Output length

Batch 8, T = 1.0; all cells significant. Llama's chat answers usually end before 1,024 tokens, so DeepSeek, a reasoning model, is the cleaner evidence of growth with length.

Llama / DeepSeek / Qwen3512 tok1,024 tok2,048 tok
A6000+12.0 / +10.4 / +1.5+17.9 / +11.5 / +4.0+20.4 / +14.5 / +4.8
A100+13.2 / +10.1 / +5.1+19.2 / +12.6 / +4.7+26.0 / +14.7 / +4.0
H100+10.5 / +10.7 / +5.1+18.6 / +12.1 / +4.1+22.3 / +13.8 / +5.4

C. Temperature

Batch 8, 256 tokens; all cells significant except *. Llama's gain shrinks at lower temperature, where the draft misses less often.

Llama / DeepSeek / Qwen3T = 0.6T = 0.8
A6000+1.9 / +9.2 / +1.9*+3.5 / +10.0 / +2.4
A100+2.6 / +9.7 / +4.6+4.7 / +9.9 / +4.0
H100+2.4 / +9.2 / +4.3+4.1 / +9.4 / +5.2

D. Larger targets

% throughput gain over SSD on H100 (batch 8, T = 1.0).

Target256 tokens1,024 tokens2,048 tokens
Llama-3.1-8B+8.4+18.6+22.3
DeepSeek-R1-Distill-Llama-8B+9.6+12.1+13.8
Qwen3-8B+5.1+4.1+5.4
Qwen3-32B+2.6+5.5+6.1
Llama-3.1-70B+3.4+6.9+7.2

Qwen3-32B shows similar gains on other GPUs: on A100, +4.0% (256 tokens), +5.6% (1,024), and +4.2% (2,048); on A6000, +4.5% (256) and +5.0% (1,024). Llama-3.1-70B gains +3.2% to +3.5% at 256 tokens on A100.

AuthorsOpenReview ↗

General Response Continued

E. Ablation

E1 isolates each component; E2 adds them one at a time to SSD, at batch 8 and under higher load and longer outputs; E3 repeats the factorial for Qwen3-8B on H100, where its gains are large enough to separate the components.

E1. Factorial (A6000, batch 8): which components help? % gain over SSD with 95% CIs; 24 pairs (12 for Qwen3-8B). All arms keep SSD's three roots per position.

Llama-8BDeepSeek-8BQwen3-8B
Wildcard root only+0.6 [−0.3, +1.6]+0.1 [−0.5, +0.6]−0.7 [−1.8, +0.3]
Wildcard root + coupling+2.5 [+1.4, +3.6]+1.9 [+1.3, +2.4]+0.1 [−1.2, +1.3]
Wildcard root + bridge+5.2 [+3.6, +6.8]+9.4 [+8.3, +10.5]+1.1 [−0.4, +2.7]
Full method+6.6 [+4.8, +8.5]+9.9 [+8.6, +11.2]+1.9 [+0.3, +3.4]

E2. Gain over SSD, step by step (A6000; Llama / DeepSeek / Qwen3-8B; 12 pairs; separate runs from E1). Each step is the gain over the one before; steps compound to the total. * = 95% CI includes zero.

Bridge token+ coupled wildcard+ donor tailTotal
Batch 8+2.3 / +5.8 / +0.5*+1.9 / +1.0 / −0.6*+2.7 / +3.4 / +2.0+7.1 / +10.5 / +1.9
Batch 8, SSD's roots (no wildcard)+2.3 / +5.8 / +0.5*–+3.1 / +2.5 / +1.7+5.5 / +8.5 / +2.2
Batch 16+3.7 / +6.1+1.9 / +0.8+2.4 / +3.0+8.2 / +10.1
Batch 32+7.1 / +8.1+1.1 / +0.4+2.6 / +2.9+11.0 / +11.7
2,048 tokens+18.6 / +9.4+0.9* / +1.1*+0.8* / +3.8+20.7 / +14.7

E3. Qwen3-8B on H100 (12 pairs; % gain over SSD unless noted, 95% CIs).

Qwen3-8B, H100
Wildcard root only+0.4 [−0.3, +1.0]
Wildcard root + coupling+0.9 [−0.1, +1.8]
Wildcard root + bridge+3.6 [+1.3, +5.9]
Full method+4.8 [+3.3, +6.2]
Matched control
SSD + bridge (three fixed roots) vs SSD+3.1 [+1.4, +4.8]
Full method vs SSD + bridge+1.6 [+0.6, +2.7]

F. Costs and whether preparation stays hidden

Llama-3.1-8B, SSD → ours. Throughput is wall-clock, so all costs are already in the gains.

A6000 B8A6000 B32H100 B8H100 B32
Round time, ms35.4 → 36.283.6 → 82.614.5 → 14.722.3 → 22.3
Tokens accepted per row per round3.04 → 3.332.90 → 3.233.03 → 3.342.89 → 3.21
Time per output token, ms11.6 → 10.928.9 → 25.64.8 → 4.47.7 → 6.9
Target waits for draft, ms per round (ours)0.120.150.080.10
Draft busy, % of round83 → 8294 → 9581 → 8190 → 91
Target KV cache, blocks707 → 631707 → 6201406 → 13301406 → 1319

At batch 8, our rounds are 0.6–1.5 ms longer than SSD's across our 8B models but carry 6–13% more tokens; at batch 32 they are no longer. The draft is busy 91–95% of each round at batch 32 in both methods, so preparation would stop being hidden soon beyond that, at about the same load for both. The bridge shrinks the target KV cache by 5–12%, but a batch of 32 needs at most 64 blocks.

G. Comparison with EAGLE-3

All EAGLE-3 runs use the chat template its heads were trained with. For Qwen3-8B we use a public community EAGLE-3 head, as no official one exists.

G1. 8B targets, our 2-GPU setup vs EAGLE-3's best layout on up to 2 GPUs (% throughput gain, batch 8)

GPU (EAGLE-3 layout)Model256 tok, T=1.0256 tok, T=0.61024 tok, T=1.0
H100 (EAGLE-3 on 1 GPU)Llama-8B+13.6+14.7+15.8
Qwen3-8B+31.2+32.3+30.6
DeepSeek-8B+42.7+42.7+50.8
A100 (EAGLE-3 on 2 GPUs)Llama-8B−3.2−3.3−3.6
Qwen3-8B+11.7+11.7+6.4
DeepSeek-8B+20.3+21.6+25.9
A6000 (EAGLE-3 on 2 GPUs)Llama-8B−3.5−3.2+1.5
Qwen3-8B+3.9+6.2−1.8
DeepSeek-8B+24.0+24.7+30.3

G2. Batch size, A6000, equal hardware (2 GPUs each)

B4B8B16
Llama-8B+13.8−3.5−7.9
DeepSeek-8B+50.0+24.0+13.8
Qwen3-8B+5.4+3.9−6.0

At batch ≥ 32, plain decoding beats both methods on two A6000s (except EAGLE-3 on Llama, by 5%).

G3. One extra GPU, A6000 (ours on 3 GPUs vs EAGLE-3 on 2)

B8B16
Llama-8B+16.9+13.8
DeepSeek-8B+47.3+44.4
Qwen3-8B+18.7+17.4

G4. Larger targets (256 tokens unless noted) (GPU counts: ours vs EAGLE-3; Llama-3.3-70B has a public EAGLE-3 head)

TargetGPUGain over EAGLE-3
Qwen3-14BH100+42.1 (2 vs best of 1 or 2); +43.5 (3 vs 2)
Qwen3-14BA100+47.2 (3 vs 2)
Qwen3-32BH100+26.6 (2 vs 2); +46.0 (3 vs 2); +12.8 (3 vs 4)
Qwen3-32B, 1024 tokH100+23.6 (2 vs 2); +54.6 (3 vs 2); +16.5 (3 vs 4)
Qwen3-32BA6000+30.8 (5 vs 4)
Llama-3.3-70BH100+31.8 (3 vs 2); −2.9 (3 vs 4)
Llama-3.3-70BA6000+18.4 (5 vs 4)

Summary of the discussion

We thank the reviewers again for their thorough reviews. We believe the paper now stands in a much stronger position. In brief, the main concerns and what we added:

Robustness of the gains over SSD. Against SSD in its own engine, with its placement, its 15-root budget and its fast fallback, our method is faster in every setting we evaluated, statistically significantly so in almost all of them. These span batch sizes 4–32, outputs of 256–2,048 tokens, T = 0.6–1.0, A6000, A100 and H100 GPUs, and 8B–70B targets (GR, Sections A–D). The gain grows where misses accumulate: it reaches +18% to +26% for Llama-8B at 1,024–2,048 tokens.

Where the gain comes from (FW7j, W2). The ablation shows that bridging gives most of the gain, and coupling plus the recycled donor tail add +4.4–4.7% over a fresh bridge token alone (GR, Section E). The per-round latency breakdown shows that the gain comes from more tokens per round, not from faster drafting: a miss yields 2.0 tokens instead of 1.0, the bridge pass overlaps the target's own work, and the target waits at most 0.15 ms per round for the draft.

Cost (dpYS). All throughput is wall-clock, so the gains are net of every cost. The dedicated draft GPU is SSD's design, and our method adds no hardware beyond it. Against EAGLE-3 in its best layout on up to the same two GPUs, our method is ahead in 21 of 27 settings at batch 8, by up to 51%, without training a draft head (GR, Section G1).

Revisions. We will correct the Qwen3-8B row of Table 1 (+2.1% like-for-like). We will replace the abstract's EAGLE-3 claim with the equal-hardware comparisons, and replace Table 3 with the factorial ablation. We will add the latency breakdown and the batch, length and temperature results, and revise the framing of the bridge: it makes misses more productive rather than cheaper.

We re-emphasize that a dedicated draft GPU is shared by all asynchronous speculation methods, including SSD, which is widely cited, reflecting the community's interest in this design. Our method adds no hardware beyond SSD's and improves on it in every setting we evaluated. It also pays off against EAGLE-3: on up to the same two GPUs, our method is ahead in 21 of 27 settings and within 4% in the rest, without training a model-specific draft head.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.