- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Stitch-and-Repair improves speculative speculative decoding by reusing draft work after a rejection. It prepares likely candidate roots plus a stochastic wildcard root. When the target produces a correction token, maximal coupling with the wildcard increases the chance that prepared work can be reused. If the correction matches no prepared root, the method drafts one fresh bridge token and then reuses a prepared donor continuation, with the reused tokens still checked by the target.
Across three 8B target models and four datasets, the method improves pooled throughput by 4.7--8.4% over the stronger baseline, with gains up to 12.2% on an individual dataset. One refinement to your notes: the wildcard is sampled from the draft distribution’s remaining mass, rather than simply sampled at random; the paper describes the other prepared roots as deterministic candidate roots.
Critical strengths
S1. Their coupling technique adapts the maximal-coupling construction to couple the correction token's residual distribution with the wildcard distribution. The authors show that this preserves the original residual distribution and achieves maximal agreement between the two distributions, supporting the method's correctness.
S2. Their proposed method does not require any training. It changes how correction tokens are sampled to make prepared work more likely to match and be reused. Since it does not depend on improving the draft model itself, it can be combined with those techniques.
Critical weaknesses
W1. Small performance gain. The method improves pooled throughput by 4.7–8.4% over the stronger baseline. While the gain is consistent across the three target models, it is very modest in magnitude.
W2. Lack of system-level analysis. The paper reports end-to-end throughput but does not break down latency for draft-model inference, target-model verification, or bridging. This makes it difficult to assess when bridging provides the greatest benefit or how much of the overall gain comes from reducing draft-model latency.
W3. Narrow applicability. The method assumes that draft-model inference can run on a separate GPU alongside the target model. This limits its applicability in settings where a dedicated GPU for drafting is unavailable.
W4. Limited evaluation. The experiments cover only 8B-scale target models, batch size eight, and one hardware configuration. It is unclear how well the results generalize to other model sizes, workloads, and hardware setups.
Questions for authors
Q1. Table 3 suggests that coupling alone provides a relatively small throughput gain. Does this mean that most of Stitch-and-Repair’s overall improvement comes from bridging? Could you quantify the contributions of coupling and bridging to the total gain?
Q2. Combination with EAGLE-3. The paper describes EAGLE-3 and Stitch-and-Repair as complementary. Have you evaluated combining Stitch-and-Repair with EAGLE-3? If so, how does it perform? Have you also evaluated combining EAGLE-3 with SSD? What was the GPU configuration for the EAGLE-3 experiments? Did EAGLE-3 use one GPU or two? If two, how were the models placed across them?
Q3. Were the prepared-root budgets the same for SSD and Stitch-and-Repair? Specifically, was SSD evaluated with an average fan-out of three, matching Stitch-and-Repair's budget?
3 comments
Thank you for the additional evaluation. I still find that the gain is modest and the technique’s applicability is limited, but I will take these results into account when reconsidering my score. That said, one of the concerns I raised (W2) has not been fully addressed.
We thank the reviewer for engaging with us and reconsidering their score. Regarding (W2) we give the full per-round breakdown here. Please let us know if we misunderstood and the reviewer had a different concern.
Per-round latency breakdown for Llama-8B at batch 8 (ms per decoding round, mean over 12 runs, same steady-state window as GR Section F).
| ms per round (Llama-8B, batch 8) | A6000: SSD | A6000: ours | H100: SSD | H100: ours |
|---|---|---|---|---|
| Target verification | 27.8 | 28.3 | 7.9 | 8.4 |
| Rest of the target's round (sampling, bookkeeping, communication) | 7.6 | 7.9 | 6.6 | 6.3 |
| Round time | 35.4 | 36.2 | 14.5 | 14.7 |
| Draft preparation (in parallel with the target) | 29.8 | 30.5 | 12.8 | 13.2 |
| Bridge pass (GPU time, overlapped with the target) | – | 5.2 | – | 1.8 |
| Target waits for the draft | – | 0.12 | – | 0.08 |
| Tokens per row per round | 3.04 | 3.33 | 3.03 | 3.34 |
| Tokens per row on a hit / on a miss | 3.66 / 1.0 | 3.69 / 2.00 | 3.67 / 1.0 | 3.69 / 2.00 |
| Hit rate | 76% | 78% | 76% | 80% |
None of the gain comes from reducing draft latency: our draft preparation is slightly slower than SSD's, and the target almost never waits for it. Instead, the gain comes from more tokens per round: a miss yields 2.00 tokens instead of SSD's 1.0, and coupling slightly raises the hit rate, while the bridge pass overlaps the target's own work and adds only 0.2–0.8 ms to the round. Bridging therefore helps most when misses are frequent and the target has time outside verification to hide the bridge pass. At batch 8 on A6000, DeepSeek-8B misses more often than Llama-8B (hit rate 68% vs 76%) and gains more (+10.4% vs +6.7%), and the gain grows with batch size, output length and temperature (GR, Sections A–C). It is smallest for Qwen3-8B, where the target has only 2.8 ms per round outside verification (vs 7.6 ms for Llama-8B), so the bridge pass cannot be fully hidden (+2.1%).
On the size of the gain and applicability: the gain is smallest at short outputs and grows where misses accumulate, reaching +18% to +26% for Llama-8B and +11% to +15% for DeepSeek-8B at 1,024–2,048 tokens across GPUs (GR, Section B). Our method improves on SSD in every setting we evaluated, significantly in almost all of them, and since all throughput is wall-clock, these gains are net of its costs. A dedicated draft GPU is shared by all asynchronous speculation methods, including SSD, which is widely cited, reflecting the community's interest in this design; our method adds no hardware beyond SSD's. Also, against EAGLE-3 in its best layout on up to the same two GPUs, our method is ahead in 21 of 27 settings at batch 8, by up to 51%, without training a draft head (GR, Section G1).
We thank the reviewer for the detailed review. All new experiments use chat-formatted prompts and report paired differences over 12–24 prompt/seed pairs with 95% confidence intervals; full tables are in the General Response (GR).
The gain is smallest for short outputs and grows with length. At 2,048 tokens it reaches +14.5% for DeepSeek-8B on A6000, a reasoning model that naturally writes this long. For Llama-8B it is +20% to +26% across GPUs, partly past Llama's natural end (GR B). Against EAGLE-3 in the same engine, we are +13.6% to +42.7% faster on 8B targets on H100 (up to +50.8% at 1,024 tokens). On larger targets, we are +42.1% faster on Qwen3-14B and +26.6% on Qwen3-32B (H100), and +32% on Llama-3.3-70B with one extra GPU for our draft (GR G). None of this needs a draft model trained per target.
Per round at batch 8 on A6000 (GR F), target verification costs the same in both methods. The target waits at most 0.15 ms for the draft, so drafting is hidden. Our extra cost is the bridge forward pass on rounds with a miss. It overlaps the roughly 8 ms per round the target spends outside verification (sampling, bookkeeping, communication). Our round is therefore only 0.8 / 0.6 / 1.5 ms longer than SSD's (Llama / DeepSeek / Qwen3-8B). In exchange, tokens per row per round rise by +9.5% / +12.5% / +6.4%: a miss yields 1.7–2.0 tokens instead of 1.0, and coupling raises the hit rate by 1.2–3.1 points. None of the gain comes from faster drafting.
We inherit this from SSD-style asynchronous speculation. On one GPU, synchronous speculative decoding is faster than both SSD and our method, and we will state this as a limitation. At equal hardware with two GPUs, two data-parallel copies of synchronous SD match our method. The asynchronous design can, however, use hardware that tensor parallelism cannot. With one extra GPU on A6000, we are +17% / +47% / +19% faster than EAGLE-3 on two GPUs (Llama / DeepSeek / Qwen; GR G3).
The revision adds A100 and H100, 8B–70B targets, batch sizes 4–32, outputs of 256–2,048 tokens and T = 0.6–1.0 (GR A–D). The gain over SSD is positive at batch 4–32 on all three GPUs. It is significant everywhere except Llama-8B at batch 4 on A6000 (+1.7%). On larger targets it is +2–4% at 256 tokens, rising to +5–7% at 1–2k tokens.
A factorial ablation (A6000, batch 8, 24 pairs; 12 for Qwen3-8B) answers this directly:
All arms keep SSD's three prepared roots per position; the wildcard replaces SSD's third fixed root. Confidence intervals are in GR E1. Most of the gain comes from bridging. Coupling adds +1.9% / +1.8% (Llama / DeepSeek) without the bridge and +1.4% / +0.4% with it, because the bridge already recovers most of a miss. Qwen3-8B gains little on A6000; on H100, the full method gains +4.8% and the coupled wildcard adds +1.6% over SSD + bridge (GR E3).
We combined our components with SSD's own EAGLE-3 integration (A6000, 12 pairs). Coupling ports over and stays exact but adds little: +0.6% [−0.5, +1.8] (Llama) and −0.3% [−0.9, +0.3] (DeepSeek). With a one-layer head, a miss is cheap, so preparing for misses is worth little. SSD with EAGLE-3 is also 15–35% slower than SSD with the 1B draft, and our method with the 1B draft is +27% / +70% faster than SSD with EAGLE-3. On placement, EAGLE-3 now gets whichever of 1 or 2 GPUs is faster (1 on H100, 2 on A100 and A6000). On large targets we also report it on 4 GPUs (GR G4). With 3 GPUs against its 4, ours is +12.8% faster on Qwen3-32B (+16.5% at 1,024 tokens) but 3% slower on Llama-3.3-70B. With 5 vs 4, ours is +18% to +31% faster.
Yes. Both methods use K = 4 and fan-out 3 at each of the 5 positions, i.e. 15 prepared roots per sequence. SSD prepares the draft's top-3 tokens; ours prepares the top-2 plus the wildcard.