- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SuffixSpec is a training-free drafter that reframes drafting as retrieval from a memory of the target model's own top-K predictions, organized along two axes: key granularity and evidence lifetime. Three tables form a backoff chain — suffix-local (last m=3 tokens), token-local (last token), and token-global (last token, persistent across requests). The design is motivated by a measurement: a single-token key has ~96% coverage but ~53% precision, whereas a three-token key is precise but hits less than half the time, so no single key length is both precise and available. Filling costs nothing: top-K predictions are written at every verification-tree node, including rejected branches, plus a prefill bootstrap that writes predictions at every prompt position to remove the cold start. A verified-only commit promotes only decode-time token-level entries across requests, since suffix entries are prompt-specific. There is no draft network — the drafter is a 10–17 MB host-side hash table. Reported results: 2.37× over autoregressive decoding, and 2.45× versus DFlash's 2.24× at 8K context in SGLang.
Critical strengths
- Measurement-driven design. Each component is justified by a specific statistic, and the summed individual ablation drops (0.53×) exceed the joint drop (0.38×), evidencing real interaction rather than three independent tricks. The finding that persisting suffix entries hurts accepted length (3.07 → 2.95) is genuinely informative.
- Cost is measured, not asserted. Drafting takes 0.22 ms against a 27.8 ms target forward, the bootstrap adds <0.5% to prefill, and extra state is ~10 MB of host memory against measured +1.98/+4.91/+2.94 GiB of GPU memory for EAGLE-3/DFlash/DSpark, with no draft-side KV cache.
- The trained-drafter comparison is controlled and honestly reported. All four methods run in the same SGLang build, on the same GPU, over the same 180 prompts, under greedy and sampling, with a concurrency sweep — and the authors report the settings where they lose rather than omitting them.
Critical weaknesses
- The headline claim is never tested in a serving engine. All nine training-free baselines run only in HuggingFace Transformers at batch one, whose AR baseline is far weaker (46 tok/s for Llama-3.1-8B versus the authors' own 12.4 ms SGLang AR step), inflating every method's speedup by an unknown and unequal amount. Token Recycling, the closest work, is also a pure lookup with no GPU-side state, so porting it into SGLang was no harder than porting SuffixSpec.
- Coverage is narrow. Every comparative result is ≤32B, the in-engine comparison is a single 8B model at TP=1, and the 70B result is one paragraph with four numbers and no baselines at all. All three models are dense; there is no MoE data point.
- Verification width is uncharacterized. The 81-node tree is near-free only because an 8B dense model at batch one is memory-bound — the measured verification/AR ratio is 1.49, a thin margin against an accepted length of 2.70. In a sparse MoE, bytes read grow with token count (81 tokens scatter across roughly half of an 896-expert layer), so the tree template likely does not transfer, and no evidence is offered that it does.
- Concurrency erodes the advantage. Speedup falls from 2.45× to 1.53× between concurrency 1 and 16, already behind DFlash at 16. Large models are served precisely in this regime, so the regime where the method wins and the regime where it would be deployed barely overlap.
- The tree is static and the memory exposes no confidence signal. The authors concede that some steps accept too few tokens to pay for verification, yet the budget is never adapted — even though their mother-tree search already produced nested subtrees for smaller budgets, so a tree-size-versus-batch sweep was available and not run.
Questions for authors
See weakness
Thank you for the review.
W1. Training-free baselines inside a serving engine
All nine training-free baselines now run in the same SGLang build as our method and AR (§4.3, Table 3). The setup is Qwen3-8B on one RTX 4090, bfloat16, FlashInfer attention, decode CUDA graphs, 180 SPEED-Bench prompts from the 2K and 8K buckets, and greedy decoding. Baselines use their official code, except UniSpec, reimplemented from its paper.
In the engine, our margin over TR grows rather than shrinks: 16.7% in Transformers (Table 2), 21.9% / 35.9% in SGLang at 2K / 8K, and already 18.2% / 27.8% with the fixed template. TR stays flat from 2K to 8K while our speedup rises, because a longer prompt bootstraps more entries.
W2. Model coverage
Protocol I compares all training-free methods on Llama-3.1-8B, Qwen3-8B and Qwen3-32B, seven tasks, greedy decoding and sampling. The engine comparison uses Qwen3-8B because EAGLE-3, DFlash and DSpark all have public checkpoints for it, so all twelve baselines run on one target. On Llama-3.1-70B (TP = 4 on four H20s), with the 8B tree and hyperparameters, we reach 2.42× over AR: 2.97× on Code, 2.22× on Creative writing, 1.88× on Open-book QA and 2.98× on Sorting (App. C.3). We now also report an MoE target (W3).
W3. Verification width and MoE
On Qwen3-8B, a 64-node round costs 27.20 ms against 20.6 ms for an AR step (1.32 AR steps) and emits 2.96 to 3.33 tokens. The width is now a node budget N set per deployment, lowered with concurrency (W4) and filled by confidence (W5).
We agree with your analysis of sparse MoE: unlike dense models that reuse the same FFN weights across N draft tokens, MoE routes them to many distinct experts, breaking weight reuse and multiplying HBM loads—while the small active size (3B) makes AR already fast (216.8 / 204.2 tok/s at 2K / 8K). This erodes the "free verification" headroom for all speculative decoding methods. We therefore ran Qwen3.5-35B-A3B in SGLang on one H20 (TP = 1, bfloat16, greedy) with N = 8:
As expected, all methods gain far less than on dense targets, trained drafters included. At 2K, EAGLE-3 and DFlash are ahead of us. At 8K we are the fastest, while EAGLE-3 falls to 1.003×: our speedup barely changes with length, since the memory grows with the prompt and adds no draft-side attention. We add this table to App. C and the smaller MoE gains to §5.
W4. Concurrency
We lower the node budget as concurrency rises, up to concurrency 8, the largest our single RTX 4090 sustains at 2K context. On the 2K bucket, each method is compared with AR at the same concurrency (§4.4, Table 4):
With the budget adjusted, we are ahead of DFlash and EAGLE-3 at every level, and ahead of DSpark at concurrency 1 and 2; DSpark leads at 4 and 8. Changing the budget is a configuration setting and needs no training.
W5. Static tree and confidence signal
The memory now gives a confidence signal that guides tree construction (§3.5). Each entry stores quantized target probabilities, which a 4×8×8×8 score table (indexed by source table, depth, rank and probability bin) maps to path scores; best-first search then keeps the top-N prefixes for node budget N. The table is counted once on 128 WildChat conversations (about 0.2 GPU-hours, no network, no gradients).
Because it is indexed by ranks and probability bins rather than token IDs, the table does not depend on the tokenizer: counted on Llama-3.1-8B, it is used unchanged on Qwen3-8B for all budgeted-tree results, with no per-model refitting. By contrast, training an EAGLE-3 drafter for Qwen3-8B reportedly takes about 700 GPU-hours (§4.3).
Same memory and node count (§4.5):
The budgeted tree raises the speedup by 3.1% to 6.4% and takes 0.16 ms (32 nodes) to 0.33 ms (64 nodes) to build, at most 1.2% of a round.