acceptodds
Under review as a conference paper at ICLR 2027

SuffixSpec: Hierarchical Context-Aware Memory for Training-Free Speculative Decoding

10 views

Abstract

Speculative decoding accelerates large language model (LLM) inference by drafting candidate tokens and verifying them in one forward pass. Training-free drafters avoid training a draft model for every target model, but they either copy text from the prompt, the generated history, or an external corpus, or index the target model's own predictions by a single token, and none of them uses the predictions that the target model already computes at every prompt position during prefill. We cast drafting as retrieval from a memory of these predictions, organized along two axes: the granularity at which contexts are keyed and the lifetime of stored evidence. Along these axes we build SuffixSpec, a hierarchical memory whose suffix-local, token-local, and token-global layers are queried in order of decreasing specificity; a prefill bootstrap populates it before the first decoding step, and a dual-timescale commit updates the persistent layer once per request with decode-time predictions only. On seven tasks and three models from 8B to 32B parameters, SuffixSpec achieves up to speedup over autoregressive decoding and improves over the strongest training-free baseline by up to . In a controlled comparison inside the SGLang serving engine on Qwen3-8B with 8K-token contexts, it outperforms the trained drafters EAGLE-3 and DFlash under greedy ( vs. and ) and stochastic decoding, while keeping about 10 MB of draft state in host memory instead of 2 to 5 GiB of extra GPU memory for the drafters. The code is available at https://anonymous.4open.science/r/suffixspec-8075.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Training-free, Hierarchical Memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jFrfOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SuffixSpec is a training-free drafter that reframes drafting as retrieval from a memory of the target model's own top-K predictions, organized along two axes: key granularity and evidence lifetime. Three tables form a backoff chain — suffix-local (last m=3 tokens), token-local (last token), and token-global (last token, persistent across requests). The design is motivated by a measurement: a single-token key has ~96% coverage but ~53% precision, whereas a three-token key is precise but hits less than half the time, so no single key length is both precise and available. Filling costs nothing: top-K predictions are written at every verification-tree node, including rejected branches, plus a prefill bootstrap that writes predictions at every prompt position to remove the cold start. A verified-only commit promotes only decode-time token-level entries across requests, since suffix entries are prompt-specific. There is no draft network — the drafter is a 10–17 MB host-side hash table. Reported results: 2.37× over autoregressive decoding, and 2.45× versus DFlash's 2.24× at 8K context in SGLang.

Critical strengths
  1. Measurement-driven design. Each component is justified by a specific statistic, and the summed individual ablation drops (0.53×) exceed the joint drop (0.38×), evidencing real interaction rather than three independent tricks. The finding that persisting suffix entries hurts accepted length (3.07 → 2.95) is genuinely informative.
  2. Cost is measured, not asserted. Drafting takes 0.22 ms against a 27.8 ms target forward, the bootstrap adds <0.5% to prefill, and extra state is ~10 MB of host memory against measured +1.98/+4.91/+2.94 GiB of GPU memory for EAGLE-3/DFlash/DSpark, with no draft-side KV cache.
  3. The trained-drafter comparison is controlled and honestly reported. All four methods run in the same SGLang build, on the same GPU, over the same 180 prompts, under greedy and sampling, with a concurrency sweep — and the authors report the settings where they lose rather than omitting them.
Critical weaknesses
  1. The headline claim is never tested in a serving engine. All nine training-free baselines run only in HuggingFace Transformers at batch one, whose AR baseline is far weaker (46 tok/s for Llama-3.1-8B versus the authors' own 12.4 ms SGLang AR step), inflating every method's speedup by an unknown and unequal amount. Token Recycling, the closest work, is also a pure lookup with no GPU-side state, so porting it into SGLang was no harder than porting SuffixSpec.
  2. Coverage is narrow. Every comparative result is ≤32B, the in-engine comparison is a single 8B model at TP=1, and the 70B result is one paragraph with four numbers and no baselines at all. All three models are dense; there is no MoE data point.
  3. Verification width is uncharacterized. The 81-node tree is near-free only because an 8B dense model at batch one is memory-bound — the measured verification/AR ratio is 1.49, a thin margin against an accepted length of 2.70. In a sparse MoE, bytes read grow with token count (81 tokens scatter across roughly half of an 896-expert layer), so the tree template likely does not transfer, and no evidence is offered that it does.
  4. Concurrency erodes the advantage. Speedup falls from 2.45× to 1.53× between concurrency 1 and 16, already behind DFlash at 16. Large models are served precisely in this regime, so the regime where the method wins and the regime where it would be deployed barely overlap.
  5. The tree is static and the memory exposes no confidence signal. The authors concede that some steps accept too few tokens to pay for verification, yet the budget is never adapted — even though their mother-tree search already produced nested subtrees for smaller budgets, so a tree-size-versus-batch sweep was available and not run.
Questions for authors

See weakness

1 comment

Thank you for the review.

W1. Training-free baselines inside a serving engine

All nine training-free baselines now run in the same SGLang build as our method and AR (§4.3, Table 3). The setup is Qwen3-8B on one RTX 4090, bfloat16, FlashInfer attention, decode CUDA graphs, 180 SPEED-Bench prompts from the 2K and 8K buckets, and greedy decoding. Baselines use their official code, except UniSpec, reimplemented from its paper.

Method2K8KMethod2K8K
PLD1.335×1.509×SSSD1.495×1.731×
PLD+1.407×1.606×LogitSpec1.441×1.696×
CopySpec1.275×1.412×UniSpec1.432×1.587×
SAM (dynamic)1.386×1.539×Token Recycling1.852×1.843×
SuffixDecoding1.439×1.729×Ours, fixed 642.189×2.355×
Ours, budgeted 642.258×2.505×

In the engine, our margin over TR grows rather than shrinks: 16.7% in Transformers (Table 2), 21.9% / 35.9% in SGLang at 2K / 8K, and already 18.2% / 27.8% with the fixed template. TR stays flat from 2K to 8K while our speedup rises, because a longer prompt bootstraps more entries.

W2. Model coverage

Protocol I compares all training-free methods on Llama-3.1-8B, Qwen3-8B and Qwen3-32B, seven tasks, greedy decoding and sampling. The engine comparison uses Qwen3-8B because EAGLE-3, DFlash and DSpark all have public checkpoints for it, so all twelve baselines run on one target. On Llama-3.1-70B (TP = 4 on four H20s), with the 8B tree and hyperparameters, we reach 2.42× over AR: 2.97× on Code, 2.22× on Creative writing, 1.88× on Open-book QA and 2.98× on Sorting (App. C.3). We now also report an MoE target (W3).

W3. Verification width and MoE

On Qwen3-8B, a 64-node round costs 27.20 ms against 20.6 ms for an AR step (1.32 AR steps) and emits 2.96 to 3.33 tokens. The width is now a node budget N set per deployment, lowered with concurrency (W4) and filled by confidence (W5).

We agree with your analysis of sparse MoE: unlike dense models that reuse the same FFN weights across N draft tokens, MoE routes them to many distinct experts, breaking weight reuse and multiplying HBM loads—while the small active size (3B) makes AR already fast (216.8 / 204.2 tok/s at 2K / 8K). This erodes the "free verification" headroom for all speculative decoding methods. We therefore ran Qwen3.5-35B-A3B in SGLang on one H20 (TP = 1, bfloat16, greedy) with N = 8:

Method2K8K
EAGLE-31.337×1.003×
DFlash1.228×1.081×
DSpark1.064×1.077×
Ours (N = 8)1.154×1.136×

As expected, all methods gain far less than on dense targets, trained drafters included. At 2K, EAGLE-3 and DFlash are ahead of us. At 8K we are the fastest, while EAGLE-3 falls to 1.003×: our speedup barely changes with length, since the memory grows with the prompt and adds no draft-side attention. We add this table to App. C and the smaller MoE gains to §5.

W4. Concurrency

We lower the node budget as concurrency rises, up to concurrency 8, the largest our single RTX 4090 sustains at 2K context. On the 2K bucket, each method is compared with AR at the same concurrency (§4.4, Table 4):

ConcurrencyOur nodesOursDSpark (8)DFlash (16)EAGLE-3 (8)
1642.258×2.170×2.220×1.676×
2482.133×2.001×2.018×1.628×
4241.896×1.945×1.851×1.423×
8161.694×1.775×1.590×1.123×

With the budget adjusted, we are ahead of DFlash and EAGLE-3 at every level, and ahead of DSpark at concurrency 1 and 2; DSpark leads at 4 and 8. Changing the budget is a configuration setting and needs no training.

W5. Static tree and confidence signal

The memory now gives a confidence signal that guides tree construction (§3.5). Each entry stores quantized target probabilities, which a 4×8×8×8 score table (indexed by source table, depth, rank and probability bin) maps to path scores; best-first search then keeps the top-N prefixes for node budget N. The table is counted once on 128 WildChat conversations (about 0.2 GPU-hours, no network, no gradients).

Because it is indexed by ranks and probability bins rather than token IDs, the table does not depend on the tokenizer: counted on Llama-3.1-8B, it is used unchanged on Qwen3-8B for all budgeted-tree results, with no per-model refitting. By contrast, training an EAGLE-3 drafter for Qwen3-8B reportedly takes about 700 GPU-hours (§4.3).

Same memory and node count (§4.5):

NodesTree2K τ2K speedup8K τ8K speedup
32fixed2.5082.025×2.8342.290×
32budgeted2.6082.093×2.9412.362×
64fixed2.8362.189×3.0962.355×
64budgeted2.9622.258×3.3322.505×

The budgeted tree raises the speedup by 3.1% to 6.4% and takes 0.16 ms (32 nodes) to 0.33 ms (64 nodes) to build, at most 1.2% of a round.

Reviewer_pxF9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SuffixSpec, a training-free speculative decoding method that organizes the target model's predictions into hierarchical suffix-local, token-local, and persistent token-global memories. By leveraging prefill logits and verification-time predictions, the method improves drafting efficiency without requiring a separately trained model.

Critical strengths
  1. The hierarchical memory and dual-timescale update strategy extend Token Recycling by addressing contextual ambiguity and cold-start limitations.
  2. SuffixSpec consistently outperforms the strongest evaluated training-free baselines across multiple models and tasks. It also achieves higher overall decoding speedups than EAGLE-3, DFlash, and DSpark on 8K-token inputs under low-concurrency settings.
  3. The method requires no additional draft model and introduces relatively little memory overhead. The ablations provide meaningful evidence for the contributions of its main components.
Critical weaknesses
  1. The paper claims state-of-the-art performance among training-free speculative decoding methods, but the comparison omits several relevant 2026 approaches, including SSSD, UniSpec, and LogitSpec. In particular, LogitSpec also exploits target-model predictive signals, albeit through a different retrieval mechanism, while SSSD and UniSpec address efficient training-free drafting under practical deployment settings. Comparisons with representative recent methods would better substantiate the state-of-the-art claim and clarify the novelty of SuffixSpec relative to current approaches.

  2. SuffixSpec requires top-K predictions at every prompt position, whereas optimized inference engines typically do not compute full-vocabulary logits at all prompt positions during standard prefill. Obtaining these predictions may introduce additional LM-head computation and temporary GPU memory overhead.Although the paper reports negligible bootstrap overhead and provides some end-to-end results, the overhead is not sufficiently characterized for longer-context serving workloads. Furthermore, the interaction with prefix caching remains unclear, as cached prompt positions may not be recomputed. A more detailed breakdown of prefill latency, peak GPU memory, TTFT, and end-to-end throughput under different context lengths and prefix-cache hit rates would strengthen the deployment claims.

  3. Section 4.1 and Appendix A.2 specify Transformers 4.46.2 for all models, including Qwen3-8B and Qwen3-32B. However, the official Qwen3 documentation requires Transformers 4.51.0 or later for standard model loading. The authors should clarify whether a patched implementation, custom model integration, or a different version was used, and provide the exact environment required to reproduce the results.

Questions for authors
  1. Can the authors include comparisons with recent training-free methods such as SSSD, UniSpec, and LogitSpec to better support the state-of-the-art claim?

  2. What are the additional LM-head computation, peak GPU memory, and end-to-end latency costs of obtaining prefill top-K predictions at longer context lengths? How does the method interact with prefix-cache reuse?

  3. Could the authors clarify the reported Transformers version and provide the exact software configuration used for the Qwen3 experiments?

  4. How do acceptance length, memory hit rates, and verification latency change from 2K to 8K inputs? Can the authors also report repeated SGLang measurements and clarify global-memory initialization and warm-up conditions?

3 comments

Thank you for the review.

Q1 / W1. Recent training-free baselines

We added SSSD, LogitSpec and UniSpec, and moved all nine training-free baselines into the same SGLang engine as our method (§4.3, Table 3). The setup is Qwen3-8B on one RTX 4090, with 180 SPEED-Bench prompts from the 2K and 8K buckets, greedy decoding and up to 512 new tokens. Speedups are measured against AR in the same engine, and τ is the number of tokens emitted per verification round.

Method2K speedup2K τ8K speedup8K τ
SSSD1.495×1.7581.731×2.122
LogitSpec1.441×1.9881.696×2.446
UniSpec1.432×2.4281.587×2.623
Token Recycling1.852×2.6371.843×2.828
Ours, fixed 64-node template2.189×2.8362.355×3.096
Ours, budgeted 64-node tree2.258×2.9622.505×3.332

We are the fastest training-free method in both buckets: 21.9% / 35.9% above TR, and 51.0% / 44.7% above SSSD, the strongest of the three new baselines. LogitSpec also uses the target's logits, but only the last one, to widen what it retrieves from the text. Our memory emits 49% / 36% more tokens per round (2.962 vs 1.988, 3.332 vs 2.446). SSSD and LogitSpec are ported from their official code, and UniSpec is reimplemented from its paper (App. A.7). Our budgeted tree uses a score table counted once on Llama-3.1-8B and reused unchanged here, so nothing is fitted to Qwen3-8B.

Q2 / W2. Cost of prefill top-K predictions and prefix caching

SGLang normally evaluates the LM head only at the last prompt position. Our bootstrap adds one LM-head pass over the prompt's hidden states and keeps the top-8 tokens per position. It needs no extra pass through the Transformer layers (§3.3, §4.4).

  • Peak GPU memory: +66.6 MiB on an 8K-token request, measured over prefill and decoding. Of this, 17.4 MiB is for the tables and working buffers and 49.2 MiB for the top-K extraction buffers (App. A.1).
  • Prefix caching: we keep SGLang's prefix cache and store the top-K row of each cached position next to its KV entry. On a hit, the KV is reused and the stored rows are replayed into the request-local tables. The prediction at position i depends only on x₁…xᵢ, so a replayed row equals a recomputed one (App. A.1). Only uncached positions pay for the LM-head pass.

TTFT at three prefix-cache hit rates (in each row, AR and our method have the same hit rate):

InputCache hit rateAR TTFTOurs TTFTExtra
2K0%273.4 ms285.2 ms+11.8 ms (+4.3%)
2K≈50%153.4 ms158.0 ms+4.6 ms (+3.0%)
2K≈80%59.8 ms63.2 ms+3.4 ms (+5.7%)
8K0%1229.7 ms1282.0 ms+52.3 ms (+4.3%)
8K≈50%695.1 ms719.9 ms+24.8 ms (+3.6%)
8K≈80%304.6 ms314.5 ms+9.9 ms (+3.3%)

The extra TTFT stays between 3% and 6% and shrinks in absolute terms as more of the prompt is cached (from 52.3 to 9.9 ms at 8K). End to end, the extra prefill time at 8K is recovered within two decoding rounds (§4.4).

Q3 / W3. Transformers version

Thank you for catching this. The version was misreported: all runs, including Qwen3-8B and Qwen3-32B, use Transformers 5.8.1, which loads Qwen3 natively. We corrected §4.1 and App. A.2, and App. A.7 (Table 8) now lists the full engine environment:

ComponentVersionComponentVersion
SGLang0.5.14Python3.12.14
PyTorch2.11.0+cu128Transformers5.8.1
FlashInfer0.6.12sglang-kernel0.4.4
Triton3.6.0NumPy2.3.5
NVIDIA driver595.58.03

Engine runs use bfloat16, TP = 1, FlashInfer attention, decode CUDA graphs and the non-thinking chat template. The anonymized repository contains the modified SGLang source and the dependency overrides needed to rebuild this environment.

Q4. 2K vs 8K inputs, repeated runs, initialization

Our 64-node configuration (Table 3). Cost is the time of one round in AR steps (τ / speedup):

InputτCost per roundSpeedup
2K2.9621.312.258×
8K3.3321.332.505×

Share of lookups resolved at each level on the same 2K/8K workload. The shares cover every candidate-row lookup during tree construction, including lookups on rejected branches:

InputSuffix-localToken-localToken-globalDefault entry
2K20.52%64.22%4.37%10.89%
8K24.64%63.21%5.89%6.25%

From 2K to 8K, τ rises by 12.5% while a round becomes only slightly more expensive (1.31 to 1.33 AR steps), so the speedup grows. A longer prompt bootstraps more entries: the suffix-local share rises from 20.5% to 24.6%, and the share that falls through to the default entry drops from 10.9% to 6.3%.

We repeated the 2K/8K comparison three times for AR, our method, TR and the trained drafters, and throughput differs between runs by at most 0.8% (§4.1).

Each method serves the 180 prompts in one engine instance, starting with the 2K bucket. Our global table starts empty, with no warm-up and no reset between requests (§4.1). Order has little effect: fully shuffling the Protocol I prompts changes throughput by 0.4% (App. C.1).

Thank you for the detailed reply from the author, which has largely resolved my previous doubts. Overall, the innovation of this article mainly lies in the systematic integration and improvement of existing technologies. SuffixSpec demonstrates significant advantages in low-concurrency scenarios, but its advantages in high-concurrency and MoE scenarios are relatively limited. In some settings, it still falls short of DSpark.

Taking all this into consideration, I will maintain my original positive rating. Thank you for the author's supplementary work. I wish the review process goes smoothly!

Reviewer_s7jwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SuffixSpec is a training-free drafter that casts drafting as retrieval from a memory of the target model's own predictions. Predictions made at every prefill position, at every node of the verification tree, and during earlier requests are kept in three tables keyed by the last three tokens, by the last token within the request, and by the last token across requests, and are queried from most to least specific. On seven tasks and three models it reports up to 2.37x over autoregressive decoding, and 2.45x against EAGLE-3 (2.08x) and DFlash (2.24x) inside SGLang at 8K-token contexts, with about 10 MB of host state instead of GPU-resident drafters.

Critical strengths
  1. Figure 1 measures why one key length cannot suffice: coverage falls from 96 to 31 percent as m goes from 1 to 4 while precision rises from 53 to 88 percent. The three tables that follow have distinct and stated roles, queried from most to least specific, and the prefill bootstrap adds under 0.5 percent to prefill time, so the added machinery is cheap and its purpose is explicit rather than asserted.

  2. Table 2 re-measures the training-free baselines inside one stack instead of quoting their own papers, which is the right way to make them comparable, and at 8K-token contexts the method reports 2.45x against EAGLE-3 at 2.08x and DFlash at 2.24x while needing no draft-side GPU memory. A drafter that costs host DRAM rather than HBM is a deployment property the trained drafters do not offer, and it is established here by measurement rather than by estimate.

Critical weaknesses
  1. The framing rests on an absolute claim that does not hold. The abstract states that none of the existing training-free drafters uses the predictions that the target model already computes at every prompt position during prefill, and Section 1 states that all of them discard those predictions. Chen et al. (arXiv:2510.15312, Section 4.3) collect next-token logits for all positions once prefill completes, build a calibrated token tree from them, and cache that tree as a lightweight source for future drafting. The manuscript does not cite this work. What remains unclaimed by it is the hierarchical keying and the dual-timescale commit, not the use of prefill predictions.

  2. The closest suffix-tree competitor is misattributed. Suffix Decoding is cited as Liu et al., 2024b in Section 2 and in Table 5, but the entry under that key is Online speculative decoding, a trained-distillation method. Oliaro et al., whose SuffixDecoding appears as arXiv:2411.04975, is absent from the reference list, and the entries for Liu et al., 2024a and 2024b are printed twice with identical text. The baseline that Table 2 reports at 1.49x and 1.37x therefore cannot be identified by a reader.

  3. Two incomparable stacks are reported as one headline. The 2.37x is measured with Transformers on one RTX 4090, at batch one, with at most 512 new tokens per request, whereas the 2.45x is measured in SGLang on one H20 with 8K-token buckets; the abstract places the two in adjacent sentences without marking the change of setup. The stated margin of 0.36x is the arithmetic difference of two speedup ratios, 2.37 minus 2.01, and not a relative gain, which is 17.9 percent.

  4. Comparability with the trained drafters is not established. Table 3 sets SuffixSpec's fixed 81-node, depth-8 tree against EAGLE-3, DFlash and DSpark, yet the manuscript never states the draft tree, the verification budget, or the acceptance rule used for each of the three. Speedup in this setting is governed jointly by tree shape and by verifier cost, so a gap of 2.45x against 2.08x cannot be attributed to the drafter until these are equalized and reported for every system.

  5. The accepted length is never defined and does not reconcile. Table 2 reports tau equal to 3.15 for the 2.37x configuration, while Section 4.5 states that m equal to 3 is best at 2.37x with tau equal to 3.07 for that same configuration, and it is not stated whether tau includes the bonus token of Eq. (1). Table 6 then reports 2.85 accepted tokens at 2.49x, which implies a per-step cost of about 1.145 autoregressive steps, whereas Appendix C.3 measures 1.49, a cost that would place the same operating point near 1.9x.

  6. Attribution to the three design decisions rests on one figure without printed values. Figure 4 removes the suffix key, taking 2.37x to 2.11x, the prefill bootstrap, leaving 2.23x, and the persistent layer, leaving 2.24x, and those individual drops sum to 0.53x against a measured 0.38x for removing all three. No ablation varies K, the bucket count, or the commit frequency, none extends beyond 8K tokens, and the token-local and token-global tables are never separated.

  7. The memory claim is stated in three ways. The abstract and Table 3 give about 10 MB of host state, while Appendix A.1 gives about 17 MB plus a sparse suffix dictionary below 14 MiB for Llama-3.1-8B. The comparison also sets host DRAM against GPU HBM, so the phrase of 10 MB instead of 2 to 5 GiB contrasts different resources, and the measured 1.98 to 4.91 GiB of Table 3 is not reconciled with the 0.5 to 2.5 GiB KV estimate of Section 4.2.

Revision items I would look for: replace the two absolute sentences with the narrower statement that no prior work keys prefill predictions hierarchically or carries them across requests, add the missing citation and compare directly with that method; correct the citation of Suffix Decoding to Oliaro et al. (arXiv:2411.04975), remove the duplicated Liu et al. entries, and state for every training-free baseline whether it was run from official code or reimplemented; report the verification conditions behind Table 3 for all four systems, that is the draft tree, the verification budget and the acceptance rule, and qualify the claim at matched budget if the trees cannot be matched; give one definition of tau, one speedup that is unambiguously throughput or latency, per-step cost separately for greedy and sampling, and bring Table 3 and Table 6 into agreement; sweep K, the bucket count and the commit frequency, extend beyond 8K tokens, print the values behind Figure 4, and state a single memory figure that separates host residency from GPU residency.

Questions for authors
  1. Chen et al. (arXiv:2510.15312) already reuse prefill logits to build and cache a calibrated token tree for later drafting. How does SuffixSpec differ from that method, and how should the sentence in the abstract be read in its light?

  2. What draft tree, verification budget and acceptance rule were used for EAGLE-3, DFlash and DSpark in Table 3? Were their tree sizes matched to 81 nodes, and if not, how much of the 2.45x against 2.08x remains after matching?

  3. Does tau include the bonus token of Eq. (1), and which value belongs to the final configuration of the 2.37x row: the 3.15 of Table 2 or the 3.07 of Section 4.5?

  4. Is the 2.49x at T equal to 0 in Table 6 the same setting as the 2.45x at 8K in Table 3? If the per-step cost differs between greedy and sampling, what is it in each case, given the 1.49 measured in Appendix C.3?

  5. Which figure is the resident cost of the memory: the 10 MB, or the 17 MB plus 14 MiB of Appendix A.1? Is the 2 to 5 GiB the +1.98 and +4.91 GiB measured in Table 3, or the 0.5 to 2.5 GiB KV estimate of Section 4.2?

4 comments

Thank you for the detailed review and for checking the numbers so carefully. All of the inconsistencies you found have been fixed.

Q1 / W1. Prefill predictions and sd.npu (Chen et al., arXiv:2510.15312)

Thank you for pointing us to this work. sd.npu is a mobile-NPU inference system whose drafting component (§4.3) builds a calibrated token tree from the prefill logits and caches it for later drafting. We now cite it in §1 and §2, and we have replaced both absolute sentences with a narrower claim along the lines you suggested. The abstract now says that training-free drafters

"... either copy text from the prompt, the generated history, or an external corpus, or index the target model's own predictions, made during verification or during prefill, by a single token."

and §1 now reads:

"No existing drafter keys these predictions hierarchically or separates those that should persist across requests from those that should not."

The difference from sd.npu is the one you identified:

  1. sd.npu anchors predictions on single tokens; we key them by the last three tokens and back off to the last token.
  2. We also write the target's top-K at every verification node, including rejected branches.
  3. Across requests we commit only the token entries written during decoding, and drop prefill-only and suffix entries.

The ablation without the suffix key isolates the first difference. With prefill and verification predictions under a single-token key, the speedup falls from 2.37× to 2.11× (Llama-3.1-8B, Protocol I).

Q2 / W4. Verification conditions and matched budgets

Table 3 now states what every system verifies per round and adds EAGLE-3 at our node budgets. All rows use the same engine, GPU, 180 prompts and greedy exact-match verification (configurations in App. A.7). Our budgeted rows use a score table that was counted once on Llama-3.1-8B and reused unchanged on Qwen3-8B (§3.5). Cost is the time of one round, drafting plus verification, in AR steps (τ / speedup).

MethodVerified per roundDraft configuration2K τ2K speedup8K τ8K speedupCost 2K / 8K
EAGLE-36 nodes3 steps, top-2 (default)1.9191.556×2.0311.626×1.23 / 1.25
EAGLE-332 nodes6 steps, top-42.5581.723×2.6611.789×1.48 / 1.49
Ours32 nodesbudgeted, depth ≤ 82.6082.093×2.9412.362×1.25 / 1.25
EAGLE-364 nodes7 steps, top-42.6651.706×2.7461.718×1.56 / 1.60
Ours64 nodesbudgeted, depth ≤ 82.9622.258×3.3322.505×1.31 / 1.33
DFlash16 positionsblock length 162.9842.220×2.7972.072×1.34 / 1.35
DSpark8 positionsblock length 7 + anchor3.0462.170×3.3152.368×1.40 / 1.40

At equal budgets we are 21.5% / 32.0% faster than EAGLE-3 with 32 nodes and 32.4% / 45.8% faster with 64 (2K / 8K). At 32 nodes on 2K the two reach similar τ (2.61 vs 2.56), so most of the gap comes from the cost of a round. Each extra EAGLE-3 level adds a sequential pass of the draft model, which is why its speedup drops from 32 to 64 nodes; our extra nodes cost only lookups. In the submission's H20 setting, 2.45× against 2.08× compared our 81-node template with EAGLE-3's default tree, a 17.8% margin. At matched budgets the margin is 21.5% to 45.8%.

DFlash and DSpark each verify one block whose length is set by the released checkpoint (16 and 8 positions). We run them at those lengths, and the paper makes the matched-budget statement only against EAGLE-3. The last column shows that our 64-node round is still cheaper than a DFlash or DSpark round (1.31–1.33 vs 1.34–1.40 AR steps), so the larger tree does not make our rounds more expensive than theirs.

Q3 / W5. Accepted length

τ is the number of tokens emitted per verification round: the accepted drafts plus the bonus token of Eq. (1). It is defined the same way in both protocols, and the definition is now stated in §4.1. The 2.37× configuration has τ = 3.15 (Table 2). The 3.07 came from the separate run used for the suffix-length sweep; the revision no longer attaches it to the 2.37× configuration.

Q4 / W5. Former Table 6 and the cost of a round

The temperature sweep is now measured in the same engine as Table 3, on an RTX 4090, with the cost of a round taken from the same runs (App. C.2, Table 9). It replaces the former table with the 2.49× row. Our 64-node tree under greedy decoding and sampling (top-p 0.9):

Decoding2K τ2K speedup2K cost8K τ8K speedup8K cost
Greedy2.9622.258×1.313.3322.505×1.33
T = 0.52.9372.291×1.283.1862.423×1.31
T = 1.02.6642.079×1.283.0212.297×1.31
T = 1.32.4141.880×1.282.6552.022×1.31

The greedy cost is τ / speedup from Table 3. A direct profile of one 64-node round gives 27.20 ms, against 20.6 ms per AR step, a ratio of 1.32 (App. A.1, Table 5). The sampling runs use a second RTX 4090, and each run is paired with AR on that GPU at the same temperature.

In every row, speedup ≈ τ / cost, so the three quantities agree. The cost of a round does not change with temperature, because sampling and acceptance take only 0.41 ms (1.6% of a round). The speedup under sampling therefore follows τ.

Q5 / W7. Memory

The abstract and §4.4 now give one figure for the implementation behind Tables 3 and 4: 66.6 MiB of GPU memory at peak. We measured PyTorch-allocated memory on an 8K-token request, with the same KV pool as AR and prefix caching off. In this implementation the tables, tree construction, top-K extraction and write-back all run on the GPU. The other figures belong to other implementations, and the table below separates host from GPU memory:

ImplementationUsed forWhereMemory
SGLang, GPU tables (this revision)Tables 3, 4, 9GPU66.6 MiB at peak: 17.4 MiB tables and working buffers + 49.2 MiB top-K extraction buffers
Transformers, host tablesTable 2 (Protocol I)host DRAMabout 17 MB (Llama-3.1-8B), plus a suffix dictionary below 14 MiB at 32K-token prompts
SGLang, host tables (submission, H20)App. C.3host DRAMabout 10 MB (Qwen3-8B)

The 2 to 5 GiB is the measured extra peak GPU memory of the trained drafters: +1.98 GiB for EAGLE-3, +4.91 GiB for DFlash and +2.94 GiB for DSpark. These drafters hold draft weights and a draft-side KV cache. We removed the 0.5 to 2.5 GiB KV estimate, and the paper now reports only measured values. The comparison is now GPU memory against GPU memory: 66.6 MiB for our method against 1.98 to 4.91 GiB.

W2. Citations and baseline sources

SuffixDecoding is now cited as Oliaro et al. (arXiv:2411.04975). Online Speculative Decoding (Liu et al., 2024) appears once, in §2, as a trained-distillation method, and the duplicated Liu et al. entries are removed. App. A.7 (Table 7) lists the source of every Protocol II method:

MethodSource
PLD, PLD+, CopySpec, SAM (dynamic), SuffixDecoding, SSSD, LogitSpec, TRauthors' official code, ported to SGLang
UniSpecreimplemented from the paper (node budget 125)
EAGLE-3, DFlash, DSparkpublic Qwen3-8B checkpoints

In Protocol I, each baseline runs its released draft module inside our shared Transformers stack with the authors' recommended settings (App. A.2).

W3. Two setups in the abstract; relative margin

The abstract now reports the two protocols in separate sentences and names the setup of each number:

Protocol I (§4.2, Table 2)Protocol II (§4.3, Table 3)
StackTransformers 5.8.1, batch 1SGLang 0.5.14
GPURTX 4090 (8B), H20 (32B)RTX 4090
ModelsLlama-3.1-8B, Qwen3-8B, Qwen3-32BQwen3-8B
Prompts610, seven tasks, ≤ 512 new tokens180 SPEED-Bench, 2K and 8K, ≤ 512 new tokens
Our treefixed 81-node templatebudgeted 32/64 nodes, plus fixed templates of the same size
Resultup to 2.37×, +17.9% over TR2.258× / 2.505×, +21.9% / +35.9% over TR

All margins are now relative: 2.37 / 2.01 − 1 = 17.9% (16.7% on Qwen3-8B and 12.0% on Qwen3-32B). Both protocols report throughput speedup over AR in the same stack, and Protocol II counts decode time only (§4.1).

W6. Ablations

The values behind Figure 4 (Llama-3.1-8B, Protocol I, greedy, same 81-node tree):

ConfigurationSpeedup over ARDrop
Full2.37×–
Without suffix key2.11×0.26
Without prefill bootstrap2.23×0.14
Without persistent (token-global) layer2.24×0.13
Without all three (one token-keyed table filled during decoding)1.99×0.38

The single drops add up to 0.53, more than the joint drop of 0.38, because each component is worth more when the others are present. For example, the bootstrap writes suffix entries as well as token entries, so part of its gain comes through the suffix key. Removing either one alone loses that shared part, while removing both loses it only once.

On separating the tables: removing the token-global layer alone costs 0.13×. Across about 8M lookups, the suffix-local, token-local and token-global tables resolve 23.8%, 63.1% and 12.0% of expansions, and 1.2% reach the default entry (§4.5).

On the bucket count: at 65,536 buckets the state collision rate is 1.90% / 1.88% (Llama-3.1-8B / Qwen3-8B), and the top-1 precision of the hashed table is within 0.55 / 0.54 points of a collision-free table (App. D, Table 12). More buckets can recover at most that gap.

On the commit frequency: an entry committed during a request is never read by that request, because the same token already has a local entry and the lookup consults it first (Eq. 3). For requests served one after another, as in Tables 2 and 3, committing after every round would leave exactly the same global table at the start of the next request. Request order also matters little: fully shuffling the Protocol I prompts changes throughput by 0.4% (App. C.1).

I thank the authors for answering every item with a revision rather than a qualification, and in particular for reporting what each system verifies per round and re-running EAGLE-3 at our node budgets rather than leaving the 81-node template unmatched. Making every margin relative, printing the values behind Figure 4, and explaining why the single removals sum to 0.53 while the joint removal costs 0.38 removes the inconsistency I raised.

Two concerns are still: no sweep over K, and no measurement beyond 8K tokens. I maintain my original score.

Comments on the paper

1 comment

Thank you all for the reviews. New or changed text is marked in blue in the revised PDF. The main changes are as follows.

1. One serving engine for every method (§4.3, Table 3). We ported the nine training-free baselines, including the newly added SSSD, LogitSpec and UniSpec, into SGLang 0.5.14 and ran them together with EAGLE-3, DFlash and DSpark on Qwen3-8B (one RTX 4090, 180 SPEED-Bench prompts from the 2K and 8K buckets, greedy, up to 512 new tokens). Every speedup is measured against AR in the same engine.

MethodDraft model2K8K
Best other training-free (SSSD)no1.495×1.731×
Token Recyclingno1.852×1.843×
EAGLE-3, 32 nodes (best of 6/32/64)yes1.723×1.789×
DFlash, 16 positionsyes2.220×2.072×
DSpark, 8 positionsyes2.170×2.368×
Ours, fixed 64-node templateno2.189×2.355×
Ours, budgeted 64-node treeno2.258×2.505×

2. Budgeted tree construction (§3.5). Each memory entry now also stores the quantized target probabilities of its candidates. A small score table turns them into path scores, and a best-first search picks the tree for a given node budget at every round. The table is indexed by source, depth, rank and probability bin rather than by token IDs, so it does not depend on the model or the tokenizer. We counted it once on Llama-3.1-8B (about 0.2 GPU-hours, no gradient step) and reused it unchanged on Qwen3-8B, which produced every budgeted-tree result in Table 3. The table is thus shared across models instead of being built for each one; by comparison, an EAGLE-3 drafter for Qwen3-8B is reported to take about 700 GPU-hours to train. With the fixed template, our memory is already 18.2% and 27.8% faster than TR, so the conclusions about the memory do not depend on this addition.

3. Serving costs measured in the engine (§4.4). The prefill bootstrap adds 5.1% and 5.2% to TTFT on a 2K- and an 8K-token prompt and 66.6 MiB of GPU memory at peak. The node budget is lowered as concurrency grows (Table 4). Three repeated runs differ in throughput by at most 0.8%.

4. Corrections. We fixed the SuffixDecoding citation, the Transformers version (5.8.1) and the definition of τ (it includes the bonus token), and all margins are now relative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.