acceptodds
Under review as a conference paper at ICLR 2027

Efficient speculative decoding with a fixed-cost parallel drafter

10 views

Abstract

Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative decoding, Lossless LLM serving, Block-diffusion drafting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aP54OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LongSpark, a parallel block drafter whose additional decoding computation and context state are independent of prefix length for a fixed architecture and block size. It materializes the given context into three fixed-size, learned context features, each extracted over a different time period: short (into a boundary state), mid (into a recent KV-cache window), and long (into a global summary). Experiments with Qwen3-4B, 8B, and 14B report favorable throughput and latency on short- and long-context workloads by keeping drafting cost constant, though it may slightly reduce the accepted lengths.

Critical strengths

The paper provides abundant numerical evaluations of the proposed LongSpark. They suggest that the constant drafting cost can outweigh a lower accepted length τ\tau (in particular in Table 1). Moreover, Figures 5a and 5b, along with the long-context ablations in Table 5, are especially helpful for understanding the choice of three context features. I find it interesting that, although the global summary contributes the smallest degradation in τ\tau (Figure 5a), it also contributes to stability in draft training (Figure 5b). In addition, I appreciate that the paper reports the low-concurrency regime where DSpark remains faster.

Critical weaknesses

W1. This work's novelty needs clearer positioning relative to prior work

The current manuscript does not properly reference prior work in several aspects. I think the authors should be more honest about what is already known in prior work; without referencing it, especially in the description of the proposed methodology, the manuscript risks giving the impression that it is new.

  • Novelty of fixed-cost drafting. I am not convinced that fixed-cost drafting itself is a major new contribution, as the abstract and introduction suggest. Indeed, Sec. 4 already acknowledges that Medusa generates proposals from the target's last hidden state at prefix-independent cost for a fixed number of prediction heads. But not only that, ReDrafter also conditions a recurrent draft head on the target's last hidden state and token embeddings, giving prefix-independent proposal cost for fixed draft length and beam width. Thus, the idea of prefix-independent proposal computation already appears in earlier works. I think LongSpark's distinctive contribution is the design that lets its block drafter use richer multiscale context with prefix-independent additional computation and context state. I want the abstract and introduction to emphasize this design and its demonstrated benefits as LongSpark's contribution.

  • Comparison with Jacobi-based decoding. The emphasis on drafting cost also makes me wonder how LongSpark compares with Jacobi-based decoding, which neither Sec. 4 nor the experiments consider. Speculative Jacobi Decoding (SJD) uses the same target forward pass to verify candidates and obtain the next proposals, without a separate drafter. Speculative Coupled Decoding (SCD) improves SJD through coupling across iterations while preserving the target distribution. Moreover, I would say Lookahead Decoding, an earlier method that uses Jacobi iterations for language generation, is a more direct baseline.

  • Training objective. Equation (9) combines CE and L1L_1 terms with the same 0.1/0.9 coefficients as DSpark's training objective. Its exponential position weighting also appears in DFlash. I would like the authors to make these relationships explicit and explain how Equation (9) relates to the KL–TV hybrid objective in LK Losses, particularly its use of token-level CE and a fixed mixing coefficient.

  • The markov incremental update. The "incremental update" mentioned in Sec. 2 and described in Appx. A.2 is basically a slightly generalized, alternative formulation of renowned online softmax (e.g., see the FlashAttention paper). The main text should also explain why it uses the formulas in Appx. A.2. (e.g., to obtain a standard softmax attention over the previous contexts and the new incoming block with constant time & memory).

W2. The serving comparison needs a clearer account of the baseline

  • DSpark's adaptive verification. As I read Sec. 3.1, all speculative methods use seven draft tokens per verification round. DSpark includes a hardware-aware scheduler that adapts verification length to reduce wasted verification under high serving load (Cheng et al., Sec. 3.2). I could not tell whether this scheduler was disabled or whether seven was only a maximum length.
Questions for authors
  1. The paper refers to the matrix G as a bank of "summary queries"; is it only a metaphorical name, or does it have its own secret (autoencoder-like) objective function to drive G to be an effective global context summary? If G is trained only with the objective in Eq. (9), how do you qualitatively show that G indeed plays the role of a global summary and improves draft acceptance efficiency?
  2. Figure 3 omits the per-query log-normalizers described in Appx. A.2 and needed for the summary update; is it intentional? Please clearly show this state alongside the summaries and clarify the connection from the "Weighted sum" output to "Incremental update."
6 comments
AuthorsOpenReview ↗

Response (1 / 2)

Thank you for your careful review and helpful suggestions. We address your concerns below.

W1.1 Fixed-cost drafting

I think LongSpark's distinctive contribution is the design that lets its block drafter use richer multiscale context with prefix-independent additional computation and context state.

Thank you for this clarification. We agree that LongSpark's contribution is better framed around its fixed-size multiscale context interface for a parallel block drafter. We will revise the abstract and introduction accordingly, and add a citation and discussion of ReDrafter [1].

We also compare LongSpark with Medusa and ReDrafter on long-context workloads; see our response to Reviewer 2GJM.

W1.2 Jacobi-based decoding

The emphasis on drafting cost also makes me wonder how LongSpark compares with Jacobi-based decoding [...] Lookahead Decoding [...] is a more direct baseline.

Thank you for this suggestion. Indeed, Jacobi-based decoding does not require a separately trained drafter and has low drafting overhead. We will add a discussion of these methods and include Lookahead [2] as a more direct baseline. The results are shown below:

DatasetLongSpark · C1Lookahead · C1
GSM8K5.98 (4.05×)1.82 (1.23×)
MATH-5005.62 (3.85×)1.96 (1.32×)
AIME254.75 (3.28×)1.89 (1.24×)
MBPP5.09 (3.42×)1.58 (1.05×)
HumanEval5.15 (3.48×)1.67 (1.13×)
LiveCodeBench4.89 (3.25×)1.62 (1.08×)
MT-Bench3.75 (2.56×)1.60 (1.08×)
Alpaca3.43 (2.34×)1.47 (1.00×)

We evaluate Qwen3-8B with SGLang at C1 over three seeds, adapting the official Lookahead algorithm to Qwen3 and SGLang. Cells report the accepted length (speedup over backend-matched AR). Lookahead's training-free proposals come with lower acceptance in these experiments.

W1.3 Training objective

I would like the authors to make these relationships explicit and explain how Equation (9) relates to the KL–TV hybrid objective in LK Losses.

Our objective combines DSpark's [3] token-level CE and L1L_1 terms with DFlash's [4] exponential position weighting. It is also related to the KL–TV hybrid objective in LK Losses [5]: our L1L_1 term is twice the TV distance, and the expected acceptance probability at a given prefix under speculative sampling is 1−TV(p,q)1-\mathrm{TV}(p,q). Our objective uses CE on the ground-truth token and fixed mixing coefficients (0.1 for CE and 0.9 for L1L_1), while the LK hybrid combines forward KL with TV using an adaptive mixing schedule.

We will make these relationships explicit in the paper.

W1.4 and Q2: Incremental update

Figure 3 omits the per-query log-normalizers described in Appx. A.2 and needed for the summary update [...] Please clearly show this state alongside the summaries and clarify the connection from the "Weighted sum" output to "Incremental update."

Thank you for pointing out the connection to online softmax [6]. Our incremental update uses the same principle: for each fixed summary query, we maintain a summary value and a log-normalizer. After verification, we compute the corresponding statistics for the newly retained KV rows and merge them with the stored state. The result is equivalent to softmax attention over the full prefix, while the summary update processes only the new rows.

We will explain this connection to online softmax in the main text. We will also revise Figure 3 to show the log-normalizers alongside the summaries and clarify how the weighted-sum output feeds into the incremental update.

AuthorsOpenReview ↗

Response (2 / 2)

W2. Adaptive verification

I could not tell whether this scheduler was disabled or whether seven was only a maximum length.

The main results use seven fixed draft tokens per round, with DSpark's adaptive scheduler disabled. We chose a fixed length because it achieved higher throughput on our hardware and at the serving loads evaluated here. Below, we compare it with adaptive DSpark using the SGLang community implementation:

WorkloadStatic DSparkAdaptive DSparkLongSpark
GSM8K6.17 (7,870.6)5.65 (7,071.5)5.93 (8,104.0)
MATH-5005.80 (8,269.9)5.80 (7,903.7)5.57 (8,644.4)
AIME254.99 (7,427.6)4.89 (7,101.3)4.72 (7,685.6)
MBPP5.18 (6,929.9)5.17 (6,620.3)5.03 (7,255.8)
HumanEval5.48 (7,547.7)5.48 (7,200.1)5.16 (7,718.5)
LiveCodeBench5.09 (5,476.4)5.10 (5,326.3)4.83 (5,605.5)
MT-Bench3.69 (5,347.1)3.60 (5,091.4)3.53 (5,589.7)
Alpaca3.59 (5,062.3)3.50 (4,809.1)3.46 (5,294.2)
LongSpec 32K3.34 (1,736.6)3.34 (1,708.7)3.32 (1,917.6)
CodeSpan 64K2.87 (1,217.3)2.86 (1,177.2)3.47 (1,571.8)

Following the paper's disaggregated setup, we use Qwen3-8B, T=1T=1, and one dedicated drafter GPU. Short tasks use TP1/C32 and three seeds; long-context tasks use TP4/C16 and ten seeds. Cells report accepted length (TPS). We will clarify the fixed-length setting in the paper.

Q1. Role of the global summary

If G is trained only with the objective in Eq. (9), how do you qualitatively show that G indeed plays the role of a global summary and improves draft acceptance efficiency?

$ G$ is trained jointly with the drafter using Eq. (9), without an auxiliary objective.

The summary is computed through attention over the full prefix:

 Mt=softmax⁡(GK1:tTd)V1:t.\ M_t = \operatorname{softmax}\left(\frac{ G K_{1:t}^{\mathsf T}}{\sqrt d}\right) V_{1:t}.

Each query in $ G$ produces a weighted sum of the target values, forming a fixed-size representation of the prefix. The drafter attends to these summary values alongside the recent KV window, giving each draft position access to information from earlier context.

We use MATH-500 as a representative case to examine the summary's role. The table shows the gain in the probability of accepting at least jj draft tokens when the global summary is included. The gain increases for later positions:

Position jj1234567
Gain (pp)1.502.954.154.855.686.116.52

Thank you for this question. We will add a detailed discussion and case study to the revised paper.


References

[1] Cheng et al. Recurrent Drafter for Fast Speculative Decoding in Large Language Models. arXiv:2403.09919, 2024. https://arxiv.org/abs/2403.09919

[2] Fu et al. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML, 2024. https://proceedings.mlr.press/v235/fu24a.html

[3] Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026. https://arxiv.org/abs/2607.05147

[4] Chen et al. DFlash: Block Diffusion for Flash Speculative Decoding. ICML, 2026. https://arxiv.org/abs/2602.06036

[5] Samarin et al. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding. arXiv:2602.23881, 2026. https://arxiv.org/abs/2602.23881

[6] Milakov and Gimelshein. Online Normalizer Calculation for Softmax. arXiv:1805.02867, 2018. https://arxiv.org/abs/1805.02867

Dear Reviewer,

With less than 12 hours remaining in the discussion period, please let us know if you have any remaining concerns or further questions. We would be happy to respond promptly.

Thank you for your time.

Best regards, The Authors

Reviewer_WTAiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LongSpark is a parallel speculative drafter that uses a fixed-size context interface: a target boundary state, a recent KV window, and an incrementally maintained global summary. It aims to keep draft-side context cost bounded as the prefix grows, and reports long-context throughput and latency results.

Critical strengths

The paper addresses a practical long-context serving bottleneck and reports end-to-end measurements and component ablations. The use of a bounded global summary alongside recent detail is a concrete approach to retaining context at fixed draft-side cost.

Critical weaknesses

The evidence does not yet isolate the benefit over a strong bounded-context drafter. The DSpark baseline in the paper appears to use context state that grows with sequence length, while DSpark has also been deployed with sliding-window attention. As a result, the large draft-state advantage and measured speedups over that baseline do not establish the advantage over a comparable fixed-window implementation. I lean weak reject pending a clearer, matched comparison.

Questions for authors

Could the authors clarify whether the Qwen3 DSpark baseline uses sliding-window attention and whether its reads of target-derived context are bounded? Please compare against a fixed-window DSpark under matched target model, hardware, context length, concurrency and timing boundaries. Report throughput, TPOT, acceptance length, peak memory and context/summary maintenance and communication costs. This would clarify the value of the global summary beyond a bounded recent window. A comparison with the later DLite history-free approach (https://arxiv.org/abs/2610.23787) would also be informative; I recognize that it appeared after the current submission version.

2 comments
AuthorsOpenReview ↗

Response (1 / 2)

Thank you for your thoughtful review. We address your concerns below.

Q1. Fixed-window DSpark

Could the authors clarify whether the Qwen3 DSpark baseline uses sliding-window attention and whether its reads of target-derived context are bounded? Please compare against a fixed-window DSpark under matched target model, hardware, context length, concurrency and timing boundaries.

The DSpark baseline uses the official Qwen3-8B checkpoint and attends to the full prefix. Following your suggestion, we also evaluate DSpark-SWA256, using the same checkpoint with attention restricted to the most recent 256 target-context tokens. We compare both with LongSpark under the paper's deployment and measurement settings:

BenchmarkLongSparkDSparkDSpark-SWA256
GSM8K5.93 (2.24×)6.17 (2.17×)6.01 (2.12×)
MATH-5005.57 (2.42×)5.80 (2.32×)5.46 (2.23×)
AIME254.72 (2.27×)4.99 (2.19×)4.57 (2.08×)
MBPP5.03 (1.98×)5.18 (1.89×)5.08 (1.87×)
HumanEval5.16 (2.11×)5.48 (2.06×)4.86 (1.87×)
LiveCodeBench4.83 (1.88×)5.09 (1.84×)4.79 (1.78×)
MT-Bench3.53 (1.54×)3.69 (1.47×)3.51 (1.42×)
Alpaca3.46 (1.49×)3.59 (1.42×)3.45 (1.38×)

Cells report accepted length (speedup over AR). Below, we compare the methods on long-context datasets:

Long-context benchmarkMethodAccepted lengthSpeedup (mean TPS / AR)TPSTPOT (ms)
LongSpec 32KLongSpark3.321.70×1,917.68.2
LongSpec 32KDSpark3.341.54×1,736.69.0
LongSpec 32KDSpark-SWA2562.891.52×1,714.69.2
CodeSpan 64KLongSpark3.471.78×1,571.810.0
CodeSpan 64KDSpark2.871.38×1,217.313.0
CodeSpan 64KDSpark-SWA2562.821.55×1,373.611.5

Following the paper's setup, we use L20X GPUs, T=1T=1, and a dedicated drafter GPU. The first table uses TP1/C32 and three seeds; the second uses TP4/C16 and ten seeds for all three methods and metrics.

Q2. Cost profile

Report throughput, TPOT, acceptance length, peak memory and context/summary maintenance and communication costs.

We separately profile memory, maintenance, and communication under the same deployment. Both tables report mean ± standard deviation over ten seeds.

ContextMethodPeak memory (GiB)Drafter maintenance (s)
32KLongSpark240.2 ± 0.23.32 ± 0.23
32KDSpark271.5 ± 0.46.52 ± 0.18
32KDSpark-SWA256271.4 ± 0.57.28 ± 0.19
64KLongSpark432.4 ± 0.22.60 ± 0.32
64KDSpark496.9 ± 0.55.13 ± 0.16
64KDSpark-SWA256493.8 ± 8.95.90 ± 0.17

Peak memory is the maximum synchronized NVML sum across four target GPUs and one drafter GPU. Maintenance is cumulative CUDA-event time over 120 seconds.

ContextMethodPrefill (MiB/request)Decode (KiB/token)Total (MiB/request)
32KLongSpark6.299 ± 0.000232.386 ± 4.111238.458 ± 4.107
32KDSpark1280.000 ± 0.000233.122 ± 6.8101512.894 ± 6.803
32KDSpark-SWA2561280.000 ± 0.000252.335 ± 5.4791532.088 ± 5.474
64KLongSpark6.299 ± 0.000238.842 ± 5.550244.908 ± 5.545
64KDSpark2560.000 ± 0.000275.253 ± 4.3052834.984 ± 4.301
64KDSpark-SWA2562560.000 ± 0.000273.497 ± 4.7162833.230 ± 4.711

Communication counts bidirectional tensor traffic with 1,024 output tokens per request. Decode is normalized by the 1,023 tokens after the first; total includes prefill and full decode. Our SWA implementation limits attention reads but retains and writes full-history KV; its costs reflect this implementation. The SGLang community has noted similar behavior for sliding-window DFlash.

AuthorsOpenReview ↗

Response ( 2 / 2)

Q3. DLite

A comparison with the later DLite history-free approach [...] would also be informative; I recognize that it appeared after the current submission version.

Thank you for the suggestion. We evaluated the official DLite [1] Qwen3-8B checkpoint and will discuss DLite in the revised paper. The results are shown below:

BenchmarkLongSparkDLite
GSM8K5.93 (8,104.0)5.82 (7,705.6)
MATH-5005.57 (8,644.4)5.51 (8,893.5)
AIME254.72 (7,685.6)4.68 (8,207.5)
MBPP5.03 (7,255.8)5.08 (7,214.8)
HumanEval5.16 (7,718.5)5.30 (7,974.1)
LiveCodeBench4.83 (5,605.5)5.03 (5,917.7)
MT-Bench3.53 (5,589.7)3.66 (5,958.3)
Alpaca3.46 (5,294.2)3.43 (5,403.5)

Results use concurrency 32 and three seeds. Cells report accepted length (TPS).

Long-context benchmarkMethodAccepted lengthTPS
LongSpec 32KLongSpark3.321,917.6
LongSpec 32KDLite2.591,692.1
CodeSpan 64KLongSpark3.471,571.8
CodeSpan 64KDLite2.191,218.8

Results use concurrency 16 and ten seeds.

Agentic workloads. Following DLite, we also evaluated AgentLongBench [2], MemoryAgentBench [3], and BFCL:

BenchmarkAR TPSLongSpark TPS (τ\tau)DLite TPS (τ\tau)DSpark TPS (τ\tau)
AgentLongBench119.73284.12 (3.777)257.82 (3.139)178.50 (2.315)
MemoryAgentBench102.26215.89 (2.955)197.23 (2.530)147.13 (2.076)
BFCL132.88306.88 (3.456)333.67 (3.458)264.21 (3.018)

Following DLite, we report decode TPS, excluding prefill and tool execution. Results use Qwen3-8B at T=0T=0 and C1, with three seeds; TPS is pooled across requests and seeds. Parentheses give accepted length. BFCL includes calls with more than 32K input tokens.

We note that this comparison is not fully controlled: DLite uses different training data and a different training procedure from LongSpark, including approximately 600K additional Nemotron samples and a two-stage procedure with a history-aware teacher. We will add these comparisons to the paper.


References

[1] Xu et al. DLite: History-free Speculative Decoding for Agentic AI. arXiv:2610.23787, 2026. https://arxiv.org/abs/2610.23787

[2] Fang et al. AgentLongBench: A Controllable Long Benchmark for Long-Contexts Agents via Environment Rollouts. arXiv:2601.20730, 2026. https://arxiv.org/abs/2601.20730

[3] Hu et al. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. ICLR, 2026. https://openreview.net/forum?id=DT7JyQC3MR

Reviewer_2GJMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LongSpark is a block-diffusion drafter (like DFlash/DSpark) whose per-round cost doesn't grow with context length. Instead of attending to the whole prefix, it reads fixed-size views of the target's state: the current hidden states, a 256-token recent KV window, and a 16-entry global summary per layer/head (learned fixed queries over the target's KV, updated incrementally). On Qwen3-4B/8B/14B, it has slightly lower acceptance length than DSpark but higher throughput (1.99x vs 1.92x on 8B), with larger gains at high concurrency and at 32K-128K context.

Critical strengths
  • Simple idea: the drafter doesn't need the full prefix since the target verifies everything. The incremental summary update is clean.
  • Broad evaluation: 3 model sizes, 8 short + 3 long-context benchmarks, several concurrency levels, ablations.
Critical weaknesses
  • Novelty: bounded-context / constant-state drafters already exist and are not cited:

    • MagicDec (Sadhukhan et al., 2024): fixed-budget sparse KV drafting for long context + large batch.
    • TriForce (Sun et al., COLM 2024): fixed-budget KV drafting for constant drafting cost.
    • Mamba Drafters (Choi et al., EMNLP Findings 2025): SSM drafter, near-constant time/memory with context length.

    The new part is mainly the learned-query summary added to a DSpark-style drafter.

  • Weak long-context baselines:

    • Missing: LongSpec (cited but not compared), MagicDec/TriForce.
    • Sliding window only (~"w/o global summary") is a strong simple baseline; the summary (learned fixed queries) adds only 4-6% TPS at 32K/64K (Table 5).
  • Small absolute gains compared to main baselines (DSpark):

    • Short context: 3-5% faster than DSpark at concurrency 32, slower at concurrency 8.
    • 128K: only 1.33x over autoregressive decoding.
  • Memory claim (406x smaller drafter state) is a bit misleading: the target KV for Qwen3-8B at 128K is ~18 GiB, so DSpark's 2.5 GiB is ~14% of total, and the overal savings resulting from LongSpark is ~10-15%.

Questions for authors
  • Comparison to LongSpec, MagicDec/TriForce, and sliding-window-only drafter? Just a subset of these are fine.
  • Results with drafter colocated on the target GPUs? The current setup has drafter running on a separate GPU which is not typical for vLLM / SGLang.
3 comments
AuthorsOpenReview ↗

Response (1 / 2)

Thank you for your constructive feedback. We address your concerns below.

Novelty

Bounded-context / constant-state drafters already exist and are not cited: [...] MagicDec [...] TriForce [...] Mamba Drafters.

Thank you for pointing out these related works. As Reviewer aP54 also suggested, we agree that LongSpark's contribution is better framed around its fixed-size multiscale context interface for a parallel block drafter. Its design combines the target's boundary state and recent KV window with a learned-query global summary, which is updated incrementally from target KV produced during verification. This gives the drafter access to both recent detail and information from the full prefix while keeping its additional context state and per-round drafting cost independent of prefix length.

We will revise the abstract and introduction accordingly, and add a discussion of MagicDec [1], TriForce [2], and Mamba Drafters [3].

Role of the global summary

The global summary improves draft acceptance and contributes to training stability, as shown in Figure 5 of the paper. We discuss its role and position-wise acceptance gains in our response to Reviewer aP54.

Magnitude of gains

Following Reviewer WTAi's suggestion, we also evaluated agentic workloads using DLite's decode-throughput metric. LongSpark improves throughput over DSpark by 59% on AgentLongBench and 47% on MemoryAgentBench; see our response to Reviewer WTAi for the complete results. We will add these comparisons to the revised paper.

Memory

Memory claim (406x smaller drafter state) is a bit misleading: the target KV for Qwen3-8B at 128K is ~18 GiB, so DSpark's 2.5 GiB is ~14% of total, and the overal savings resulting from LongSpark is ~10-15%.

We will revise the paper to clarify the distinction between drafter context state and total system memory.

Long-context baselines and global-summary ablation

Comparison to LongSpec, MagicDec/TriForce, and sliding-window-only drafter? Just a subset of these are fine.

Thank you for the suggestion. Following your suggestion, we evaluated LongSpec [4], Mamba, TriForce, Medusa [5], and ReDrafter [6] against LongSpark on Qwen3-0.6B at 32K and 64K.

Context / concurrencyARLongSparkLongSpecMambaTriForceMedusaReDrafter
32K / C1303.9 (1.000×)526.0 (1.731×)275.7 (0.907×)268.3 (0.882×)178.0 (0.586×)413.7 (1.361×)251.3 (0.827×)
32K / C4564.5 (1.000×)977.5 (1.732×)408.5 (0.724×)492.9 (0.873×)309.4 (0.548×)632.6 (1.121×)439.2 (0.778×)
64K / C1241.2 (1.000×)607.0 (2.517×)102.5 (0.425×)336.4 (1.394×)120.2 (0.499×)261.3 (1.084×)151.9 (0.630×)
64K / C4403.2 (1.000×)861.7 (2.137×)193.4 (0.480×)513.1 (1.272×)328.0 (0.813×)421.7 (1.046×)301.4 (0.748×)

All methods were trained on OpenPerfectBlend for two epochs, with the remaining training settings matching the paper, and evaluated over three seeds with TP1 and T=0T=0. TriForce uses the cache-v2 backend. Cells report mean TPS (speedup over AR). We will add these comparisons to the paper.

We also compare against DSpark-SWA256 and DLite, with the DLite comparison including three agentic workloads; see our response to Reviewer WTAi.

AuthorsOpenReview ↗

Response ( 2 / 2)

Colocated deployment

Results with drafter colocated on the target GPUs? The current setup has drafter running on a separate GPU which is not typical for vLLM / SGLang.

Thank you for this question. The SGLang and vLLM communities are exploring separate drafter and verifier engines and independent draft and verification services, respectively. Our main experiments adopt a disaggregated target–drafter deployment of this kind. Following your suggestion, we also evaluated colocated deployment, where the target and drafter share a GPU:

DatasetLongSparkDSpark
GSM8K5.98 / 10,537.66.20 / 10,865.8
MATH-5005.49 / 11,285.85.75 / 11,206.3
AIME254.75 / 9,804.74.98 / 9,598.7
MBPP5.06 / 9,715.45.21 / 9,863.4
HumanEval5.15 / 9,826.95.49 / 10,138.6
LiveCodeBench4.82 / 5,892.25.13 / 6,001.8
MT-Bench3.53 / 7,434.83.69 / 7,447.8
Alpaca3.47 / 7,160.83.60 / 7,232.5

Results use concurrency 32. Cells report accepted length / TPS.

We also evaluated colocated deployment on long-context workloads:

Dataset / concurrencyARLongSparkDSpark
LongSpec 32K / C16— / 517.13.62 / 915.53.52 / 878.0
CodeSpan 64K / C8— / 271.13.70 / 603.32.81 / 455.3

Cells report accepted length / TPS. We will add these results to the revised paper.


References

[1] Sadhukhan et al. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding. ICLR, 2025. https://arxiv.org/abs/2408.11049

[2] Sun et al. TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding. COLM, 2024. https://arxiv.org/abs/2404.11912

[3] Choi et al. Mamba Drafters for Speculative Decoding. Findings of EMNLP, 2025. https://arxiv.org/abs/2506.01206

[4] Yang et al. LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification. ACL, 2026. https://aclanthology.org/2026.acl-long.83/

[5] Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML, 2024. https://proceedings.mlr.press/v235/cai24b.html

[6] Cheng et al. Recurrent Drafter for Fast Speculative Decoding in Large Language Models. arXiv:2403.09919, 2024. https://arxiv.org/abs/2403.09919

Dear Reviewer,

With less than 12 hours remaining in the discussion period, please let us know if you have any remaining concerns or further questions. We would be happy to respond promptly.

Thank you for your time.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision summary

Dear Reviewers and Area Chairs,

We thank the area chairs for their time and the reviewers for their careful reading and constructive feedback. We have revised the paper accordingly, and the updated manuscript has been uploaded. Below is a summary of the changes:

1. Clarified the contribution and related work (aP54, 2GJM). We now distinguish LongSpark from prior fixed-cost and bounded-context drafters. The revised paper emphasizes its combination of a boundary state, a recent KV window, and a learned global summary. We also expanded the discussion of prior drafters and Jacobi-style methods, and clarified the connections between our training objective, summary update, and prior work.

2. Added bounded-context and alternative baselines (aP54, WTAi, 2GJM). We added DSpark-SWA, which limits DSpark to the same recent-window size as LongSpark, as well as comparisons with DLite, LongSpec, Mamba, TriForce, Medusa, and ReDrafter.

3. Expanded the serving evaluation (aP54, WTAi, 2GJM). We clarified the fixed verification horizon and added comparisons with DSpark’s adaptive verification. We also evaluated colocated target–drafter deployment and reported memory, context-maintenance, communication, and completion-time measurements.

4. Explained and analyzed the global summary (aP54, 2GJM). We clarified how summary queries are trained and how the summary and its log-normalizers are updated. We also added a position-wise acceptance analysis and a qualitative case study, alongside the long-context ablations, to illustrate how the summary contributes beyond the recent window.

We thank the reviewers again for their valuable feedback, which has helped us improve the paper. We hope the revised manuscript addresses their concerns. If the reviewers have any further questions, we would be happy to address them promptly.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.