- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces LongSpark, a parallel block drafter whose additional decoding computation and context state are independent of prefix length for a fixed architecture and block size. It materializes the given context into three fixed-size, learned context features, each extracted over a different time period: short (into a boundary state), mid (into a recent KV-cache window), and long (into a global summary). Experiments with Qwen3-4B, 8B, and 14B report favorable throughput and latency on short- and long-context workloads by keeping drafting cost constant, though it may slightly reduce the accepted lengths.
Critical strengths
The paper provides abundant numerical evaluations of the proposed LongSpark. They suggest that the constant drafting cost can outweigh a lower accepted length (in particular in Table 1). Moreover, Figures 5a and 5b, along with the long-context ablations in Table 5, are especially helpful for understanding the choice of three context features. I find it interesting that, although the global summary contributes the smallest degradation in (Figure 5a), it also contributes to stability in draft training (Figure 5b). In addition, I appreciate that the paper reports the low-concurrency regime where DSpark remains faster.
Critical weaknesses
W1. This work's novelty needs clearer positioning relative to prior work
The current manuscript does not properly reference prior work in several aspects. I think the authors should be more honest about what is already known in prior work; without referencing it, especially in the description of the proposed methodology, the manuscript risks giving the impression that it is new.
-
Novelty of fixed-cost drafting. I am not convinced that fixed-cost drafting itself is a major new contribution, as the abstract and introduction suggest. Indeed, Sec. 4 already acknowledges that Medusa generates proposals from the target's last hidden state at prefix-independent cost for a fixed number of prediction heads. But not only that, ReDrafter also conditions a recurrent draft head on the target's last hidden state and token embeddings, giving prefix-independent proposal cost for fixed draft length and beam width. Thus, the idea of prefix-independent proposal computation already appears in earlier works. I think LongSpark's distinctive contribution is the design that lets its block drafter use richer multiscale context with prefix-independent additional computation and context state. I want the abstract and introduction to emphasize this design and its demonstrated benefits as LongSpark's contribution.
-
Comparison with Jacobi-based decoding. The emphasis on drafting cost also makes me wonder how LongSpark compares with Jacobi-based decoding, which neither Sec. 4 nor the experiments consider. Speculative Jacobi Decoding (SJD) uses the same target forward pass to verify candidates and obtain the next proposals, without a separate drafter. Speculative Coupled Decoding (SCD) improves SJD through coupling across iterations while preserving the target distribution. Moreover, I would say Lookahead Decoding, an earlier method that uses Jacobi iterations for language generation, is a more direct baseline.
-
Training objective. Equation (9) combines CE and terms with the same 0.1/0.9 coefficients as DSpark's training objective. Its exponential position weighting also appears in DFlash. I would like the authors to make these relationships explicit and explain how Equation (9) relates to the KL–TV hybrid objective in LK Losses, particularly its use of token-level CE and a fixed mixing coefficient.
-
The markov incremental update. The "incremental update" mentioned in Sec. 2 and described in Appx. A.2 is basically a slightly generalized, alternative formulation of renowned online softmax (e.g., see the FlashAttention paper). The main text should also explain why it uses the formulas in Appx. A.2. (e.g., to obtain a standard softmax attention over the previous contexts and the new incoming block with constant time & memory).
W2. The serving comparison needs a clearer account of the baseline
- DSpark's adaptive verification. As I read Sec. 3.1, all speculative methods use seven draft tokens per verification round. DSpark includes a hardware-aware scheduler that adapts verification length to reduce wasted verification under high serving load (Cheng et al., Sec. 3.2). I could not tell whether this scheduler was disabled or whether seven was only a maximum length.
Questions for authors
- The paper refers to the matrix G as a bank of "summary queries"; is it only a metaphorical name, or does it have its own secret (autoencoder-like) objective function to drive G to be an effective global context summary? If G is trained only with the objective in Eq. (9), how do you qualitatively show that G indeed plays the role of a global summary and improves draft acceptance efficiency?
- Figure 3 omits the per-query log-normalizers described in Appx. A.2 and needed for the summary update; is it intentional? Please clearly show this state alongside the summaries and clarify the connection from the "Weighted sum" output to "Incremental update."
6 comments
Response (2 / 2)
W2. Adaptive verification
I could not tell whether this scheduler was disabled or whether seven was only a maximum length.
The main results use seven fixed draft tokens per round, with DSpark's adaptive scheduler disabled. We chose a fixed length because it achieved higher throughput on our hardware and at the serving loads evaluated here. Below, we compare it with adaptive DSpark using the SGLang community implementation:
| Workload | Static DSpark | Adaptive DSpark | LongSpark |
|---|---|---|---|
| GSM8K | 6.17 (7,870.6) | 5.65 (7,071.5) | 5.93 (8,104.0) |
| MATH-500 | 5.80 (8,269.9) | 5.80 (7,903.7) | 5.57 (8,644.4) |
| AIME25 | 4.99 (7,427.6) | 4.89 (7,101.3) | 4.72 (7,685.6) |
| MBPP | 5.18 (6,929.9) | 5.17 (6,620.3) | 5.03 (7,255.8) |
| HumanEval | 5.48 (7,547.7) | 5.48 (7,200.1) | 5.16 (7,718.5) |
| LiveCodeBench | 5.09 (5,476.4) | 5.10 (5,326.3) | 4.83 (5,605.5) |
| MT-Bench | 3.69 (5,347.1) | 3.60 (5,091.4) | 3.53 (5,589.7) |
| Alpaca | 3.59 (5,062.3) | 3.50 (4,809.1) | 3.46 (5,294.2) |
| LongSpec 32K | 3.34 (1,736.6) | 3.34 (1,708.7) | 3.32 (1,917.6) |
| CodeSpan 64K | 2.87 (1,217.3) | 2.86 (1,177.2) | 3.47 (1,571.8) |
Following the paper's disaggregated setup, we use Qwen3-8B, , and one dedicated drafter GPU. Short tasks use TP1/C32 and three seeds; long-context tasks use TP4/C16 and ten seeds. Cells report accepted length (TPS). We will clarify the fixed-length setting in the paper.
Q1. Role of the global summary
If G is trained only with the objective in Eq. (9), how do you qualitatively show that G indeed plays the role of a global summary and improves draft acceptance efficiency?
$ G$ is trained jointly with the drafter using Eq. (9), without an auxiliary objective.
The summary is computed through attention over the full prefix:
Each query in $ G$ produces a weighted sum of the target values, forming a fixed-size representation of the prefix. The drafter attends to these summary values alongside the recent KV window, giving each draft position access to information from earlier context.
We use MATH-500 as a representative case to examine the summary's role. The table shows the gain in the probability of accepting at least draft tokens when the global summary is included. The gain increases for later positions:
| Position | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|
| Gain (pp) | 1.50 | 2.95 | 4.15 | 4.85 | 5.68 | 6.11 | 6.52 |
Thank you for this question. We will add a detailed discussion and case study to the revised paper.
References
[1] Cheng et al. Recurrent Drafter for Fast Speculative Decoding in Large Language Models. arXiv:2403.09919, 2024. https://arxiv.org/abs/2403.09919
[2] Fu et al. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML, 2024. https://proceedings.mlr.press/v235/fu24a.html
[3] Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026. https://arxiv.org/abs/2607.05147
[4] Chen et al. DFlash: Block Diffusion for Flash Speculative Decoding. ICML, 2026. https://arxiv.org/abs/2602.06036
[5] Samarin et al. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding. arXiv:2602.23881, 2026. https://arxiv.org/abs/2602.23881
[6] Milakov and Gimelshein. Online Normalizer Calculation for Softmax. arXiv:1805.02867, 2018. https://arxiv.org/abs/1805.02867
Dear Reviewer,
With less than 12 hours remaining in the discussion period, please let us know if you have any remaining concerns or further questions. We would be happy to respond promptly.
Thank you for your time.
Best regards, The Authors
Response (1 / 2)
Thank you for your careful review and helpful suggestions. We address your concerns below.
W1.1 Fixed-cost drafting
Thank you for this clarification. We agree that LongSpark's contribution is better framed around its fixed-size multiscale context interface for a parallel block drafter. We will revise the abstract and introduction accordingly, and add a citation and discussion of ReDrafter [1].
We also compare LongSpark with Medusa and ReDrafter on long-context workloads; see our response to Reviewer 2GJM.
W1.2 Jacobi-based decoding
Thank you for this suggestion. Indeed, Jacobi-based decoding does not require a separately trained drafter and has low drafting overhead. We will add a discussion of these methods and include Lookahead [2] as a more direct baseline. The results are shown below:
We evaluate Qwen3-8B with SGLang at C1 over three seeds, adapting the official Lookahead algorithm to Qwen3 and SGLang. Cells report the accepted length (speedup over backend-matched AR). Lookahead's training-free proposals come with lower acceptance in these experiments.
W1.3 Training objective
Our objective combines DSpark's [3] token-level CE and L1 terms with DFlash's [4] exponential position weighting. It is also related to the KL–TV hybrid objective in LK Losses [5]: our L1 term is twice the TV distance, and the expected acceptance probability at a given prefix under speculative sampling is 1−TV(p,q). Our objective uses CE on the ground-truth token and fixed mixing coefficients (0.1 for CE and 0.9 for L1), while the LK hybrid combines forward KL with TV using an adaptive mixing schedule.
We will make these relationships explicit in the paper.
W1.4 and Q2: Incremental update
Thank you for pointing out the connection to online softmax [6]. Our incremental update uses the same principle: for each fixed summary query, we maintain a summary value and a log-normalizer. After verification, we compute the corresponding statistics for the newly retained KV rows and merge them with the stored state. The result is equivalent to softmax attention over the full prefix, while the summary update processes only the new rows.
We will explain this connection to online softmax in the main text. We will also revise Figure 3 to show the log-normalizers alongside the summaries and clarify how the weighted-sum output feeds into the incremental update.