- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
QSpar proposes a training-free self-speculative decoding framework for long-context LLM inference. It addresses the weight access bottleneck(memory bound) that remains after sparse attention reduces KV-cache read, particularly at small batch sizes. The method combines a weight quantized sparse drafter with a Reranker that recomputes selected candidate logits using the BF16 LM head and a margin-aware Reducer that retains limited alternative tokens for tree verification.
Critical strengths
1.QSpar achieves an average 1.49× speedup over Vegas under a fixed KV-cache budget while preserving the target’s greedy outputs, demonstrating practical value for long-context inference.
2.Candidate reranking and selective alternative retention recover acceptance at low overhead, with error analysis and ablations supporting the effectiveness of both components.
Critical weaknesses
1.The evaluation omits drafters such as DFlash and EAGLE-3. Comparisons with vLLM and sparse self-speculation baselines leave unclear when QSpar offers a preferable deployment trade-off, limiting the demonstrated practical significance.
Questions for authors
1.Given that DFlash is a widely used and extensively evaluated drafter, the authors should compare QSpar against it under matched experimental conditions and clarify the deployment scenarios in which each method is preferable.
4 comments
Short-context and Thinking-mode Performance. In contrast, DFlash achieves higher throughput on AIME25, where the initial input contexts are relatively short (Table 2). On Qwen3-8B with thinking mode disabled, DFlash achieves a 5.68× speedup, compared with 1.37× for QSpar. This can be attributed to DFlash's high acceptance length resulting from strong drafter–verifier alignment and its low parallel drafting cost. In contrast, QSpar maintains a high acceptance length in short-context settings, but achieves lower speedups than DFlash because sparse attention provides limited acceleration, leaving the execution cost of its target-sized drafter relatively high.
Enabling thinking mode with the same DFlash checkpoint reduces its acceptance length from 8.69 to 3.67, accompanied by a speedup reduction from 5.68× to 2.48×. In contrast, QSpar maintains a relatively stable acceptance length, decreasing only from 10.01 to 8.86. Furthermore, despite using a smaller drafter than QSpar, EAGLE-3 achieves lower acceptance lengths and throughput across all evaluated Qwen3-8B configurations. These results suggest that when drafter–verifier alignment degrades, reduced acceptance can limit the performance benefits of training-based methods despite their low drafting costs.
Table 2. Short-context performance on AIME25. Speedup over BF16 AR decoding, with average acceptance length in parentheses.
\hline \begin{array}{c}\mathrm{Model}\\\\\mathrm{Thinking\\ Mode}\end{array} & \begin{array}{c}\mathrm{QSpar\\ (Vegas)}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 60\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 15\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 10\end{array} \\\\ \hline \mathrm{Qwen3-8B\\ (off)} & 1.37\times\\,(10.01) & 1.09\times\\,(3.66) & 1.29\times\\,(2.48) & 5.68\times\\,(8.69) & 4.70\times\\,(7.18) \\\\ \mathrm{Qwen3-8B\\ (on)} & 1.28\times\\,(8.86) & 1.06\times\\,(3.31) & 1.24\times\\,(2.11) & 2.48\times\\,(3.67) & 2.43\times\\,(3.54) \\\\ \hline \mathrm{Llama3.1-8B} & 1.35\times\\,(9.95) & 1.82\times\\,(6.18) & 1.23\times\\,(3.72) & 2.64\times\\,(4.75) & 2.18\times\\,(4.34) \\\\ \hline \end{array}$$ **Deployment Trade-offs.** DFlash achieves higher speedups in short-context settings with strong drafter–verifier alignment, while QSpar performs better in long-context settings and maintains relatively stable acceptance lengths across thinking modes without additional training. These results suggest a hybrid deployment strategy for serving environments with dynamically varying request characteristics, switching between learned drafters and QSpar based on context length and drafter–verifier alignment. Therefore, QSpar provides not only a competitive training-free alternative for long-context deployment but also a promising component of hybrid speculative decoding systems. We will incorporate these comparisons and detailed experimental settings into the Evaluation section and discuss the hybrid deployment strategy as future work. [1] https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16?utm_source=chatgpt.com [2] https://huggingface.co/thoughtworks/Qwen3-8B-Eagle3?utm_source=chatgpt.com [3] https://huggingface.co/z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat?utm_source=chatgpt.com [4] https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B?utm_source=chatgpt.comThe authors have addressed my questions and provided satisfactory responses. I indicated a rating of Weak Accept. Best wishes.
Thank you for your positive assessment and constructive feedback. We are glad the additional results addressed your main concerns, and we will incorporate them into the revised paper.
Thank you for your careful and constructive review. We appreciate the time you spent evaluating our work and the helpful suggestions you provided. Your feedback also helped us identify promising opportunities to extend our work toward practical serving systems. We address your points below.
W1 & Q1. Comparison with Training-Based Speculative Decoding
To clarify QSpar's practical deployment trade-offs, we conducted additional comparisons with two representative training-based speculative decoding methods, DFlash and EAGLE-3.
Experimental Setup. We used publicly available target-specific drafter checkpoints for Qwen3-8B [1, 2] and Llama-3.1-8B [3, 4] for DFlash and EAGLE-3. We evaluated both methods using their original draft budgets and configurations with draft budgets matched to QSpar. On RULER, we varied the context length from 8K to 128K while maintaining a fixed aggregate KV-cache token budget of (B × L = 128K). On AIME25, following DFlash's experimental setup, we measured end-to-end decode throughput and acceptance length with up to 2,048 output tokens.
Long-context Performance. On RULER (Table 1), the acceptance lengths of DFlash and EAGLE-3 generally decrease as context length increases, whereas QSpar maintains high acceptance lengths. Consequently, QSpar-Vegas achieves the highest acceptance length and speedup across all evaluated configurations. At 128K context, QSpar achieves a 2.21× speedup on Qwen3-8B, outperforming DFlash (1.33×) and EAGLE-3 (0.52×). Similarly, on Llama-3.1-8B, QSpar achieves a 1.93× speedup, compared with 1.18× for DFlash.
Table 1. Long-context performance on RULER. Speedup over BF16 autoregressive (AR) decoding, with average acceptance length in parentheses.
\hline \mathrm{Model} & \mathrm{Length} & \begin{array}{c}\mathrm{QSpar\\ (Vegas)}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 60\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 15\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 10 \end{array} \\\\ \hline \mathrm{Qwen3-8B} & 8K & 1.51\times\\,(9.76) & 0.43\times\\,(4.03) & 0.95\times\\,(3.00) & 1.49\times\\,(4.07) & 1.44\times\\,(3.51) \\\\ & 16K & 1.78\times\\,(9.62) & 0.52\times\\,(3.86) & 0.89\times\\,(2.81) & 1.44\times\\,(2.71) & 1.24\times\\,(2.39) \\\\ & 32K & 1.97\times\\,(9.40) & 0.64\times\\,(3.39) & 0.73\times\\,(2.36) & 1.39\times\\,(2.44) & 1.24\times\\,(2.26) \\\\ & 64K & 1.98\times\\,(9.02) & 0.65\times\\,(3.26) & 0.53\times\\,(1.94) & 1.41\times\\,(2.20) & 1.31\times\\,(1.98) \\\\ & 128K & 2.21\times\\,(9.24) & 0.52\times\\,(2.48) & 0.26\times\\,(1.48) & 1.33\times\\,(1.81) & 1.26\times\\,(1.71) \\\\ \hline \mathrm{Llama3.1-8B} & 8K & 1.22\times\\,(8.38) & 0.15\times\\,(1.33) & 0.40\times\\,(1.13) & 0.85\times\\,(2.14) & 0.87\times\\,(2.14) \\\\ & 16K & 1.39\times\\,(8.51) & 0.17\times\\,(1.16) & 0.38\times\\,(1.03) & 1.01\times\\,(2.04) & 0.97\times\\,(2.02) \\\\ & 32K & 1.60\times\\,(8.85) & 0.23\times\\,(1.17) & 0.34\times\\,(1.04) & 1.13\times\\,(1.95) & 1.07\times\\,(1.92) \\\\ & 64K & 1.82\times\\,(9.38) & 0.24\times\\,(1.17) & 0.30\times\\,(1.04) & 1.19\times\\,(1.93) & 1.14\times\\,(1.88) \\\\ & 128K & 1.93\times\\,(8.71) & 0.26\times\\,(1.18) & 0.19\times\\,(1.04) & 1.17\times\\,(1.64) & 1.18\times\\,(1.62) \\\\ \hline \end{array}$$