acceptodds
Under review as a conference paper at ICLR 2027

QSpar: Co-Designing Sparse Attention and Quantized Weights for Long-Context Self-Speculative Decoding

Abstract

Reducing KV-cache traffic alone is insufficient to eliminate drafting overhead in long-context self-speculative decoding. Under a fixed GPU memory budget, longer contexts constrain the feasible batch size, limiting the amortization of weight access across requests. A sparse-attention-based drafter further reduces draft-side KV-cache traffic but leaves repeated access to the target-sized drafter weights unchanged, exposing a substantial weight-side bottleneck. Low-bit weight quantization reduces this cost, but naively combining it with sparse attention degrades acceptance length. We address this runtime–acceptance trade-off through two key observations: token flips concentrate at low-margin positions, and the BF16 target token usually remains among a few top-ranked draft candidates. Based on these observations, QSpar combines a weight-quantized sparse drafter with a candidate Reranker and a margin-aware Reducer. The Reranker recomputes candidate logits using BF16 LM-head weights, while the Reducer selectively retains alternatives under a bounded candidate budget. Final verification uses the original full-attention BF16 model to preserve the exact target-model generation. Under a fixed KV-cache budget, QSpar achieves average and maximum end-to-end speedups over Vegas, and up to over vLLM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Quantization, Sparse Attention, KV Cache, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aR3AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QSpar proposes a training-free self-speculative decoding framework for long-context LLM inference. It addresses the weight access bottleneck(memory bound) that remains after sparse attention reduces KV-cache read, particularly at small batch sizes. The method combines a weight quantized sparse drafter with a Reranker that recomputes selected candidate logits using the BF16 LM head and a margin-aware Reducer that retains limited alternative tokens for tree verification.

Critical strengths

1.QSpar achieves an average 1.49× speedup over Vegas under a fixed KV-cache budget while preserving the target’s greedy outputs, demonstrating practical value for long-context inference.

2.Candidate reranking and selective alternative retention recover acceptance at low overhead, with error analysis and ablations supporting the effectiveness of both components.

Critical weaknesses

1.The evaluation omits drafters such as DFlash and EAGLE-3. Comparisons with vLLM and sparse self-speculation baselines leave unclear when QSpar offers a preferable deployment trade-off, limiting the demonstrated practical significance.

Questions for authors

1.Given that DFlash is a widely used and extensively evaluated drafter, the authors should compare QSpar against it under matched experimental conditions and clarify the deployment scenarios in which each method is preferable.

4 comments

Thank you for your careful and constructive review. We appreciate the time you spent evaluating our work and the helpful suggestions you provided. Your feedback also helped us identify promising opportunities to extend our work toward practical serving systems. We address your points below.

W1 & Q1. Comparison with Training-Based Speculative Decoding

“The evaluation omits drafters such as DFlash and EAGLE-3. Comparisons with vLLM and sparse self-speculation baselines leave unclear when QSpar offers a preferable deployment trade-off, limiting the demonstrated practical significance.”

”Given that DFlash is a widely used and extensively evaluated drafter, the authors should compare QSpar against it under matched experimental conditions and clarify the deployment scenarios in which each method is preferable.”

To clarify QSpar's practical deployment trade-offs, we conducted additional comparisons with two representative training-based speculative decoding methods, DFlash and EAGLE-3.

Experimental Setup. We used publicly available target-specific drafter checkpoints for Qwen3-8B [1, 2] and Llama-3.1-8B [3, 4] for DFlash and EAGLE-3. We evaluated both methods using their original draft budgets and configurations with draft budgets matched to QSpar. On RULER, we varied the context length from 8K to 128K while maintaining a fixed aggregate KV-cache token budget of (B ×\times L = 128K). On AIME25, following DFlash's experimental setup, we measured end-to-end decode throughput and acceptance length with up to 2,048 output tokens.

Long-context Performance. On RULER (Table 1), the acceptance lengths of DFlash and EAGLE-3 generally decrease as context length increases, whereas QSpar maintains high acceptance lengths. Consequently, QSpar-Vegas achieves the highest acceptance length and speedup across all evaluated configurations. At 128K context, QSpar achieves a 2.21× speedup on Qwen3-8B, outperforming DFlash (1.33×) and EAGLE-3 (0.52×). Similarly, on Llama-3.1-8B, QSpar achieves a 1.93× speedup, compared with 1.18× for DFlash.

Table 1. Long-context performance on RULER. Speedup over BF16 autoregressive (AR) decoding, with average acceptance length in parentheses.

\hline \mathrm{Model} & \mathrm{Length} & \begin{array}{c}\mathrm{QSpar\\ (Vegas)}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 60\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 15\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 10 \end{array} \\\\ \hline \mathrm{Qwen3-8B} & 8K & 1.51\times\\,(9.76) & 0.43\times\\,(4.03) & 0.95\times\\,(3.00) & 1.49\times\\,(4.07) & 1.44\times\\,(3.51) \\\\ & 16K & 1.78\times\\,(9.62) & 0.52\times\\,(3.86) & 0.89\times\\,(2.81) & 1.44\times\\,(2.71) & 1.24\times\\,(2.39) \\\\ & 32K & 1.97\times\\,(9.40) & 0.64\times\\,(3.39) & 0.73\times\\,(2.36) & 1.39\times\\,(2.44) & 1.24\times\\,(2.26) \\\\ & 64K & 1.98\times\\,(9.02) & 0.65\times\\,(3.26) & 0.53\times\\,(1.94) & 1.41\times\\,(2.20) & 1.31\times\\,(1.98) \\\\ & 128K & 2.21\times\\,(9.24) & 0.52\times\\,(2.48) & 0.26\times\\,(1.48) & 1.33\times\\,(1.81) & 1.26\times\\,(1.71) \\\\ \hline \mathrm{Llama3.1-8B} & 8K & 1.22\times\\,(8.38) & 0.15\times\\,(1.33) & 0.40\times\\,(1.13) & 0.85\times\\,(2.14) & 0.87\times\\,(2.14) \\\\ & 16K & 1.39\times\\,(8.51) & 0.17\times\\,(1.16) & 0.38\times\\,(1.03) & 1.01\times\\,(2.04) & 0.97\times\\,(2.02) \\\\ & 32K & 1.60\times\\,(8.85) & 0.23\times\\,(1.17) & 0.34\times\\,(1.04) & 1.13\times\\,(1.95) & 1.07\times\\,(1.92) \\\\ & 64K & 1.82\times\\,(9.38) & 0.24\times\\,(1.17) & 0.30\times\\,(1.04) & 1.19\times\\,(1.93) & 1.14\times\\,(1.88) \\\\ & 128K & 1.93\times\\,(8.71) & 0.26\times\\,(1.18) & 0.19\times\\,(1.04) & 1.17\times\\,(1.64) & 1.18\times\\,(1.62) \\\\ \hline \end{array}$$

Short-context and Thinking-mode Performance. In contrast, DFlash achieves higher throughput on AIME25, where the initial input contexts are relatively short (Table 2). On Qwen3-8B with thinking mode disabled, DFlash achieves a 5.68× speedup, compared with 1.37× for QSpar. This can be attributed to DFlash's high acceptance length resulting from strong drafter–verifier alignment and its low parallel drafting cost. In contrast, QSpar maintains a high acceptance length in short-context settings, but achieves lower speedups than DFlash because sparse attention provides limited acceleration, leaving the execution cost of its target-sized drafter relatively high.

Enabling thinking mode with the same DFlash checkpoint reduces its acceptance length from 8.69 to 3.67, accompanied by a speedup reduction from 5.68× to 2.48×. In contrast, QSpar maintains a relatively stable acceptance length, decreasing only from 10.01 to 8.86. Furthermore, despite using a smaller drafter than QSpar, EAGLE-3 achieves lower acceptance lengths and throughput across all evaluated Qwen3-8B configurations. These results suggest that when drafter–verifier alignment degrades, reduced acceptance can limit the performance benefits of training-based methods despite their low drafting costs.

Table 2. Short-context performance on AIME25. Speedup over BF16 AR decoding, with average acceptance length in parentheses.

\hline \begin{array}{c}\mathrm{Model}\\\\\mathrm{Thinking\\ Mode}\end{array} & \begin{array}{c}\mathrm{QSpar\\ (Vegas)}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 60\end{array} & \begin{array}{c}\mathrm{EAGLE3}\\\\\gamma 10\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 15\end{array} & \begin{array}{c}\mathrm{DFlash}\\\\\gamma 10\end{array} \\\\ \hline \mathrm{Qwen3-8B\\ (off)} & 1.37\times\\,(10.01) & 1.09\times\\,(3.66) & 1.29\times\\,(2.48) & 5.68\times\\,(8.69) & 4.70\times\\,(7.18) \\\\ \mathrm{Qwen3-8B\\ (on)} & 1.28\times\\,(8.86) & 1.06\times\\,(3.31) & 1.24\times\\,(2.11) & 2.48\times\\,(3.67) & 2.43\times\\,(3.54) \\\\ \hline \mathrm{Llama3.1-8B} & 1.35\times\\,(9.95) & 1.82\times\\,(6.18) & 1.23\times\\,(3.72) & 2.64\times\\,(4.75) & 2.18\times\\,(4.34) \\\\ \hline \end{array}$$ **Deployment Trade-offs.** DFlash achieves higher speedups in short-context settings with strong drafter–verifier alignment, while QSpar performs better in long-context settings and maintains relatively stable acceptance lengths across thinking modes without additional training. These results suggest a hybrid deployment strategy for serving environments with dynamically varying request characteristics, switching between learned drafters and QSpar based on context length and drafter–verifier alignment. Therefore, QSpar provides not only a competitive training-free alternative for long-context deployment but also a promising component of hybrid speculative decoding systems. We will incorporate these comparisons and detailed experimental settings into the Evaluation section and discuss the hybrid deployment strategy as future work. [1] https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16?utm_source=chatgpt.com [2] https://huggingface.co/thoughtworks/Qwen3-8B-Eagle3?utm_source=chatgpt.com [3] https://huggingface.co/z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat?utm_source=chatgpt.com [4] https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B?utm_source=chatgpt.com

The authors have addressed my questions and provided satisfactory responses. I indicated a rating of Weak Accept. Best wishes.

Thank you for your positive assessment and constructive feedback. We are glad the additional results addressed your main concerns, and we will incorporate them into the revised paper.

Reviewer_7k6FOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QSpar for long-context self-speculative decoding where GPU memory is shared by weights and KV cache. It combines sparse attention, low-bit drafter weights, Reranker and Reducer mechanisms with a BF16 verifier, delivering up to 2.21× speedup over vLLM.

Critical strengths
  1. The paper addresses a real serving bottleneck rather than treating draft latency as the only cost in speculative decoding.

  2. The paper presents a plausible and practically relevant systems idea.

Critical weaknesses
  1. The strongest end-to-end evidence comes from Qwen3-8B on RULER with B x L fixed at 128K. This controls allocated KV tokens, but it is not the same as a mixed-length concurrent serving trace.

  2. The analysis provides no formal performance guarantee and no complete exactness statement.

Questions for authors
  1. The manuscript should distinguish lossless target output from acceptance-length recovery. The drafter and candidate tree remain approximate even when the original BF16 verifier produces the same greedy output. A token-by-token identity test against ordinary BF16 decoding would make this claim concrete.

  2. The policy parameters appear to be selected retrospectively. Figure 2 uses one model, one sparse-attention method, AWQ, and RULER to motivate K_rerank=8 and tau=2, while the deployed reducer uses delta_rerank rather than the verifier margin delta_ver shown in the analysis. The paper should provide a held-out calibration protocol, quantify the relationship between these two margins, report confidence intervals for top-K target recall, and show whether the same settings remain competitive on models and workloads not used for selection. The appendix already indicates that tau, M, K_reducer, and draft length interact substantially.

5 comments

We sincerely thank you for your thoughtful and constructive review. We greatly appreciate the time and effort you devoted to evaluating our work. Your valuable feedback has helped us further strengthen our evaluation under realistic serving conditions, clarify QSpar's output exactness, and provide more rigorous validation of its policy parameters. We address each of your comments in detail below.

W1. Evaluation under Mixed-Length Concurrent Serving

“The strongest end-to-end evidence comes from Qwen3-8B on RULER with B x L fixed at 128K. This controls allocated KV tokens, but it is not the same as a mixed-length concurrent serving trace.”

While Figure 4 evaluates requests with uniform context lengths under a fixed KV-cache budget, real-world serving systems process concurrent requests with varying context lengths. To examine whether QSpar maintains its performance benefits in such settings, we conducted additional experiments with Mixed-Length Serving Performance.

Experimental Setup. We evaluated Qwen3-8B and Llama-3.1-8B by batching RULER requests with different context lengths (8K, 16K, 32K, and 64K) under a fixed aggregate KV-cache token budget of 128K. Each request generates 128 output tokens, and we report the average decode throughput across three random request-order seeds. We constructed three workloads with different request-length distributions:

  • Uniform: Requests are evenly distributed across the four context lengths (1:1:1:1).
  • Trace-inspired (Mooncake): We derived the request-length distribution from the publicly available Mooncake (Kimi) serving trace [1]. After excluding requests shorter than 4K tokens and mapping the remaining requests to the nearest context length, the resulting distribution for 64K:32K:16K:8K was approximately 3:6:13:78%. We approximated this distribution as 1:2:4:24 for our experiments.
  • Multi-turn-inspired: We modeled a scenario where context length increases by a fixed number of tokens per turn. Assuming uniformly distributed context lengths, the number of requests in each interval ((L/2,L]) is proportional to (L), yielding a length ratio of 8:4:2:1 for 64K:32K:16K:8K.

Mixed-Length Serving Performance. As shown in Table 1, QSpar-Vegas consistently improves decode throughput over BF16 AR across all mixed-length workloads on both Qwen3-8B and Llama-3.1-8B. The speedup is relatively smaller for the short-context-heavy Trace-inspired workload and highest for the long-context-heavy Multi-turn-inspired workload, consistent with QSpar's greater benefits from reduced KV-cache traffic at longer contexts. These results demonstrate that QSpar's performance benefits extend beyond uniform-length batches to concurrent batches with heterogeneous context lengths.

Table 1. End-to-end decode speedup under mixed-length batched serving with a fixed aggregate KV-cache token budget of 128K.

\hline \mathrm{Model} & \mathrm{Workload} & \begin{array}{c} \mathrm{Request\\ Length\\ Ratio}\\\\ \mathrm{(64K:32K:16K:8K)} \end{array} & \begin{array}{c} \mathrm{QSpar\\ (Vegas,AWQ)}\\\\ \mathrm{Speedup} \end{array} \\\\ \hline \mathrm{Qwen3-8B} & \mathrm{Uniform} & 1:1:1:1 & 1.45\times \\\\ & \mathrm{Trace\\! -\\! inspired} & 1:2:4:24 & 1.40\times \\\\ & \mathrm{Multi\\! -\\! turn\\! -\\! inspired} & 8:4:2:1 & \mathbf{1.52\times} \\\\ \hline \mathrm{Llama-3.1-8B} & \mathrm{Uniform} & 1:1:1:1 & 1.37\times \\\\ & \mathrm{Trace\\! -\\! inspired} & 1:2:4:24 & 1.31\times \\\\ & \mathrm{Multi\\! -\\! turn\\! -\\! inspired} & 8:4:2:1 & \mathbf{1.53\times} \\\\ \hline \end{array}$$ [1] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST '25), pp. 155–170.

W2 & Q1. Exactness Guarantee and Token-level Output Equivalence

“The analysis provides no formal performance guarantee and no complete exactness statement.”

”The manuscript should distinguish lossless target output from acceptance-length recovery. The drafter and candidate tree remain approximate even when the original BF16 verifier produces the same greedy output. A token-by-token identity test against ordinary BF16 decoding would make this claim concrete.”

We conducted additional experiments comparing QSpar's generated outputs with BF16 autoregressive (AR) decoding to clarify the distinction between target-output exactness and acceptance-length recovery.

Exactness Guarantee. Although QSpar employs an approximate quantized sparse drafter and candidate tree, the final output is determined by the original full-attention BF16 verifier. During verification, a candidate token is accepted only if it matches the verifier's greedy prediction; otherwise, the verifier's predicted token is emitted. Therefore, if the full-attention BF16 verifier produces the same logits for a given prefix as standard BF16 autoregressive decoding, QSpar produces the same greedy output. The approximate drafter and candidate tree affect acceptance length and runtime, but not the output under this condition. This algorithmic guarantee does not imply bitwise identity across different batching or kernel execution paths.

Empirical Output Equivalence. To evaluate output equivalence under practical floating-point numerical variability [2], we used BF16 AR decoding with batch size 1 as the reference and compared its generated token sequences against BF16 AR with batch size 2 and three QSpar variants (Quest, StreamingLLM, and Vegas), each with batch size 1. We used the differences between BF16 AR batch sizes 1 and 2 as a baseline for execution-dependent numerical variability and evaluated sequence-level agreement with a generated token length of 128 on RULER-32K and lengths of 64, 128, and 256 on AIME25.

Table 2. Output sequence agreement with BF16 AR reference (batch size 1).

\hline \mathrm{Model} & \mathrm{Task} & \begin{array}{c} \mathrm{Generated}\\\\ \mathrm{Token\\ Length} \end{array} & \begin{array}{c} \mathrm{BF16\\ AR}\\\\ \mathrm{Batch\\ 2} \end{array} & \begin{array}{c} \mathrm{QSpar}\\\\ \mathrm{(Quest)} \end{array} & \begin{array}{c} \mathrm{QSpar}\\\\ \mathrm{(Stream)} \end{array} & \begin{array}{c} \mathrm{QSpar}\\\\ \mathrm{(Vegas)} \end{array} \\\\ \hline \mathrm{Qwen3-8B} & \mathrm{RULER\\ 32K} & 128 & 112/130 & 111/130 & 111/130 & 111/130 \\\\ & \mathrm{AIME} & 64 & 24/30 & 23/30 & 22/30 & 22/30 \\\\ & \mathrm{AIME} & 128 & 15/30 & 17/30 & 13/30 & 13/30 \\\\ & \mathrm{AIME} & 256 & 7/30 & 12/30 & 7/30 & 7/30 \\\\ \hline \mathrm{Llama-3.1-8B} & \mathrm{RULER\\ 32K} & 128 & 115/130 & 121/130 & 120/130 & 120/130 \\\\ & \mathrm{AIME} & 64 & 25/30 & 27/30 & 27/30 & 26/30 \\\\ & \mathrm{AIME} & 128 & 22/30 & 20/30 & 20/30 & 19/30 \\\\ & \mathrm{AIME} & 256 & 13/30 & 11/30 & 11/30 & 11/30 \\\\ \hline \end{array}$$ As shown in **Table 2**, QSpar exhibits output agreement rates broadly comparable to those observed when changing the BF16 AR batch size. Moreover, end-task accuracy remained unchanged across all evaluated settings. These results indicate that the observed output differences are consistent with the numerical variability of BF16 AR execution, supporting QSpar's exactness design, in which drafter approximations do not introduce additional target-output errors. We will incorporate the exactness clarification and additional output-equivalence results into the appendix. [2] https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Q2. Retrospective Selection of Parameters

“The policy parameters appear to be selected retrospectively. Figure 2 uses one model, one sparse-attention method, AWQ, and RULER to motivate K_rerank=8 and tau=2, while the deployed reducer uses delta_rerank rather than the verifier margin delta_ver shown in the analysis. The paper should provide a held-out calibration protocol, quantify the relationship between these two margins, report confidence intervals for top-K target recall, and show whether the same settings remain competitive on models and workloads not used for selection. The appendix already indicates that tau, M, K_reducer, and draft length interact substantially.”

To assess the robustness of QSpar's original policy parameters, we conducted additional experiments addressing three questions: (1) whether the original configuration remains competitive on disjoint calibration prompts and generalizes to held-out settings, (2) whether the reranker margin can reliably identify uncertain draft positions as an alternative to the verifier margin, and (3) whether a small reranking candidate set can reliably include the BF16 verifier's target token. We summarize the held-out calibration findings here and provide the detailed calibration results in Appendix A.6 (Tables 6–10).

Q2.1. Held-Out Calibration protocol

Original Rationale. Our original parameter choices were motivated by the empirical observations in Figure 2. Specifically, Figure 2(c) shows that the full-attention BF16 verifier's target token remains among the quantized sparse drafter's top-8 candidates at most flipped positions, motivating Krerank=8K_{\mathrm{rerank}}=8. Figure 2(a) shows that token flips are rare when the verifier margin is at least 2, motivating the initial threshold τ=2\tau=2. To limit verification overhead, we set Kreducer=2K_{\mathrm{reducer}}=2 and M=2M=2.

Calibration Setup. To assess whether our original parameter-selection approach achieves competitive throughput and can serve as a practical calibration protocol, we constructed a calibration set of 26 RULER-32K prompts disjoint from the final evaluation prompts (Qwen3-8B, Vegas, AWQ). We measured end-to-end decode throughput across all combinations of τ∈1,2,3,4\tau\in{1,2,3,4}, M∈1,2,3M\in{1,2,3}, Kreducer∈2,3K_{\mathrm{reducer}}\in{2,3}, and Ndraft∈6,10,14N_{\mathrm{draft}}\in{6,10,14}, yielding 72 configurations. We fixed Krerank=8K_{\mathrm{rerank}}=8 in this sweep and separately validated this choice through additional ablation test (Appendix A.6, Table 8).

Parameter Sensitivity. The original configuration achieves 106.5 tokens/s, corresponding to 99.4% of the highest measured throughput (107.1 tokens/s) among the 72 configurations (Appendix A.6, Table 6). We further analyzed which parameters contribute most to the throughput differences observed across the 72 configurations. Specifically, we decomposed throughput variance into main effects, which capture the individual contribution of each parameter, and interaction effects, which capture how the combined changes of multiple parameters influence throughput. We then measured each factor's contribution to the total variance. As shown in Appendix A.6 Table 7, draft length is the dominant factor, explaining 77.2% of the throughput variance. In contrast, the effects of τ\tau, M, and KreducerK_{\mathrm{reducer}}, as well as major two-way interactions, are smaller than the measured run-to-run standard deviation (1.46 tokens/s). These findings indicate that draft length is the dominant source of throughput variation in the evaluated calibration setting, while the original candidate-selection parameters remain relatively robust within the tested ranges. This motivates calibrating only draft length while retaining the original candidate-selection parameters. Since QSpar combines quantization and sparse attention to construct its drafter, the optimal draft length may vary across different sparse-attention methods.

Calibration Protocol. Accordingly, we retained the original candidate-selection parameters (τ,M,Kreducer,Krerank)=(2,2,2,8)(\tau,M,K_{\mathrm{reducer}},K_{\mathrm{rerank}})=(2,2,2,8) and calibrated only draft length for each sparse-attention method. With (M=2) fixed, draft length 10 achieved the highest throughput for Vegas, while draft length 6 performed best for StreamingLLM and Quest (Appendix A.6, Table 9).

Generalization. To evaluate generalization, we applied the calibrated configurations to 15 held-out settings spanning different models (Qwen3-4B/8B and Llama-3.1-8B), workloads (RULER-32K and the first 4K reasoning tokens of AIME25), and sparse-attention methods (Vegas, StreamingLLM, and Quest). We compared the calibrated configurations against the highest measured throughput among the evaluated configurations in each setting. As shown in Appendix A.6 (Table 10), the calibrated configurations achieve 97.4–100% of the best measured throughput across all 15 settings. These results demonstrate that the policy remains competitive across different evaluation conditions.

Reviewer_UtNBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the accuracy degeneration problem when combining weight quantization to sparse attention in drafter models. The authors propose to use a reranking mechanism that use the BF16 LM head as a reranker, that recovers the top-1 token. Experiments show that QSpar outperformed two baselines on two benchmarks and two models.

Critical strengths

S1. This paper is clearly motivated. Both motivation and intuition are clearly demonstrated with empirical findings and theoretical hints. The flow of the paper is good.

Critical weaknesses

W1. A naive baseline would be to use the quantized LLM body with a full BF16 LM head. Please include and add this baseline as an ablation study. This is important because the idea of including >1 tokens in the drafter model has been studied in the literature [1], making the reranker idea a single core contribution.

W2. Qwen3 thinking model card explicitly mentioned that do NOT use greedy decode because it can lead to performance degradation and endless repetitions. So please repeat the end task accuracy as well. It would also be great if the authors can include a third model like ministral.

[1] Huo, M., Zhang, J., Wang, H., Xu, J., Chen, Z., Tai, H. and Chen, Y., 2025. Spec-llava: Accelerating vision-language models with dynamic tree-based speculative decoding. arXiv preprint arXiv:2509.11961.

Questions for authors

Suggestion 1: I suggest the authors to revise the paragraph in line 086 to 093. The findings should be made explicit and precise, with presenting why they matter.

5 comments

Thank you for taking the time to provide a detailed and thoughtful review. Your feedback has been very helpful in improving the clarity and rigor of our paper by strengthening the experimental validation of QSpar's design choices and decoding reliability. We address each point in detail below.

W1. Ablation Study with a Full BF16 LM Head

“A naive baseline would be to use the quantized LLM body with a full BF16 LM head. Please include and add this baseline as an ablation study. This is important because the idea of including >1 tokens in the drafter model has been studied in the literature, making the reranker idea a single core contribution.”

To evaluate whether a full BF16 LM head can serve as a simpler alternative to QSpar's Reranker, we conducted an additional ablation study on AIME25.

Experimental Setup. We evaluated Qwen3-8B and Llama-3.1-8B using the same quantized sparse Transformer body while varying the LM-head configuration: (1) full BF16, (2) INT4-quantized, and (3) INT4 with selective BF16 reranking. We measured acceptance length, LM-head latency, and end-to-end decode throughput for each configuration.

Acceptance Recovery vs. LM-head Overhead. As shown in Table 1, the full BF16 LM head achieves a higher acceptance length than the INT4 LM head. However, its higher LM-head latency results in lower end-to-end throughput despite the acceptance gain. In contrast, INT4 + Reranker achieves nearly the same acceptance length as the full BF16 LM head while reducing LM-head latency by 2.76× on Qwen3-8B and 2.64× on Llama-3.1-8B. Consequently, end-to-end decode throughput improves by 6.1% and 4.1%, respectively, compared with the full BF16 LM head. These results show that, in the evaluated settings, selective BF16 reranking recovers acceptance comparable to a full BF16 LM head with lower LM-head latency and higher end-to-end throughput.

Table 1. Ablation of LM-head precision and selective BF16 reranking on AIME25. LM-head latency is reported in milliseconds per speculative round, while Head Time (%) denotes its fraction of total decoding time.

\hline \mathrm{Model} & \mathrm{LM\\ Head} & \begin{array}{c}\mathrm{Acceptance}\\\\\mathrm{Length}\end{array} & \begin{array}{c}\mathrm{Head\\ Latency}\\\\\mathrm{(ms)}\end{array} & \begin{array}{c}\mathrm{Head\\ Time}\\\\\mathrm{(\\%)}\end{array} & \begin{array}{c}\mathrm{Throughput}\\\\\mathrm{(tok/s)}\end{array} \\\\ \hline \mathrm{Qwen3-8B} & \mathrm{Full\\ BF16} & \mathbf{8.323} & 6.93 & 7.6\\% & 365.7 \\\\ & \mathrm{INT4} & 8.040 & 2.06 & 2.4\\% & 375.6\\ (\mathbf{+2.7\\%}) \\\\ & \mathrm{INT4+Reranker} & \mathbf{8.323} & 2.51 & 2.9\\% & 387.8\\ (\mathbf{+6.1\\%}) \\\\ \hline \mathrm{Llama-3.1-8B} & \mathrm{Full\\ BF16} & \mathbf{9.964} & 5.88 & 6.9\\% & 470.5 \\\\ & \mathrm{INT4} & 9.628 & 1.79 & 2.2\\% & 479.5\\ (\mathbf{+1.9\\%}) \\\\ &\mathrm{INT4+Reranker} & \mathbf{9.962} & 2.23 & 2.7\\% & 489.7\\ (\mathbf{+4.1\\%}) \\\\ \hline \end{array}$$ We also acknowledge prior multi-candidate drafting methods, including Spec-LLaVA[1]. QSpar differs by using additional candidates specifically to mitigate quantization-induced errors. In particular, the Reducer retains alternative token candidates at a small number of low-margin draft positions to mitigate acceptance loss from residual hidden-state errors while keeping verification overhead bounded. [1] Huo, M., Zhang, J., Wang, H., Xu, J., Chen, Z., Tai, H. and Chen, Y., 2025. Spec-llava: Accelerating vision-language models with dynamic tree-based speculative decoding. arXiv preprint arXiv:2509.11961.

W2. End-task Accuracy under Greedy Decoding and Extend to Ministral

“Qwen3 thinking model card explicitly mentioned that do NOT use greedy decode because it can lead to performance degradation and endless repetitions. So please repeat the end task accuracy as well. It would also be great if the authors can include a third model like ministral.”

To address the concern about repetitive generation under greedy decoding, we evaluated end-task accuracy and repetition frequency on AIME25 and conducted an additional ablation study excluding repetitive requests. We also extended our evaluation to a new model family, Ministral-3-8B.

Greedy Decoding Accuracy. We evaluated dense, unquantized Qwen3 models (4B, 8B, 14B, and 32B) on AIME25, a long-generation reasoning task, under greedy decoding with a maximum generation length of 32K tokens. As shown in Table 2, Qwen3-4B and Qwen3-8B reached the maximum generation length due to repeated token sequences in 4 and 1 out of 30 AIME25 requests, respectively. In contrast, no such cases were observed for Qwen3-14B or Qwen3-32B.

Table 2. AIME25 accuracy and repetition under greedy and model-card-recommended decoding.

\hline \mathrm{Model} & \mathrm{Greedy} & \mathrm{Model\\ Card} \\\\ \begin{array}{l} \phantom{\mathrm{Repetition}} \\\\ \hline \mathrm{Qwen3-4B} \\\\ \mathrm{Qwen3-8B} \\\\ \mathrm{Qwen3-14B} \\\\ \mathrm{Qwen3-32B} \end{array} & \begin{array}{cc} \mathrm{Repetition} & \mathrm{Accuracy} \\\\ \hline 4/30 & 63.3\\% \\\\ 1/30 & 66.7\\% \\\\ 0/30 & 70.4\\% \\\\ 0/30 & 72.9\\% \end{array} & \begin{array}{cc} \mathrm{Repetition} & \mathrm{Accuracy} \\\\ \hline 0/30 & 65.6\\% \\\\ 0/30 & 67.3\\% \\\\ 0/30 & 70.4\\% \\\\ 0/30 & 72.9\\% \end{array} \\\\ \hline \end{array}$$ **Robustness to Repetitive Generation.** To isolate the impact of repetitive generation on QSpar's performance, we remeasured acceptance length and end-to-end throughput relative to BF16 AR after excluding repetitive requests (**Table 3**). Even after excluding these requests, QSpar maintained high acceptance lengths and achieved speedups consistent with the performance trends reported in Figure 5. These results indicate that QSpar's acceleration benefits do not depend on repetitive generation. **Extension to Ministral.** We further evaluated QSpar on Ministral-3-8B-Reasoning-2512 [2], a model from a different family that exhibited no repetitive generation under greedy decoding and achieved 78.7% accuracy on AIME25 (**Table 3**). With a draft length of 10, QSpar achieved acceptance lengths of 8.18, 6.48, and 8.29 on Ministral-3-8B with Vegas, StreamingLLM, and Quest, respectively, yielding speedups of 1.64×, 1.47×, and 1.07× over BF16 AR decoding. These results demonstrate that QSpar generalizes beyond the Qwen3 and Llama-3.1 model families, maintaining high acceptance lengths and improving end-to-end decode throughput even on workloads requiring long output generation. **Table 3. Acceptance length and end-to-end decode speedup on AIME25.** $$\begin{array}{llccc} \hline \mathrm{Model} & \mathrm{Method} & \mathrm{Acceptance\\ Length} & \mathrm{Speedup} & \mathrm{Fig.\\ 5\\ Speedup} \\\\ \hline \mathrm{Qwen3-4B} & \mathrm{Vegas} & 8.46 & 1.42\times & 1.44\times \\\\ & \mathrm{StreamingLLM} & 6.98 & 1.07\times & 1.09\times \\\\ & \mathrm{Quest} & 8.36 & 0.97\times & 0.92\times \\\\ \hline \mathrm{Qwen3-8B} & \mathrm{Vegas} & 8.65 & 1.84\times & 1.85\times \\\\ & \mathrm{StreamingLLM} & 7.33 & 1.82\times & 1.79\times \\\\ & \mathrm{Quest} & 8.59 & 1.14\times & 1.18\times \\\\ \hline \mathrm{Ministral-3-8B} & \mathrm{Vegas} & 8.18 & 1.64\times & - \\\\ & \mathrm{StreamingLLM} & 6.48 & 1.47\times & - \\\\ & \mathrm{Quest} & 8.29 & 1.07\times & - \\\\ \hline \end{array}$$ We will incorporate these additional end-task accuracy analyses, repetition-related ablations, and Ministral evaluation results into the Evaluation section and appendix. [2] https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512/tree/81eaece1948f3875421d9a45bc55487d10e2d894

S1. Clarify Findings and Emphasize Significance

“I suggest the authors to revise the paragraph in line 086 to 093. The findings should be made explicit and precise, with presenting why they matter.”

We revised Lines 086–093 to clarify the importance of addressing acceptance loss and two key findings from our experiments. These findings provide a clearer motivation for the designs of QSpar's Reranker and Reducer. The revisions have been incorporated into the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.