- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
AB-Sparse chooses a different attention block size for each KV head. It combines this with INT4 block summaries and GPU kernels to improve decoding accuracy and speed while keeping the full KV cache.
Critical strengths
The paper observes different heads need different block sizes which interesting and tries to develop a method based on that. Table 1 shows meaningful gains over Quest and ArkVale: 2.17-5.43 score points on RULER and 1.52-2.62 on LongBench, depending on the model and baseline, although statistical significance is not established.
Critical weaknesses
1-There is no integration with serving engines such as vLLM or SGLang; although these systems describe custom Python/CUDA code and their compatibility with paged attention, they do not actually show how their approach can be deployed within vLLM or SGLang. The main reason given in the abstract for the paper stems from the most deployable form of sparse attention. If deployment is a concern, it is necessary for them to demonstrate that their method improves the serving engines. However, making block sparsity fine-grained—such as adapting it on a per-head basis—is not easy to implement efficiently in such engines and does not yield clear advantages. Without this, it is difficult to see the benefits of the proposed approach.
2-The speed benefit is in part due to an optimisation which the baselines can also make use of. Appendix A.3 contains controls for uniform blocking and quantisation. It demonstrates that INT4 leads to a 14-18% reduction in latency across all the configurations; AB-Sparse drops from 5.38 to 4.53 ms. Figure 17 also indicates that uniform B=32 is still faster when the precision is the same. Therefore, before asserting that adaptation is effectively free, the main comparison should include accuracy-latency curves with equally quantised baselines.
3-Both the related work and the experiments fail to take into account relevant published decoding methods such as BLASST (MLSys 2026), which omits attention blocks; RocketKV and ShadowKV (ICML 2025), which carry out dynamic decoding selection; and SeerAttention-R (ICLR 2026), in particular for the reasoning experiments. When making the comparisons it is necessary to consider ShadowKV's offloading and SeerAttention-R's training cost. The true contribution seems to lie in the per-head block size rather than in head adaptation itself.
4-Some setup details are missing. Section 4.1 uses Qwen3’s 128K context native, but both models specify 32K native context, extended to 128K. Report the actual scaling, prompting, and thinking-mode settings. The method needs to be more clear on details of how combining query-head scores within GQA and updating quantization scales as the cache grows.
5-The reported 1.59X throughput improvement over Quest is for Llama-3.1-8B at batch size 4. Figure 9 measures attention latency, not complete request latency. Page mapping is a sensible compatibility design, but continuous batching, prefix caching, and tensor parallelism are not demonstrated. INT4 compresses the summaries quadratic; it does not compress the retained KV cache quadratic. Peak GPU memory is also not measured.
Questions for authors
Please check the weakness section.
9 comments
Response to Reviewer aaoM (Part 2)
4. Setup details
Positional scaling.
We clarify that the Qwen3 model cards provide a YaRN configuration for extending the context length from 32K to 128K [1, 2]. For our long-context evaluations, we follow this configuration: rope_type="yarn", factor=4.0, and original_max_position_embeddings=32768.
Prompt and thinking-mode.
- RULER: We use the benchmark’s meta-chat template with task-specific answer prefixes [3], without explicitly enabling Qwen3’s thinking mode.
- LongBench: We use the benchmark’s task-specific prompts without an additional model chat template [4], without explicitly enabling Qwen3’s thinking mode.
- Reasoning benchmarks: We use Qwen3’s chat template with thinking explicitly enabled (
enable_thinking=True), following its official recommendations [1, 2], and instruct the model to reason step by step and place the final answer in\boxed{…}.
GQA score combination.
Each query head independently scores and selects blocks from its associated shared KV cache. Selection scores are not aggregated across query heads.
Quantization scales.
Quantization scales remain fixed during generation and are not updated as the KV cache grows.
[1] Qwen3-8B model card: https://huggingface.co/Qwen/Qwen3-8B [2] Qwen3-32B model card: https://huggingface.co/Qwen/Qwen3-32B [3] NVIDIA RULER, prompt templates: https://github.com/NVIDIA/RULER/blob/main/scripts/data/template.py [4] LongBench, task-specific prompts: https://github.com/THUDM/LongBench/blob/main/LongBench/config/dataset2prompt.json
5. Research scope
AB-Sparse targets attention computation during decoding. Accordingly, the original submission reports attention latency, including metadata/centroid updates and sparse selection. To assess whether these gains translate into overall decoding benefits, we have added end-to-end decoding latency results in Figure 19, Appendix A.5 of the revised manuscript. We evaluate Llama-3.1-8B, Qwen3-8B, and Qwen3-32B on an NVIDIA A100 GPU with a batch size of 1 at context lengths of 64K, 128K, and 256K. Table 3 summarizes the corresponding speedups over full attention and Quest. Overall, AB-Sparse achieves 1.06–1.47× end-to-end decoding speedups over full attention and 1.02–1.13× over Quest. Although smaller than the attention-level gains, these results demonstrate lower decoding latency across all evaluated configurations, with larger relative gains at longer contexts.
| Model | 64K | 128K | 256K |
|---|---|---|---|
| Llama-3.1-8B | 1.12× / 1.02× | 1.28× / 1.07× | 1.47× / 1.13× |
| Qwen3-8B | 1.12× / 1.02× | 1.26× / 1.07× | 1.44× / 1.12× |
| Qwen3-32B | 1.06× / 1.02× | 1.10× / 1.03× | 1.19× / 1.07× |
Table 3: End-to-end decoding speedups of AB-Sparse. Each cell reports speedups over full attention / Quest, respectively.
Continuous batching dynamically schedules requests across decoding iterations; prefix caching reuses KV states for shared prefixes to avoid repeated prefill computation; and tensor parallelism in attention layer distributes attention head across devices. These system optimizations are orthogonal with AB-Sparse.
AB-Sparse targets efficient KV cache selection and access rather than KV cache compression. The 4% budget controls the KV cache size accessed during attention, while the full KV cache remains stored. INT4 quantization applies to centroid summaries, not the KV cache itself.
We report peak allocated GPU memory during decoding of 17.00/23.08 GiB for Llama-3.1-8B and 17.58/24.40 GiB for Qwen3-8B at 16K/64K, respectively, with batch size 1 and BF16 weights/KV cache.
Follow-up
Dear Reviewer,
Thank you again for your constructive feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional experiments.
Specifically, we have added SGLang integration results, matched-precision accuracy–latency comparisons, and end-to-end decoding latency and memory measurements. We have also clarified how our head-adaptive selection granularity is orthogonal to the mechanisms emphasized in the methods you mentioned, and provided additional experimental and implementation details.
We hope these additions address your concerns and would welcome any remaining questions. If you find that they resolve the issues raised in your review, we would appreciate your reconsideration of your assessment.
Thank you for your time and consideration.
Best regards, The Authors
Response to Reviewer aaoM (Part 1)
We thank the reviewer for the constructive suggestions. We address each concern as follows.
1. Compatibility and integration with SGLang
Serving engine compatibility.
We explain how AB-Sparse integrates with existing serving engines as follows:
In summary, AB-Sparse’s logical–physical decoupling preserves uniform physical paging while enabling head-adaptive selection, making it compatible with the paged KV-cache architecture of modern serving engines.
SGLang integration results.
To validate this design, we integrate AB-Sparse into SGLang by extending an existing Quest integration PR with adaptive block sizes and centroid quantization. We evaluate Llama-3.1-8B on an NVIDIA H100 GPU, with batch size 1. Table 1 reports the end to end decoding latency of AB-Sparse and baselines. AB-Sparse achieves 1.33–1.64× decoding speedups over Full attention and 1.04–1.08× over QUEST, demonstrating its practical decoding benefits within SGLang.
Table 1. End-to-end decoding latency (ms) with SGLang integration.
2. Matched-precision accuracy–latency comparison
We clarify that centroid quantization is itself a contribution of AB-Sparse: we identify and exploit the robustness of centroid-based selection to low-precision representations and develop an efficient quantized implementation.
Appendix A.3 of the original submission already reports attention latency for uniform B=16, uniform B=32, and AB-Sparse with INT4 centroids on Llama-3.1-8B at 64K. We now supplement these measurements with INT4 accuracy results under the same 4% retrieved-token budget, reporting accuracy and latency together in Table 2. AB-Sparse reduces latency relative to B=16 while retaining similar accuracy, and improves accuracy over B=32 at a modest latency cost.
Table 2. INT4 accuracy and attention latency.
Furthermore, Figure 18 in Appendix A.4 of the revised manuscript jointly reports accuracy and latency across additional models and longer contexts, using BF16 centroids and the same 4% retrieved-token budget. Across these settings, AB-Sparse extends the empirical accuracy–latency Pareto frontier of the evaluated uniform baselines, demonstrating the benefit of adaptive granularity independently of centroid quantization.
3. Related work clarification
AB-Sparse addresses an optimization dimension orthogonal to those emphasized in these methods: head-adaptive selection granularity.
BLASST focuses on block-skipping decisions; RocketKV combines KV eviction with query-dependent sparse selection; ShadowKV combines low-rank representations, KV offloading, and sparse selection; and SeerAttention-R learns block-selection policies through trained gating. These methods optimize sparse attention through different mechanisms, but do not optimize the granularity at which individual heads perform selection.
AB-Sparse makes this granularity an explicit, head-adaptive optimization variable. This is not merely a matter of assigning different block sizes to different heads: it requires balancing selection resolution against estimation overhead and efficiently executing selection with heterogeneous per-head granularities. Our contributions address both the algorithmic and systems challenges through head-adaptive block-size allocation, centroid quantization, and efficient kernels supporting heterogeneous granularities. Together, these contributions establish selection granularity as an actionable optimization dimension.
We will expand the related-work discussion to make this distinction and the associated technical contributions explicit in the revised manuscript.