acceptodds
Under review as a conference paper at ICLR 2027

AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference

Abstract

Block sparse attention makes long-context decoding affordable by partitioning the KV cache into fixed-size blocks and loading only the blocks a query needs, and it has become the most deployable form of sparse attention because it leaves paged KV cache management untouched. Its weakness is accuracy: at a 4% KV budget, state-of-the-art methods trail full attention by up to 10 points on standard long context benchmarks. We trace a large part of this gap to a single design choice, the use of one block size for every attention head. Measuring attention recall head by head, we find that heads differ widely in their sensitivity to block granularity: some keep near-perfect recall, others lose over 90% of it when blocks grow from 16 to 32 tokens, and no single size serves both. We present AB-Sparse, a training-free algorithm–system co-design that gives each head its own block size. A lightweight one-time calibration assigns block sizes that transfer across domains; INT4 per-channel asymmetric quantization of block centroids, which only rank blocks and never enter the attention output, keeps centroid memory below that of the uniform baseline; and custom GPU kernels batch heads with heterogeneous block counts without padding and map variable-size blocks onto standard paged attention. On Llama-3.1-8B, Qwen3-8B, and Qwen3-32B, AB-Sparse raises the accuracy of Quest and ArkVale by 3.0–3.7 points on RULER and 1.9–2.6 points on LongBench on average (up to 5.4 points), recovering 27–85% of the gap to full attention, with comparable or higher decoding throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, long context inference, attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aaoMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AB-Sparse chooses a different attention block size for each KV head. It combines this with INT4 block summaries and GPU kernels to improve decoding accuracy and speed while keeping the full KV cache.

Critical strengths

The paper observes different heads need different block sizes which interesting and tries to develop a method based on that. Table 1 shows meaningful gains over Quest and ArkVale: 2.17-5.43 score points on RULER and 1.52-2.62 on LongBench, depending on the model and baseline, although statistical significance is not established.

Critical weaknesses

1-There is no integration with serving engines such as vLLM or SGLang; although these systems describe custom Python/CUDA code and their compatibility with paged attention, they do not actually show how their approach can be deployed within vLLM or SGLang. The main reason given in the abstract for the paper stems from the most deployable form of sparse attention. If deployment is a concern, it is necessary for them to demonstrate that their method improves the serving engines. However, making block sparsity fine-grained—such as adapting it on a per-head basis—is not easy to implement efficiently in such engines and does not yield clear advantages. Without this, it is difficult to see the benefits of the proposed approach.

2-The speed benefit is in part due to an optimisation which the baselines can also make use of. Appendix A.3 contains controls for uniform blocking and quantisation. It demonstrates that INT4 leads to a 14-18% reduction in latency across all the configurations; AB-Sparse drops from 5.38 to 4.53 ms. Figure 17 also indicates that uniform B=32 is still faster when the precision is the same. Therefore, before asserting that adaptation is effectively free, the main comparison should include accuracy-latency curves with equally quantised baselines.

3-Both the related work and the experiments fail to take into account relevant published decoding methods such as BLASST (MLSys 2026), which omits attention blocks; RocketKV and ShadowKV (ICML 2025), which carry out dynamic decoding selection; and SeerAttention-R (ICLR 2026), in particular for the reasoning experiments. When making the comparisons it is necessary to consider ShadowKV's offloading and SeerAttention-R's training cost. The true contribution seems to lie in the per-head block size rather than in head adaptation itself.

4-Some setup details are missing. Section 4.1 uses Qwen3’s 128K context native, but both models specify 32K native context, extended to 128K. Report the actual scaling, prompting, and thinking-mode settings. The method needs to be more clear on details of how combining query-head scores within GQA and updating quantization scales as the cache grows.

5-The reported 1.59X throughput improvement over Quest is for Llama-3.1-8B at batch size 4. Figure 9 measures attention latency, not complete request latency. Page mapping is a sensible compatibility design, but continuous batching, prefix caching, and tensor parallelism are not demonstrated. INT4 compresses the summaries ​quadratic; it does not compress the retained KV cache ​quadratic. Peak GPU memory is also not measured.

Questions for authors

Please check the weakness section.

9 comments
AuthorsOpenReview ↗

Response to Reviewer aaoM (Part 1)

We thank the reviewer for the constructive suggestions. We address each concern as follows.

1. Compatibility and integration with SGLang

Serving engine compatibility.

We explain how AB-Sparse integrates with existing serving engines as follows:

  • Modern serving engines (vLLM/SGLang) organize KV caches into physical pages. Block-sparse attention performs centroid-based estimation and top-k selection, then supplies the selected physical-page indices to the attention backend. Centroid maintenance and block selection are additional operations layered on top of the existing paged KV cache, without changing its underlying physical layout.
  • AB-Sparse further introduces head-adaptive logical block sizes. Importantly, head-adaptive selection does not require heterogeneous physical pages. AB-Sparse decouples logical selection granularity from physical cache storage: each head performs estimation and selection at its assigned logical block size, and the selected logical blocks are mapped to per-head physical-page indices for paged attention. Different logical block sizes therefore coexist within a uniform physical paging structure.

In summary, AB-Sparse’s logical–physical decoupling preserves uniform physical paging while enabling head-adaptive selection, making it compatible with the paged KV-cache architecture of modern serving engines.

SGLang integration results.

To validate this design, we integrate AB-Sparse into SGLang by extending an existing Quest integration PR with adaptive block sizes and centroid quantization. We evaluate Llama-3.1-8B on an NVIDIA H100 GPU, with batch size 1. Table 1 reports the end to end decoding latency of AB-Sparse and baselines. AB-Sparse achieves 1.33–1.64× decoding speedups over Full attention and 1.04–1.08× over QUEST, demonstrating its practical decoding benefits within SGLang.

Context lengthFull attentionQUEST (B=32)AB-Sparse
64K7.525.885.67
128K9.346.526.17
256K*13.458.868.22

Table 1. End-to-end decoding latency (ms) with SGLang integration.

2. Matched-precision accuracy–latency comparison

We clarify that centroid quantization is itself a contribution of AB-Sparse: we identify and exploit the robustness of centroid-based selection to low-precision representations and develop an efficient quantized implementation.

Appendix A.3 of the original submission already reports attention latency for uniform B=16, uniform B=32, and AB-Sparse with INT4 centroids on Llama-3.1-8B at 64K. We now supplement these measurements with INT4 accuracy results under the same 4% retrieved-token budget, reporting accuracy and latency together in Table 2. AB-Sparse reduces latency relative to B=16 while retaining similar accuracy, and improves accuracy over B=32 at a modest latency cost.

MethodCentroid precisionRULER accuracy (%)Attention latency (ms)
Uniform B=16INT481.695.33
Uniform B=32INT476.934.25
AdaptiveINT481.494.53

Table 2. INT4 accuracy and attention latency.

Furthermore, Figure 18 in Appendix A.4 of the revised manuscript jointly reports accuracy and latency across additional models and longer contexts, using BF16 centroids and the same 4% retrieved-token budget. Across these settings, AB-Sparse extends the empirical accuracy–latency Pareto frontier of the evaluated uniform baselines, demonstrating the benefit of adaptive granularity independently of centroid quantization.

3. Related work clarification

AB-Sparse addresses an optimization dimension orthogonal to those emphasized in these methods: head-adaptive selection granularity.

BLASST focuses on block-skipping decisions; RocketKV combines KV eviction with query-dependent sparse selection; ShadowKV combines low-rank representations, KV offloading, and sparse selection; and SeerAttention-R learns block-selection policies through trained gating. These methods optimize sparse attention through different mechanisms, but do not optimize the granularity at which individual heads perform selection.

AB-Sparse makes this granularity an explicit, head-adaptive optimization variable. This is not merely a matter of assigning different block sizes to different heads: it requires balancing selection resolution against estimation overhead and efficiently executing selection with heterogeneous per-head granularities. Our contributions address both the algorithmic and systems challenges through head-adaptive block-size allocation, centroid quantization, and efficient kernels supporting heterogeneous granularities. Together, these contributions establish selection granularity as an actionable optimization dimension.

We will expand the related-work discussion to make this distinction and the associated technical contributions explicit in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer aaoM (Part 2)

4. Setup details

Positional scaling.

We clarify that the Qwen3 model cards provide a YaRN configuration for extending the context length from 32K to 128K [1, 2]. For our long-context evaluations, we follow this configuration: rope_type="yarn", factor=4.0, and original_max_position_embeddings=32768.

Prompt and thinking-mode.

  • RULER: We use the benchmark’s meta-chat template with task-specific answer prefixes [3], without explicitly enabling Qwen3’s thinking mode.
  • LongBench: We use the benchmark’s task-specific prompts without an additional model chat template [4], without explicitly enabling Qwen3’s thinking mode.
  • Reasoning benchmarks: We use Qwen3’s chat template with thinking explicitly enabled (enable_thinking=True), following its official recommendations [1, 2], and instruct the model to reason step by step and place the final answer in \boxed{…}.

GQA score combination.

Each query head independently scores and selects blocks from its associated shared KV cache. Selection scores are not aggregated across query heads.

Quantization scales.

Quantization scales remain fixed during generation and are not updated as the KV cache grows.

[1] Qwen3-8B model card: https://huggingface.co/Qwen/Qwen3-8B [2] Qwen3-32B model card: https://huggingface.co/Qwen/Qwen3-32B [3] NVIDIA RULER, prompt templates: https://github.com/NVIDIA/RULER/blob/main/scripts/data/template.py [4] LongBench, task-specific prompts: https://github.com/THUDM/LongBench/blob/main/LongBench/config/dataset2prompt.json

5. Research scope

AB-Sparse targets attention computation during decoding. Accordingly, the original submission reports attention latency, including metadata/centroid updates and sparse selection. To assess whether these gains translate into overall decoding benefits, we have added end-to-end decoding latency results in Figure 19, Appendix A.5 of the revised manuscript. We evaluate Llama-3.1-8B, Qwen3-8B, and Qwen3-32B on an NVIDIA A100 GPU with a batch size of 1 at context lengths of 64K, 128K, and 256K. Table 3 summarizes the corresponding speedups over full attention and Quest. Overall, AB-Sparse achieves 1.06–1.47× end-to-end decoding speedups over full attention and 1.02–1.13× over Quest. Although smaller than the attention-level gains, these results demonstrate lower decoding latency across all evaluated configurations, with larger relative gains at longer contexts.

Model64K128K256K
Llama-3.1-8B1.12× / 1.02×1.28× / 1.07×1.47× / 1.13×
Qwen3-8B1.12× / 1.02×1.26× / 1.07×1.44× / 1.12×
Qwen3-32B1.06× / 1.02×1.10× / 1.03×1.19× / 1.07×

Table 3: End-to-end decoding speedups of AB-Sparse. Each cell reports speedups over full attention / Quest, respectively.

Continuous batching dynamically schedules requests across decoding iterations; prefix caching reuses KV states for shared prefixes to avoid repeated prefill computation; and tensor parallelism in attention layer distributes attention head across devices. These system optimizations are orthogonal with AB-Sparse.

AB-Sparse targets efficient KV cache selection and access rather than KV cache compression. The 4% budget controls the KV cache size accessed during attention, while the full KV cache remains stored. INT4 quantization applies to centroid summaries, not the KV cache itself.

We report peak allocated GPU memory during decoding of 17.00/23.08 GiB for Llama-3.1-8B and 17.58/24.40 GiB for Qwen3-8B at 16K/64K, respectively, with batch size 1 and BF16 weights/KV cache.

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional experiments.

Specifically, we have added SGLang integration results, matched-precision accuracy–latency comparisons, and end-to-end decoding latency and memory measurements. We have also clarified how our head-adaptive selection granularity is orthogonal to the mechanisms emphasized in the methods you mentioned, and provided additional experimental and implementation details.

We hope these additions address your concerns and would welcome any remaining questions. If you find that they resolve the issues raised in your review, we would appreciate your reconsideration of your assessment.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_Xk5FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AB-Sparse improves long-context decoding by assigning different block sizes to different KV heads. It combines offline block-size calibration, INT4 quantization of block summaries, and GPU kernels that support heterogeneous blocks while preserving the paged KV layout.

The method is training-free but requires calibration. Using 128 Wikipedia samples, it selects the largest block size from {16, 32, 64} that retains at least 95% of the attention recall achieved with block size 16. No model weights are updated.

Experiments on three models show consistent benchmark-average improvements over Quest and ArkVale on RULER and original LongBench at a 4% token-access budget. For example, Llama-3.1-8B RULER improves from 78.23 to 81.74 over Quest, compared with 88.71 for full attention. The paper also reports lower attention latency and improved mathematics performance over Quest.

Critical strengths
  1. Consistent downstream accuracy gains Adaptive block sizing improves both Quest and ArkVale across all three models’ benchmark averages. Results with two summary representations support the method’s broader applicability.

  2. Practical implementation Quantized summaries, batched selection, and page mapping address the overhead introduced by heterogeneous blocks. The quantization and kernel ablations support these design choices.

Critical weaknesses
  1. The matched-resource advantage remains unclear

Main comparisons change both block allocation and summary precision. Appendix A.3 includes useful controls, but they suggest an intermediate accuracy–latency tradeoff rather than an unconditional improvement: uniform B=16 achieves slightly higher accuracy, while uniform B=32 with INT4 summaries is faster in the reported example. Broader matched-precision comparisons are needed.

Resource claims also need clearer boundaries. Figure 9 measures attention latency, and Figure 10’s throughput scope is insufficiently specified. The 4% budget concerns KV positions accessed for attention; it does not demonstrate storing only 4% of the full KV cache.

  1. The evaluation does not establish a contemporary efficiency–accuracy frontier

Quest and ArkVale are appropriate direct baselines, but LServe is a particularly relevant missing comparison because it also addresses the block-size tradeoff through logical and physical paging.

RULER and original LongBench provide meaningful evidence, but harder targeted tests would strengthen the accuracy-preservation claim. Suitable examples include supported-length subsets of LongBench v2 or LongBench Pro, plus NoLiMa for indirect retrieval. The mathematics results lack full-attention scores and actual output-length statistics, so they establish improvement over Quest rather than preservation of dense-model reasoning accuracy

Questions for authors
  1. What does Figure 10’s throughput include? Please specify whether it measures complete-model decoding and includes summary updates and selection. What are the GPU count, precisions, and peak GPU memory usage?

  2. How were calibration samples, lengths, and query positions selected? How is recall aggregated across samples and grouped query heads? What are calibration costs, and how are quantization scales updated during generation?

  3. Which Qwen checkpoint revisions and positional-scaling settings were used? The original model cards specify 32K native context and 128K with YaRN. Are the 256K measurements attention microbenchmarks or complete-model runs? Qwen3 model card

  4. Does adaptive sizing improve the accuracy–latency frontier when all uniform and adaptive configurations use INT4 summaries? How does it compare with LServe under matched conditions?

  5. For Table 2, please provide full-attention pass@1/pass@4, uncertainty, and actual output lengths. Does the benefit persist on a harder long-context task within the models’ supported windows?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xk5F

We thank the reviewer for the constructive suggestions. We address each concern as follows.

1. Efficiency clarification

Since AB-Sparse specifically optimizes attention computation, Figures 9 and 10 report attention-only latency and throughput. Measurement starts after the Q/K/V projections and ends at the attention output, before the output projection. Metadata/centroid updates and sparse selection are included.

The experiments use a single GPU, with BF16 model weights and KV cache; AB-Sparse uses INT4 quantization for centroids. AB-Sparse’s peak GPU memory usage during decoding is 17.00/23.08 GiB for Llama-3.1-8B and 17.58/24.40 GiB for Qwen3-8B at 16K/64K, respectively, with batch size 1.

We clarify that AB-Sparse targets reducing KV-cache memory access during attention, rather than reducing KV-cache storage. The 4% budget specifies the KV positions accessed, while the full KV cache remains stored.

2. Calibration clarification

We use 30 Wikipedia sequences of approximately 32K–64K tokens for calibration, separate from the evaluation data. Query positions are sampled every 10 tokens during decoding. Recall is averaged over calibration samples and the query heads. Calibration takes up to 30 minutes per model and is performed once offline, adding no calibration overhead to online inference. Quantization scales remain fixed during generation.

3. Qwen3 model card clarification

We use the official Qwen3-8B [1] and Qwen3-32B [2] checkpoints at revisions b968826d and 9216db57, respectively. For long-context evaluations, we enable YaRN with rope_type="yarn", factor=4.0, and original_max_position_embeddings=32768, following the configuration provided in their model cards.

The 256K results report attention-only latency, including metadata updates and sparse selection. When the complete model exceeds GPU memory, we measure all layers in separate groups and sum their attention latencies.

[1] Qwen3-8B model card. https://huggingface.co/Qwen/Qwen3-8B [2] Qwen3-32B model card. https://huggingface.co/Qwen/Qwen3-32B

4. Matched-precision accuracy–latency comparison

Appendix A.3 of the original submission already reports attention latency for uniform B=16, uniform B=32, and AB-Sparse with INT4 centroids on Llama-3.1-8B at 64K. We now provide supplementary INT4 accuracy results for the same setting in table 1:

MethodCentroid precisionRULER accuracy (%)Attention latency (ms)
Uniform B=16INT481.695.33
Uniform B=32INT476.934.25
AdaptiveINT481.494.53

Table 1. INT4 accuracy and attention latency.

Furthermore, Figure 18 in Appendix A.4 of the revised manuscript jointly reports accuracy and latency across additional models and longer contexts, using BF16 centroids and the same 4% retrieved-token budget. AB-Sparse extends the empirical accuracy–latency Pareto frontier of the evaluated uniform baselines, demonstrating the benefit of adaptive granularity independently of centroid quantization.

We clarify that AB-Sparse’s key contribution is head-adaptive block granularity, which adjusts the accuracy–latency trade-off across attention heads. In contrast, LServe uses uniformly fine-grained logical-page summaries to score and select larger physical pages for KV access. AB-Sparse further combines adaptive granularity with centroid quantization and custom kernels.

We compare AB-Sparse with LServe using Llama-3.1-8B at a 64K context length and batch size 1 on a single A100 GPU. LServe achieves 79.75% RULER accuracy with an attention latency of 6.34 ms, whereas AB-Sparse achieves 81.49% accuracy with an attention latency of 4.53 ms.

5. Additional benchmark results

We report the full-attention reference and mean output lengths as follows:

  • Order: AIME24/AMC23/MATH500
  • Full attention pass@1 (%): 43.3/75.0/80.0
  • Full attention pass@4 (%): 50.8/78.1/83.5
  • Uncertainty (accuracy standard deviations): 2.9/2.4/1.9 pp.
  • Mean output lengths (tokens): 26,497.81/18,869.66/10,577.54

We additionally evaluate LongBench v2, following the reviewer’s suggestion. We evaluate all examples that fit within the supported context window of both models. AB-Sparse consistently improves upon the baseline, demonstrating its benefit persist on a harder long-context task.

ModelFull attentionQUEST (B=32)AB-Sparse
Llama-3.1-8B32.9332.1332.73
Qwen3-8B39.0637.7538.96

Table 2. LongBench v2 accuracy (%).

Thank you for the clarifications and additional experiments. The matched-INT4 comparison addresses my main concern about adaptive block sizing. At 64K, AB-Sparse reduces attention latency by about 15% relative to uniform B=16, with a 0.20% drop in RULER accuracy. The additional BF16 results support the benefit of adaptive sizing across more settings.

The LServe and LongBench v2 results strengthen the evaluation. The response also clarifies that the speedups apply to the attention stage and that the full KV cache remains stored. The new evidence addresses my main concerns, so I am raising my rating from 2 to 3.

Thank you for reviewing our responses and additional experiments, and for updating your assessment. We appreciate your constructive feedback, which has helped us improve the work.

Reviewer_iVcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AB-Sparse uses offline calibration to assign each KV head a block size, asymmetric per-channel INT4 quantization for ranking centroids, and kernels supporting heterogeneous block counts and paged attention. At a 4% retrieved-token budget, it improves Quest and ArkVale's benchmark-average scores on Llama-3.1-8B, Qwen3-8B, and Qwen3-32B on RULER (16K–96K) and LongBench v1. Attention-latency measurements extend to 256K; model-throughput measurements use Llama-3.1-8B at 64K.

Critical strengths
  1. The improvement extends across block-scoring rules. Gains occur with both min-max and bounding-volume summaries across three models, supporting applicability to more than one block-selection estimator. Figure 5 also shows that Wikipedia-calibrated assignments improve recall over uniform B=32 on code, QA, and summarization inputs, providing evidence of cross-domain transfer.
  2. Quantization and kernel design have separate supporting measurements. INT4 achieves accuracy close to BF16 in the tested RULER settings, while the heterogeneous kernels reduce operation latency relative to the stated naive implementations.
Critical weaknesses
  1. The contribution is a modest extension of existing head-adaptive sparse attention. Section 5 describes prior head-specific policy selection in FastGen and offline calibration of head-specific sparsity in TCA-Attention. AB-Sparse adds block granularity to this design strategy: Eq. (2) selects the largest candidate block satisfying a relative-recall threshold. Its specific insight is that heads differ in their sensitivity to block size; centroid quantization and heterogeneous kernels make the resulting allocation practical (§§2.3–3.4). The measurements and ablations support this extension, but I find the additional methodological insight and demonstrated practical benefit insufficiently substantial to justify acceptance.
  2. Accuracy on long natural-text inputs and long generations is insufficiently characterized. Table 1 reports RULER accuracy through 96K, but the LongBench v1 results give category averages without input-length distributions or truncation limits (§4.1). Consequently, the natural-text results do not identify the context lengths over which the gains hold. Figure 9's 256K timings provide no additional accuracy evidence. Table 2 reports improvement over Quest under a 32K generation cap, but omits full-attention scores and observed output lengths, leaving the quality loss relative to the original model unclear. These gaps limit the evidence for the long-document and long-reasoning applications motivating the paper (§1).
Questions for authors
  • How does AB-Sparse's accuracy–latency trade-off compare with uniform B=16/B=32 on the other evaluated models and longer contexts when the methods use the same centroid precision and retrieved-token budget? Reporting accuracy and latency together would clarify the practical value of adaptive granularity.
  • Please specify LongBench's actual token-length distribution and truncation protocol, and provide the full-attention reference and observed output lengths for Table 2. If the current natural-text data contain few long inputs, a targeted LongBench v2 evaluation could address this coverage gap; MRCR or Graphwalks would provide complementary synthetic tests of conversation disambiguation or graph traversal. Such tests should use supported context lengths and report full-attention performance so that approximation loss is interpretable.
2 comments
AuthorsOpenReview ↗

Response to Reviewer iVcF

We thank the reviewer for the constructive suggestions. We address each concern as follows.

1. Contribution clarification

We clarify that AB-Sparse’s contribution goes beyond head adaptation itself. Compared with head-adaptive approaches that primarily adjust token budgets, AB-Sparse introduces a distinct optimization dimension: selection granularity, through per-head block-size allocation. Furthermore, AB-Sparse exploits the robustness of centroid-based selection to low-precision representations and provides custom kernels for efficient quantized estimation and heterogeneous block granularities. These components jointly contribute to its accuracy–latency trade-off.

2. Matched-precision accuracy–latency comparison

Thank you for suggesting a joint accuracy–latency comparison across additional models and longer contexts.

Appendix A.3 of the original submission already provides this comparison for Llama-3.1-8B at 64K: Figure 16 reports accuracy with unquantized centroids, and Figure 17 includes the corresponding BF16-centroid decode attention latencies.

We now extend the comparison to Llama-3.1-8B at 96K and Qwen3-8B and Qwen3-32B at 64K. Figure 18 in Appendix A.4 of the revised manuscript reports accuracy and latency together by plotting RULER accuracy against decode attention latency, with each point representing the same method and configuration. All methods use BF16 centroids and the same 4% retrieved-token budget. Within each setting, all methods share the same model, evaluation inputs, generation settings, and measurement protocol. Decode attention latency is measured on a single A100 GPU with batch size 1.

Across the four settings, AB-Sparse reduces latency by 13.89–16.85% relative to uniform B=16 while improving RULER accuracy by 3.00–3.71 percentage points over uniform B=32. These results show that AB-Sparse extends the empirical accuracy–latency Pareto frontier formed by the evaluated uniform baselines, demonstrating the practical value of adaptive granularity.

3. LongBench clarification

We report the actual token-length distribution for LongBench in Table 1. No truncation is applied.

ModelMean tokensP95Maximum≥32,768
Llama-3.1-8B10,75323,68765,3702.48%
Qwen3-8B10,49921,59965,4002.38%
Qwen3-32B10,49921,59965,4002.38%

Table 1. LongBench token-length distribution.

4. Reasoning benchmarks clarification

We report the full-attention reference and mean output lengths as follows:

  • Order: AIME24/AMC23/MATH500
  • Full attention accuracy (%): 50.8/78.1/83.5
  • Mean output lengths (tokens): 26,497.81/18,869.66/10,577.54

5. Evaluation on longer natural-text inputs

Our RULER evaluation already demonstrates performance on synthetic long-context tasks at context lengths up to 96K. To complement these results with longer natural-text inputs, we additionally evaluate LongBench v2 with full attention references, following the reviewer’s suggestion.

We evaluate all LongBench v2 examples that fit within the supported context window of both models. AB-Sparse consistently improves upon the baseline, demonstrating its capability with long natural-text inputs.

ModelFull attentionQUEST (B=32)AB-Sparse
Llama-3.1-8B32.9332.1332.73
Qwen3-8B39.0637.7538.96

Table 2. LongBench v2 accuracy (%).

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period draws to a close, we would appreciate your thoughts on our responses and additional experiments.

We hope these additions address your concerns and would welcome any remaining questions. If you find that they resolve the issues raised in your review, we would appreciate your reconsideration of your assessment.

Thank you for your time and consideration.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank all reviewers. To help the discussion converge, we summarize what we believe distinguishes AB-Sparse and what we do and do not claim.

  1. A new axis that prior methods hold constant. Block-sparse methods fix one selection granularity: Quest and ArkVale use a uniform block size, LServe a uniform estimation page size, and SeerAttention-R a single learned block size. Head-adaptive methods adapt other quantities: DuoAttention decides which heads to sparsify, and Ada-KV and TCA-Attention decide how many tokens each head keeps. AB-Sparse adapts how finely each head selects. This matters because heads differ widely in sensitivity to granularity: some keep near-perfect recall, while others lose over 90% of it when blocks grow from 16 to 32 tokens, so no single block size serves all heads.

  2. The gain comes from matching, not from heterogeneity. Under the same block-size histogram and the same 4% budget, randomly permuting the assignments gives 74.54 RULER (Llama-3.1-8B, 64K), below uniform Quest (77.48), whereas the calibrated assignment gives 81.49.

  3. A better accuracy-latency frontier at matched precision. With BF16 centroids, across four settings, AB-Sparse reduces decode attention latency by 13.9-16.9% relative to uniform B=16 while improving RULER by 3.0-3.7 points over uniform B=32. With INT4 centroids on Llama-3.1-8B at 64K: B=16, 81.69 / 5.33 ms; B=32, 76.93 / 4.25 ms; AB-Sparse, 81.49 / 4.53 ms. Against LServe in the same setting: 79.75 / 6.34 ms vs. 81.49 / 4.53 ms.

  4. Why it is not a simple extension. Per-head granularity creates costs that uniform schemes avoid: (i) a one-time calibration that must transfer across domains (Wikipedia-calibrated assignments improve recall on code, QA and summarization inputs); (ii) a larger centroid footprint, which we control with per-channel asymmetric INT4 centroids that only rank blocks and never enter the attention output; and (iii) ragged block counts across heads, which our kernels batch without padding and map onto standard paged attention. In SGLang, the system gives 1.04-1.08x end-to-end decoding speedup over Quest and 1.33-1.64x over full attention (batch size 1).

  5. Scope. We claim better accuracy-efficiency than block-sparse baselines at a fixed retrieved-token budget. We do not claim full-attention parity (notably on long reasoning), KV-storage reduction, or support for every serving optimization. In the revision we will (a) qualify "effective for long generation" as an improvement over the sparse baseline, (b) state which latency results are measured by layer groups and summed, (c) correct the GPU label of the SGLang results (H200), and (d) expand the related-work positioning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.