acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Adaptive Sparse Attention over a Multi-Resolution Cache Index

Abstract

Long-context inference in large language models is increasingly constrained by the cost of reading a growing key-value (KV) cache. To address this bottleneck, we introduce Hierarchical Adaptive Sparse Attention (HASA), a sparse attention operator that adapts how much detail it retrieves from different parts of the context. HASA organizes each layer’s cache into a hierarchy of token-level representations and learned summaries over progressively larger segments. For each query and layer, a learned router selects the resolution at which each historical span enters attention, ranging from token-level KV states to increasingly compact learned summaries. The summaries and router are learned through distillation and task-level fine-tuning with the backbone frozen. On Qwen3-8B, HASA improves the average score across all sixteen English LongBench tasks by +1.1 points over full attention while attending to 4.3× fewer KV entries. With a fused decode kernel, HASA decodes faster than FlashAttention-3 at all measured context lengths, reducing latency by 12.3% and 20.0% at 65K and 131K, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, KV Cache, Routing, Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vkJaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes HASA (Hierarchical Adaptive Sparse Attention) to improve LLMs' long-context performance and reduce their inference time by introducing a router and compressor for hierarchical stored KV caches. The experiments on long-context benchmarks (English LongBench, 16 tasks) with the Qwen3 model demonstrate the advantages in decoding speed and long context performance.

Critical strengths

Experiments and detailed results are provided in the Appendix, indicating repeatability.

Critical weaknesses
  • The architecture figure is too simple to illustrate the core mechanism or training pipeline. For example, the compressor and retrieval projector are hard to locate.
  • I only see the main paper. There are no ablation studies showing which parts contribute to the final performance, including training stages and modules. If these kinds of analyses are in the appendix, they should be moved forward.
  • And it's not suitable to compare with FlashAttention-3, which is an optimization for dense attention calculation. The proposed method is a sparse attention approach, which can be combined with FlashAttention. Claiming that the proposed method is faster than FlashAttention in some conditions is not helpful for proving the efficiency of this approach. However, the paper treats it as a main result.
  • Only one model has been tested: Qwen-8B/4B. The model diversity is completely not enough.
  • Model sizes should be more diverse and scale up a little bit, at least. For this kind of attention mechanism, I believe a 30B model should be tested to prove the performance and scalability.
  • Considering the stability of the long context benchmark, the 1.1 points improvement is not convincing for a reasonable contribution. It is probably worse on the other settings or prompt variants. Stability analysis should be included.
Questions for authors
  1. Why was it only tested on two models and one benchmark, although it has 16 tasks, but one source of benchmark could introduce bias?
  2. How do you know the contribution of each module and the training stage? Do you have a related analysis in the appendix?
  3. What's the correspondence between the limitations you identified in the second paragraph of the introduction and the proposed novelties? I didn't see a clear logic for the new ideas based on the previous limitations.
7 comments

Q1: The architecture figure is too simple to illustrate the core mechanism or training pipeline. For example, the compressor and retrieval projector are hard to locate.

R1: Thank you for the constructive feedback. The revised manuscript includes two detailed diagrams in Appendix H, covering the architecture and the three-stage training pipeline.

The architecture diagram separates the main components:

  • Compressors: turn each 16-token span into four L1 summaries or one L2 summary, while retaining its original KV entries as L0.
  • Retrieval projector and router: the projector produces an embedding for each span; the router compares these embeddings with the current query and chooses L0, L1, or L2 for each span.
  • Attention: reads the selected KV entries or summaries, together with the uncompressed initial tokens and recent window. The retrieval embeddings are used for routing, not as attention keys or values.

The training diagram identifies the modules and losses for each stage: Stage 1 learns the summaries, Stage 2 learns span ranking from full attention, and Stage 3 learns how to assign resolutions to balance compression and prediction quality. The backbone remains frozen throughout. We will also revise the main-text figure to make the components easier to locate.

Q2: There are no ablation studies showing which parts contribute to the final performance, including training stages and modules. If these kinds of analyses are in the appendix, they should be moved forward.

R2: We appreciate the suggestion to make the ablations more visible. Section 5.3 of the submitted paper already examines how learned summaries and the intermediate L1 level affect LongBench performance. Further analyses appear in Appendices D and E; below, we add training-stage and routing ablations.

Training stages. We evaluate all 3,750 LongBench-16 examples. Score differences are HASA minus full attention (FA), in points; brackets give 95% paired-bootstrap confidence intervals. Compression ratio (CR) is the ratio of dense KV entries to entries read by attention, averaged over examples.

Training configurationScore difference vs. FA [95% CI]CR
Stage 1 only: compressors, no router, all historical spans at L1-12.50 [-13.64, -11.37]3.19x
Stages 1-2: retrieval projector and router added, thresholds calibrated without Stage-3 training+1.02 [+0.46, +1.58]4.24x
Stages 1-3: reported HASA model, as in the submission+1.06 [+0.50, +1.61]4.35x

The last row is the result reported in the submission; the other rows are new evaluations, each against FA from the same run. Stage 2 learns a span ranking; the middle row sets routing thresholds from its scores on a calibration set, without Stage-3 training (Appendix A, "Cutpoint initialization"). The results show that compressors alone are insufficient, while adding trained retrieval and routing recovers quality. Stage 3 increases compression from 4.24x to 4.35x with similar quality.

Routing at matched compression. For each example and layer, we keep the average L0/L1/L2 span counts over decoding fixed and change only the span ranking; the compressor stays unchanged. Higher-ranked spans receive raw KV entries; lower-ranked spans receive L1 or L2 summaries. Each control achieves a CR of about 4.35x, differing from HASA by no more than 0.02x.

Alternative span rankingScore difference: HASA - alternative, points [95% CI]
Recency+10.25 [+9.22, +11.26]
Seeded random order+9.07 [+8.06, +10.08]
Quest-style ranking, every decode step+1.08 [+0.56, +1.59]
SnapKV-style ranking, once at the end of prefill+1.58 [+1.02, +2.13]
HASA's first-step ranking, reused for later steps+2.21 [+1.74, +2.69]

The Quest-style ranking orders spans at every decode step by Quest's upper bound on their attention scores. The SnapKV-style ranking orders spans once by the attention they receive from the last 32 prompt tokens. These are ranking controls within HASA, not the complete Quest or SnapKV methods. At matched compression, HASA outperforms all tested alternative rankings, showing that learned span selection improves quality.

Learned summaries and the L1 level. In Section 5.3, replacing learned summaries with selected original token KV entries changes the LongBench-16 difference from +1.06 to -0.04 points. Reading spans assigned to L1 as more compressed L2 summaries instead gives +0.64 points at 4.77x CR. Thus, learned summaries are important to the gain, while L1 provides a finer choice between raw entries and the most compressed summaries.

We will bring the key module and training-stage results into Section 5.3, with the complete tables and protocols in the appendix. We will also clarify the caption of Table 1 and the surrounding text, so that it is clear the table shows what each component adds.

Q3: It's not suitable to compare with FlashAttention-3, which is an optimization for dense attention calculation. The proposed method is a sparse attention approach, which can be combined with FlashAttention. Claiming that the proposed method is faster than FlashAttention in some conditions is not helpful for proving the efficiency of this approach. However, the paper treats it as a main result.

R3: Thank you for raising this point. We agree that HASA and FlashAttention-3 (FA3) are complementary. HASA chooses which KV representations to read, while FlashAttention makes attention computation efficient. HASA already uses exact FA3 for prefill and a FlashAttention-style custom kernel for decoding.

FA3 serves as an optimized full-attention baseline: reading fewer KV entries only saves time if the reduction outweighs routing and data-access overhead. With our revised implementation, Qwen3-8B, bf16, batch size 1, and all KV caches on the GPU, HASA's routing and attention together take 0.92 ms at 8K and 2.26 ms at 131K, versus 1.15 and 5.56 ms for FA3's attention. Including routing, attention is therefore 1.25x and 2.46x faster. The total time to generate one token falls by 3.3% and 25.8%, respectively.

We also test sparse selection combined with FA3 by running FA3 on exactly the KV entries that HASA selects. Using the original HASA kernel, its attention takes 1.6 / 1.2 / 1.0 / 0.4 ms less per token at 16K / 32K / 65K / 131K than FA3 on those entries, even when the time to copy them into FA3's input buffer is excluded. This separate comparison is also reported in the revised Appendix F.3.

As a reference for a released sparse-attention kernel, we also time SeerAttention-R, a trained sparse-attention method with released Qwen3-8B gates, using the authors' code at the same 4.4x budget. Its sparse attention takes 1.13 ms per token at 8K and 14.93 ms at 131K, plus 2.14-2.72 ms for its gate, versus 0.92 and 2.26 ms for HASA's routing and attention together.

We will make these complementary roles explicit in the main text and distinguish the comparison against full attention from the comparison using the same selected entries.

Q4: Why was it only tested on two models and one benchmark, although it has 16 tasks, but one source of benchmark could introduce bias?

R4: Thank you for the thoughtful feedback. We agree that two Qwen3 sizes provide limited model diversity. To address this, we evaluate HASA on IBM's Granite-4.2-8B.

Another model family. We train HASA modules on Granite with the backbone frozen. On LongBench-16, score differences are relative to FA on Granite; brackets give 95% paired-bootstrap confidence intervals.

Granite-4.2-8B methodScore difference vs. FA [95% CI]CR
SnapKV-0.97 [-1.43, -0.52]4.56x
HASA+1.94 [+1.26, +2.62]5.02x

On Granite, HASA improves the average LongBench-16 score by 1.94 points over FA at 5.02x compression. We will document the model settings and per-task results in the revised appendix.

Existing benchmark coverage. The submitted paper already evaluates more than one benchmark:

  • LongBench: 16 English tasks, including question answering, summarization, and code completion (Appendix C.3).
  • LongBench-v2: 503 questions (Appendix C.4).
  • RULER: 13 synthetic tasks testing retrieval and aggregation (Appendix C.5).
  • PG-19: next-token prediction on long book passages, measured by perplexity (Appendix C.6).
  • Short-context tasks: eleven tasks such as HellaSwag, ARC, MMLU, and GSM8K, on which HASA matches FA because the prompts fit in its uncompressed window (Appendix C.7).

Section 5.2 summarizes these results. We will make this coverage more explicit in the main text.

Q5: Model sizes should be more diverse and scale up a little bit, at least. For this kind of attention mechanism, I believe a 30B model should be tested to prove the performance and scalability.

R5: Thank you for suggesting a larger-model evaluation. We train HASA on Qwen3-32B with the same three-stage pipeline and a frozen backbone. The current checkpoint gives the following preliminary results; score differences are HASA minus FA, with 95% paired-bootstrap confidence intervals.

BenchmarkFA scoreHASA scoreScore difference [95% CI]CR
LongBench-v2, 503 questions34.7937.97+3.18 [+0.20, +3.78]6.24x

These results are consistent with HASA's main quality finding at a larger model scale: on Qwen3-32B, it achieves higher average scores than FA on LongBench-v2 while reading substantially fewer KV entries. We are finalizing the broader evaluation and will include the results and model configuration in the revised manuscript.

Q6: Considering the stability of the long context benchmark, the 1.1 points improvement is not convincing for a reasonable contribution. It is probably worse on the other settings or prompt variants. Stability analysis should be included.

R6: We appreciate the reviewer's thoughtful feedback on the stability of the results. We examine repeated evaluation, a retrained model, and a change in prompt format.

Repeated evaluation. On all 3,750 LongBench-16 examples, each row compares HASA with FA evaluated in the same run. Differences are in points; brackets give 95% paired-bootstrap confidence intervals.

EvaluationScore difference vs. FA [95% CI]CR
Result in the current submission+1.06 [+0.50, +1.61]4.35x
Same checkpoint, evaluated on a second machine+1.02 [+0.48, +1.55]4.35x

Both intervals exclude zero.

Retrained model. We also retrained Stage 3 of the Qwen3-8B model with a lower compression target, keeping the compressor and Stage-2 router. The new model is +1.03 [+0.54, +1.53] points above FA on LongBench-16, and on RULER-32K it is at parity with FA on the eleven tasks other than the two word-extraction tasks (-0.11 [-0.42, +0.19]), so the gain is consistent across retraining.

Prompt sensitivity. Without Qwen3's chat template, FA's score falls from 49.45 to 36.84 and HASA's from 50.51 to 36.93. HASA's advantage becomes +0.09 [-0.36, +0.54] points. Without the template the model itself collapses, with or without compression: on the eleven tasks that use the template, FA loses 18 points on average, and on question answering it falls from 59 to 12 on HotpotQA, from 43 to 13 on 2WikiMQA, and from 27 to 4 on NarrativeQA. In this setting compression does not make the model worse: HASA remains level with FA. Under the protocol the model is designed for, HASA is 1.06 points above FA. We will report these stability tests in the revised manuscript.

Q7: What's the correspondence between the limitations you identified in the second paragraph of the introduction and the proposed novelties? I didn't see a clear logic for the new ideas based on the previous limitations.

R7: We appreciate the reviewer's constructive suggestion. The second paragraph identifies two limitations, which motivate HASA's hierarchy and query-dependent routing:

LimitationHASA's responseSupporting experiment
Binary selection leaves unselected spans out of the current attention computationKeep every span represented, using original KV entries for some spans and learned summaries for othersDropping spans instead of reading their summaries reduces the LongBench-16 gain from +1.06 to +0.18 points (Appendix D).
Query-independent compression fixes the amount of detail before seeing the current queryChoose raw entries, L1 summaries, or L2 summaries separately for each query and layerWith the same compressor and matched compression, HASA exceeds recency ranking by 10.25 points and Quest-style ranking by 1.08 points.

The fused attention kernel addresses the broader goal of reducing decoding time: it reads this mixture of raw entries and summaries directly, yielding the measured speedups with routing included. We will make this mapping explicit in the introduction and link each design choice to its analysis in Section 5.3.

Reviewer_s5keOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HASA is a query-adaptive sparse attention operator for long-context decoding. Instead of evicting tokens or retrieving a subset, it keeps every context span represented at a query- and layer-dependent resolution: raw tokens, or learned summaries compressing 4 or 16 tokens into one KV entry. A per-layer router assigns each 16-token span a level via learned cutpoints. The backbone stays frozen; only the compressors, retrieval projector, and router are trained. On Qwen3-8B it slightly beats full attention on LongBench (+1.06) at 4.35× compression, with decode latency reductions growing to 20% at 131K.

Critical strengths

A genuinely different point in the design space — per-span, per-layer resolution selection with nothing dropped. The empirical package is thorough, with honest failure analysis (the paper doesn't hide its RULER deficit).

Critical weaknesses

The "beats full attention" headline doesn't hold on retrieval/aggregation: HASA trails full attention on RULER at every length, with big gaps on common/frequent-word extraction. Latency and memory savings don't coexist in one configuration: resident HASA is larger than the dense cache (25.95 vs. 19.32 GB at 131K), and with offloading the speedup mostly evaporates. Comparisons are mostly against training-free baselines despite HASA requiring substantial training, and a few baselines look broken (DuoAttention at −11.90, KNorm at −37.87). The perplexity-beats-teacher result is surprising and unexplained.

Questions for authors

What does HASA score on RULER at matched ~4.4× compression? What explains perplexity below full attention? Will the code and kernel be released?

5 comments

Q1: What does HASA score on RULER at matched ~4.4× compression?

R1: Thank you for the constructive feedback. Appendix E of the submitted paper already reports a result near the requested compression ratio (CR). For spans that the router assigns to L1, attention reads their original token KV entries instead of their L1 summaries. This variant reaches 4.65x compression and scores 89.91 versus 92.10 for full attention (FA), about 2.2 points lower (in that appendix's setting: a 128-token recent window and 100 examples per task).

Additional lower-compression experiment. We keep the compressor fixed and retrain only Stage 3 with a lower compression target and a calibration set covering multiple task families. We evaluate all 13 RULER tasks at 32K using 500 examples per task and a 512-token uncompressed recent window. Score differences are HASA minus FA; brackets give 95% paired-bootstrap confidence intervals. The RULER score difference is -1.15 [-1.45, -0.86] points.

The remaining gap is concentrated in common-word and frequent-word extraction: excluding these two tasks leaves a difference of -0.11 [-0.42, +0.19] points across the other 11 tasks. On LongBench-16, the same model scores +1.03 [+0.54, +1.53] points above FA.

Regarding the headline, the submission states the gain over full attention for LongBench (abstract and conclusion) and reports the RULER deficit in Section 5 and Appendix E. With the new model, the remaining RULER gap is confined to the two word-extraction tasks.

We will highlight the existing 4.65x result and add the new lower-compression results and per-task breakdown to the revised manuscript.

Q2: Can latency and memory savings coexist?

R2: Yes, Table 30 of the submitted paper already reports lower GPU KV storage when the raw KV cache (L0), or both L0 and the first summary level (L1), are stored in CPU memory, with decode latency at 131K 9.4% and 2.6% lower than FlashAttention-3 (FA3), respectively. To further reduce transfer delays, we optimize the attention implementation to process KV entries already on the GPU while fetching missing entries from CPU memory.

The updated measurements use Qwen3-8B with bf16 weights and KV storage, PCIe Gen5 x16, batch size 1, CUDA graphs, and exact FA3 prefill. We evaluate four prompts. The measured decode latency includes model computation, routing, and waiting for KV entries to be transferred from CPU memory to the GPU over PCIe. The paired measurement protocol is described in Appendix F.3.

GPU KV storage includes GPU-resident raw and summary caches, the index, recent window, and fetched-entry cache. At 32K / 65K / 131K, dense FA3 uses 4.83 / 9.66 / 19.32 GB of GPU KV storage and takes 9.77 / 11.03 / 13.25 ms per token. Both columns show percentage changes relative to FA3 at the same context length. Negative values mean reductions; positive values mean increases.

ConfigurationGPU KV storage change, 32K / 65K / 131KDecode latency change, 32K / 65K / 131K
Dense FA3ReferenceReference
HASA, all KV caches on GPU+34.2% / +34.3% / +34.3%-8.9% / -17.2% / -25.8%
HASA, L0 in CPU memory-31.1% / -37.9% / -35.1%-1.8% / -4.7% / -14.1%
HASA, L0+L1 in CPU memory-48.9% / -56.2% / -54.3%+4.6% / +4.6% / -3.9%

Offloading L0 alone already provides both savings at every measured length: GPU KV storage is 31-38% below FA3, and decoding is faster than FA3 at 32K, 65K, and 131K. Offloading L1 as well roughly halves GPU KV storage relative to FA3 and becomes faster than FA3 at 131K. Which tiers are kept in CPU memory is set per request, so L1 can be offloaded only for contexts long enough to benefit, or when GPU memory is the binding constraint.

At 131K, peak GPU memory allocated during decoding, including weights, is 45.4 GiB for FA3, 37.9 GiB with L0 in CPU memory, and 34.4 GiB with L0+L1 in CPU memory. Thus, L0 offload reduces GPU memory usage and decodes faster than FA3 at all three context lengths, and both offload configurations do so at 131K, even with CPU-to-GPU transfer overhead included.

We have also tested storing the KV caches in FP8 instead of bf16 to reduce their size further. On LongBench-16, HASA with FP8 storage scores -0.12 [-0.37, +0.13] points relative to HASA with bf16 storage, which is not a significant change. At 131K, GPU KV storage is then 67% below FA3 with L0 in CPU memory and 75% below with L0+L1, with decode latency 10.8% and 7.4% lower, respectively. We will update the manuscript with these measurements and implementation details.

Q3: Are comparisons fair when HASA is trained but most baselines are training-free?

R3: We appreciate the reviewer highlighting the importance of comparisons with trained sparse-attention methods. To address this concern, we add SeerAttention-R as a trained sparse-attention baseline, using its released Qwen3-8B gates. Both methods train additional modules while keeping the backbone frozen.

We evaluate SeerAttention-R and HASA on the same 3,750 LongBench-16 examples. The table reports each method's score difference from full attention (FA), with 95% paired-bootstrap confidence intervals.

MethodCRScore difference: method - FA, points [95% CI]
SeerAttention-R4.27x-0.01 [-0.50, +0.48]
HASA4.35x+1.06 [+0.50, +1.61]

The HASA row is the result reported in the submission; in the same new evaluation as SeerAttention-R, the original checkpoint gives +1.02 [+0.48, +1.55]. Paired on the same examples, HASA outperforms SeerAttention-R by 1.03 [0.42, 1.63] points on LongBench-16 at similar compression ratios. On RULER-32K at matched compression (6.32x for SeerAttention-R, 6.38x for HASA), HASA is 5.58 [4.96, 6.22] points higher. We will add this trained-baseline comparison and clarify the distinction from training-free methods in the revised manuscript.

Q4: Are the DuoAttention and KNorm results reliable?

R4: Thank you for flagging these results. We identify an evaluation-code bug affecting DuoAttention: kvpress versions before 0.5.5 omit Qwen3's query/key normalization when scoring attention heads. With kvpress 0.5.5, DuoAttention's LongBench-16 score is only 0.38 points below FA, compared with 11.90 points in the submission. The corrected results are:

BenchmarkCRScore difference: DuoAttention - FA, points [95% CI]
LongBench-163.31x-0.38 [-0.95, +0.19]
RULER-32K3.86x-4.23 [-4.64, -3.82]

The other kvpress baselines (SnapKV, PyramidKV, KVzip, ThinK, and ExpectedAttention) compute queries through a shared code path that applies this normalization in both versions, so their results are unaffected. We will replace the affected results and specify the corrected implementation version.

For KNorm, reruns on two machines with kvpress 0.5.5 still give a LongBench-16 score 37.54 points below FA. The cause remains unresolved. We will note this in the appendix.

Q5: What explains perplexity below full attention?

R5: We thank the reviewer for this thoughtful feedback. Stage 1 trains the compressor with two objectives: keeping the model's next-token predictions close to those of full attention (FA), and increasing the probability assigned to the actual next token (Section 4.1). The second objective rewards correct prediction, not just imitation of FA. Thus, even with the backbone frozen, training the compressor can improve next-token prediction and lower perplexity.

Two observations support this explanation.

  1. The gain comes from the trained summaries, not from how HASA reads the cache. On 48 PG-19 books with 32K-token contexts, when HASA reads only original KV entries without compression, its negative log-likelihood (NLL) is nearly identical to FA's (the difference is at most 0.00007 nats). Reading historical spans as L2 summaries lowers NLL by 0.047 [0.033, 0.060] nats relative to FA (95% confidence interval over books).
  2. Only trained summaries show the gain. In Table 28 of the submitted paper, where every span is read at one level, the learned L2 summaries reach a perplexity of 17.61 at 32K, below FA's 18.44. Summaries built without training are above FA at the same level: 19.70 when each span keeps its most-attended token, and 20.52 with mean pooling.

We will add this explanation and the 48-book measurement to Appendix E.

Q6: Will the code and kernel be released?

R6: Yes. Upon acceptance, we will release the code, custom attention kernels, training and evaluation pipelines, and trained module checkpoints. We will include setup instructions, supported model and hardware configurations, and the settings needed to reproduce the reported results.

Thank you again for the constructive feedback.

Reviewer_7KrFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HASA (Hierarchical Adaptive Sparse Attention), a query-adaptive sparse self-attention operator for long-context decoding. It organizes each layer's KV cache into a multi-resolution hierarchy — token-level L0 plus learned summaries over 4- and 16-token segments (L1/L2) — and uses a query-dependent router to pick, per history span, the resolution at which it enters attention. This sits between binary sparse selection and query-independent compression: every span stays represented, but its resolution adapts to the query and layer. The added modules are trained in three stages with the backbone frozen (compressors → retrieval projector + router distilling the teacher's span ordering → primal-dual learning of routing cutpoints). Validation: on Qwen3-8B, HASA exceeds full attention by +1.06 on the 16-task English LongBench macro at 4.35× attended-KV compression, and by +1.2 on LongBench-v2; Qwen3-4B gains +0.80. With a fused decode kernel it reduces decode latency vs. FlashAttention-3 by 12.3%/20.0% at 65K/131K, and hybrid prefill cuts TTFT by 38% at 131K. Experiments include paired 95% CIs, a held-out task half, and several ablations.

Critical strengths

A novel, well-substantiated core mechanism. Query-adaptive multi-resolution routing still exceeds full attention by +1.06 (paired 95% CI [+0.50, +1.61], significant) at 4.35× compression, and a clean evaluation-time ablation shows the gain comes from the learned summaries themselves (replacing them with an eviction bank drives the gain to zero, −0.04). This is the strongest evidence supporting acceptance: it shows both that the idea works and that the source of the gain is interpretable.

Critical weaknesses
  1. The most favorable efficiency configuration is not demonstrated. The paper is pitched on efficiency and headlines "4.3× fewer KV entries," yet Figure 5(b) shows that keeping all tiers on the GPU uses more memory than dense; the offload L0/L1 path that is supposed to deliver both lower memory and preserved speed has no full latency/memory measurement of its PCIe/host-bandwidth cost. Combined with the minimal speedup at short context, gains concentrated at 65K–131K, and reliance on a custom fused kernel, the single thing that most needs to be proven for an efficiency paper is precisely the thing left unverified. This is my primary reason to lean toward rejection.

  2. The router's actual contribution cannot be isolated. The paper positions itself as superior to "query-independent compression," yet it never runs the most direct control: HASA's own compressor, router disabled, every span uniformly compressed at L1 (≈4×) at iso-compression. All existing ablations retain the router. As a result, it is impossible to tell how much of the +1.06 comes from "adaptive routing" versus simply from "a well-learned compressor" — which is the paper's central claim.

Questions for authors
  1. (Clarification) Could you add a "no router, uniform L1 (≈4×), compressor-only" iso-compression baseline (the Stage-1 single-level mode should make this cheap to produce) to quantify how much of the +1.06 gain is attributable to the router?

  2. (Clarification) What are the end-to-end latency and GPU-memory curves under the offload L0/L1 configuration? At 131K, does PCIe/host bandwidth become a bottleneck that offsets the claimed "lower memory without losing speed"?

4 comments

Q1: What are the end-to-end latency and GPU-memory curves under the offload L0/L1 configuration? At 131K, does PCIe/host bandwidth become a bottleneck that offsets the claimed "lower memory without losing speed"?

R1: Thank you for the insightful feedback. Table 30 of the submitted paper reports decode latency and GPU KV storage with the raw KV cache (L0), or both L0 and the first summary tier (L1), stored in CPU memory and fetched to the GPU over PCIe as needed. At 131K, both configurations use less GPU KV memory than full attention implemented by FlashAttention-3 (FA3), while reducing decode latency by 9.4% and 2.6%, respectively.

To further reduce time spent waiting for these transfers, we optimize the attention implementation to process KV entries already on the GPU while fetching missing entries from CPU memory. With 131K-token contexts, the updated L0 and L0+L1 offload configurations use 12.53 and 8.82 GB of GPU KV memory, versus 19.32 GB for FA3. They reduce decode latency by 14.1% and 3.9%, respectively.

Measurements across context lengths. All tests use Qwen3-8B with bf16 model weights and KV storage, PCIe Gen5 x16, batch size 1, CUDA graphs, and exact FA3 prefill. We use four prompts. Each configuration is paired with FA3 in the same session over five interleaved repetitions of each prompt, each averaged over 27 decode steps. We measure the total elapsed time of each decode step, covering the model's computation (including HASA's routing) and any waiting for KV entries fetched over PCIe.

The table reports GPU KV storage, including the raw KV and summary caches stored on the GPU, the retrieval index, the recent window, and cached copies of fetched entries. At 32K / 65K / 131K, dense FA3 uses 4.83 / 9.66 / 19.32 GB of GPU KV storage and takes 9.77 / 11.03 / 13.25 ms per token. Both columns show percentage changes relative to FA3.

ConfigurationGPU KV storage change, 32K / 65K / 131KDecode latency change, 32K / 65K / 131K
Dense FA3ReferenceReference
HASA, all KV caches on GPU+34.2% / +34.3% / +34.3%-8.9% / -17.2% / -25.8%
HASA, L0 in CPU memory-31.1% / -37.9% / -35.1%-1.8% / -4.7% / -14.1%
HASA, L0+L1 in CPU memory-48.9% / -56.2% / -54.3%+4.6% / +4.6% / -3.9%

Offloading L0 alone already provides both savings at every measured length: GPU KV storage is 31-38% below FA3, and decoding is faster than FA3 at 32K, 65K, and 131K. Offloading L1 as well roughly halves GPU KV storage relative to FA3 and becomes faster than FA3 at 131K. Which tiers are kept in CPU memory is set per request, so L1 can be offloaded only for contexts long enough to benefit, or when GPU memory is the binding constraint.

Peak GPU memory. At 131K, peak GPU memory allocated during decoding, including model weights, is 45.4 GiB for FA3, 37.9 GiB with L0 in CPU memory, and 34.4 GiB with L0+L1 in CPU memory. CPU storage for L0 / L0+L1 is 19.3 / 24.1 GB. With L0 in CPU memory, peak GPU memory over the entire request, including prefill, is also lower: 46.8 GiB versus 60.3 GiB for FA3.

FP8 storage. We have also tested storing the KV caches in FP8 to reduce their size further, with no significant change in accuracy (-0.12 [-0.37, +0.13] LongBench-16 points). At 131K, GPU KV storage is then 67% below FA3 with L0 in CPU memory and 75% below with L0+L1, with decode latency 10.8% and 7.4% lower, respectively.

PCIe transfer cost at 131K. PCIe transfers reduce the speedup, but do not cancel the simultaneous GPU-memory and decode-latency savings in these tests. The recorded KV transfer volume per generated token is 47 MiB with L0 in CPU memory and 76 MiB with L0+L1 in CPU memory. Dividing by the time to generate one token gives an average transfer demand of about 4.3 and 6.3 GB/s, respectively. On the same GPU, the measured rate of reading data from CPU memory over PCIe is 33 GB/s. These averages are 13% and 19% of that rate.

To test the effect of a faster transfer path, we store L0 on a second GPU and read it over NVLink. At 131K, decode latency is then 22.1% lower than FA3, compared with 14.1% for CPU-memory offload over PCIe, while KV storage on the decoding GPU remains about 35% below FA3. The lower latency with NVLink suggests that PCIe fetches account for part of the slowdown caused by offloading. Since the average transfer demand is well below the PCIe rate, this cost comes from waiting for individual transfers rather than from the capacity of the link, which leaves room for further optimization of the kernel, for example by issuing transfers earlier in the decode step.

Thus, L0 offload reduces GPU memory usage and decodes faster than FA3 at all three context lengths, and both configurations do so at 131K, with transfer overhead included. We will add these measurements, the transfer-cost analysis, and the implementation details to the revised manuscript.

Q2: Why are gains small at short context, and how dependent are they on the custom kernel?

R2: HASA and the FA3 baseline both spend about 7 ms per token on linear layers, normalization, and positional encoding, which HASA does not accelerate (Appendix F.3). At short context lengths, attention accounts for a smaller share of total decode time, so accelerating it has a smaller effect on overall latency. With our revised implementation and all KV caches on the GPU, HASA's routing and attention together take 0.92 ms at 8K and 2.26 ms at 131K, versus 1.15 and 5.56 ms for FA3's attention. Including routing, the attention path is therefore 1.25x and 2.46x faster, while the full decode step is 3.3% and 25.8% shorter, respectively.

Long contexts matter for modern AI agents: tool definitions alone can consume tens of thousands of tokens before a conversation begins (about 55K for 58 tools in the example of [1]). This is the regime in which HASA achieves its largest latency reductions.

HASA's custom fused kernel reads the routed raw KV entries and summaries directly, avoiding an intermediate copy of the gathered KV data (Appendix F.2). FA3 is also an optimized kernel, so the comparison is between optimized implementations of sparse and full attention.

Two additional measurements show what the fused kernel contributes. First, we run FA3 on exactly the KV entries HASA selects, copied into a contiguous buffer without timing the copy. With the kernel of the submitted version, HASA's attention is faster than this reference by 1.6 / 1.2 / 1.0 / 0.4 ms per decode step at 16K / 32K / 65K / 131K. Second, we time the released block-sparse decode kernel of SeerAttention-R, a trained sparse-attention method with released Qwen3-8B gates, using the authors' code at the same 4.4x budget. Its sparse attention takes 1.13 ms per token at 8K and 14.93 ms at 131K, plus 2.14-2.72 ms for its gate, versus HASA's 0.92 and 2.26 ms.

We will release the kernels and evaluation code upon acceptance.

[1] Anthropic. Introducing advanced tool use on the Claude Developer Platform. 2025.

Q3: Could you add a "no router, uniform L1 (≈4×), compressor-only" iso-compression baseline to quantify how much of the +1.06 gain is attributable to the router?

R3: Thank you for the thoughtful suggestion. The submission reports a +1.06 [+0.50, +1.61] LongBench-16 advantage over full attention (FA) at 4.35x compression. We evaluate Qwen3-8B with the same compressor but disable routing: every historical span is read as L1 summaries, with one summary entry per four tokens. On all 3,750 LongBench-16 examples, this gives -12.50 [-13.64, -11.37] points relative to FA. As a reproducibility check, the original routed checkpoint gives +1.02 [+0.48, +1.55] in the same new evaluation. Brackets denote 95% paired-bootstrap confidence intervals. The compressor alone therefore does not explain this model's gain.

Matching compression. Uniform L1 reaches 3.19x compression rather than 4x because the initial attention-sink tokens and recent window remain uncompressed. It thus compresses less than routed HASA (4.35x) yet performs substantially worse. Reading every span as L2 summaries instead (one entry per 16 tokens) reaches 7.82x and gives -12.47 points.

To isolate span selection at matched compression, we preserve HASA's average L0/L1/L2 span counts over decoding for each example and layer, and change only which spans receive each level. Each alternative ranking assigns the highest-ranked spans to raw KV (L0), the next to L1 summaries, and the rest to L2 summaries. The compressor and summaries stay fixed. Each control matches HASA's 4.35x compression to within 0.02x.

Alternative span rankingScore difference: HASA - alternative, points [95% CI]
Recency+10.25 [+9.22, +11.26]
Seeded random order+9.07 [+8.06, +10.08]
Quest-style ranking, every decode step+1.08 [+0.56, +1.59]
SnapKV-style ranking, once at the end of prefill+1.58 [+1.02, +2.13]
HASA's first-step ranking, reused for later steps+2.21 [+1.74, +2.69]

The Quest-style ranking orders spans at every decode step by Quest's upper bound on their attention scores. The SnapKV-style ranking orders spans once by the attention they receive from the last 32 prompt tokens. These are ranking controls within HASA, not the complete Quest or SnapKV methods. At matched compression, HASA outperforms all tested alternative rankings. Its 2.21-point advantage over reusing its first-step ranking further supports updating span selection during decoding. We will include these experiments in the revised manuscript.

For Q1 (efficiency configuration), the offload measurements now provide the missing end-to-end evidence: L0 offload reduces GPU KV memory by 31–38% and is faster than FA3 at 32K/65K/131K; L0+L1 roughly halves GPU KV memory and is faster at 131K, with the 65K slowdown reported honestly. The PCIe demand analysis and NVLink comparison convincingly show that link capacity is not the bottleneck. For Q3 (router contribution), the matched-compression ranking controls are the decisive addition. In particular, the random-order control at matched compression isolates the router’s span selection, and HASA’s +9.07 over random, +1.08 over Quest-style, and +2.21 over reusing first-step rankings support the central claim that query-dependent routing matters beyond the learned compressor.

Thank you for the careful reading of our rebuttal and for raising your score. We are glad the offload measurements and the matched-compression controls settled your questions on efficiency and on the router's contribution. We would also welcome any other point you feel is not yet fully addressed, and we are happy to follow up before the discussion period ends.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Responses and Revisions

We thank all reviewers for their time and constructive feedback. Below is a brief overview, with references to the corresponding questions in our responses.

  1. GPU-memory savings and faster decoding coexist. On prompts with 131K-token contexts, using Qwen3-8B in bf16, storing raw KV entries (L0) in CPU memory reduces GPU KV storage by 35.1% and decode latency by 14.1% relative to full attention with FlashAttention-3 (FA3). Storing both L0 and L1 summaries in CPU memory reduces GPU KV storage by 54.3% and decode latency by 3.9%. Both latency measurements include transfer overhead. (Reviewer 7KrF, Q1; Reviewer s5ke, Q2.) With all KV caches on the GPU, HASA's routing and attention together are 1.25x faster than FA3's attention at 8K and 2.46x faster at 131K, and decode latency is 25.8% lower at 131K. (Reviewer 7KrF, Q2; Reviewer vkJa, Q3.)

  2. Both learned summaries and adaptive routing contribute. With the same compressor and routing disabled, the LongBench-16 score is 12.50 points below full attention, against +1.06 for HASA. At matched compression, HASA exceeds recency ranking by 10.25 points and Quest-style ranking by 1.08 points. Replacing the learned summaries with selected original KV entries removes the gain (+1.06 to -0.04, Section 5.3). (Reviewer 7KrF, Q3; Reviewer vkJa, Q2.)

  3. Evaluation now covers another model family and a 32B model. New experiments include Granite-4.2-8B and Qwen3-32B. On Granite-4.2-8B, HASA scores +1.94 [+1.26, +2.62] points above full attention on LongBench-16 at 5.02x compression. The preliminary 32B result improves LongBench-v2 by 3.18 points over full attention at 6.24x compression. (Reviewer vkJa, Q4 and Q5.)

  4. Baseline comparisons are strengthened. HASA outperforms the added trained SeerAttention-R baseline by 1.03 [0.42, 1.63] points on LongBench-16 at similar compression. We identify and correct the DuoAttention evaluation bug. (Reviewer s5ke, Q3 and Q4.)

We also address RULER performance and perplexity (Reviewer s5ke, Q1 and Q5), result stability (Reviewer vkJa, Q6), and presentation clarity (Reviewer vkJa, Q1, Q2, and Q7). We will incorporate the additional results and clearer explanations into the revised manuscript and release code, kernels, training/evaluation pipelines, and trained modules upon acceptance (Reviewer s5ke, Q6).

AuthorsOpenReview ↗

Qwen3-32B new benchmarks

Update to point 3 (Qwen3-32B)

We have completed the evaluation of our Qwen3-32B model on LongBench-16, where we compare HASA with full attention (FA), Quest and SnapKV; brackets give 95% paired-bootstrap confidence intervals.

MethodLongBench-16 scoreHASA minus method [95% CI]
FA43.32+7.99 [+7.28, +8.68]
Quest43.15+8.16 [+7.46, +8.86]
SnapKV42.32+8.99 [+8.29, +9.69]
HASA (4.06x compression)51.31

Most of the difference comes from the two code-completion tasks. On the other 14 tasks, HASA is +0.90 [+0.17, +1.65] above FA, +1.07 [+0.34, +1.82] above Quest, and +1.95 [+1.21, +2.69] above SnapKV. Further details are in our update to Reviewer vkJa (Q5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.