- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes HASA (Hierarchical Adaptive Sparse Attention) to improve LLMs' long-context performance and reduce their inference time by introducing a router and compressor for hierarchical stored KV caches. The experiments on long-context benchmarks (English LongBench, 16 tasks) with the Qwen3 model demonstrate the advantages in decoding speed and long context performance.
Critical strengths
Experiments and detailed results are provided in the Appendix, indicating repeatability.
Critical weaknesses
- The architecture figure is too simple to illustrate the core mechanism or training pipeline. For example, the compressor and retrieval projector are hard to locate.
- I only see the main paper. There are no ablation studies showing which parts contribute to the final performance, including training stages and modules. If these kinds of analyses are in the appendix, they should be moved forward.
- And it's not suitable to compare with FlashAttention-3, which is an optimization for dense attention calculation. The proposed method is a sparse attention approach, which can be combined with FlashAttention. Claiming that the proposed method is faster than FlashAttention in some conditions is not helpful for proving the efficiency of this approach. However, the paper treats it as a main result.
- Only one model has been tested: Qwen-8B/4B. The model diversity is completely not enough.
- Model sizes should be more diverse and scale up a little bit, at least. For this kind of attention mechanism, I believe a 30B model should be tested to prove the performance and scalability.
- Considering the stability of the long context benchmark, the 1.1 points improvement is not convincing for a reasonable contribution. It is probably worse on the other settings or prompt variants. Stability analysis should be included.
Questions for authors
- Why was it only tested on two models and one benchmark, although it has 16 tasks, but one source of benchmark could introduce bias?
- How do you know the contribution of each module and the training stage? Do you have a related analysis in the appendix?
- What's the correspondence between the limitations you identified in the second paragraph of the introduction and the proposed novelties? I didn't see a clear logic for the new ideas based on the previous limitations.
7 comments
Q3: It's not suitable to compare with FlashAttention-3, which is an optimization for dense attention calculation. The proposed method is a sparse attention approach, which can be combined with FlashAttention. Claiming that the proposed method is faster than FlashAttention in some conditions is not helpful for proving the efficiency of this approach. However, the paper treats it as a main result.
R3: Thank you for raising this point. We agree that HASA and FlashAttention-3 (FA3) are complementary. HASA chooses which KV representations to read, while FlashAttention makes attention computation efficient. HASA already uses exact FA3 for prefill and a FlashAttention-style custom kernel for decoding.
FA3 serves as an optimized full-attention baseline: reading fewer KV entries only saves time if the reduction outweighs routing and data-access overhead. With our revised implementation, Qwen3-8B, bf16, batch size 1, and all KV caches on the GPU, HASA's routing and attention together take 0.92 ms at 8K and 2.26 ms at 131K, versus 1.15 and 5.56 ms for FA3's attention. Including routing, attention is therefore 1.25x and 2.46x faster. The total time to generate one token falls by 3.3% and 25.8%, respectively.
We also test sparse selection combined with FA3 by running FA3 on exactly the KV entries that HASA selects. Using the original HASA kernel, its attention takes 1.6 / 1.2 / 1.0 / 0.4 ms less per token at 16K / 32K / 65K / 131K than FA3 on those entries, even when the time to copy them into FA3's input buffer is excluded. This separate comparison is also reported in the revised Appendix F.3.
As a reference for a released sparse-attention kernel, we also time SeerAttention-R, a trained sparse-attention method with released Qwen3-8B gates, using the authors' code at the same 4.4x budget. Its sparse attention takes 1.13 ms per token at 8K and 14.93 ms at 131K, plus 2.14-2.72 ms for its gate, versus 0.92 and 2.26 ms for HASA's routing and attention together.
We will make these complementary roles explicit in the main text and distinguish the comparison against full attention from the comparison using the same selected entries.
Q4: Why was it only tested on two models and one benchmark, although it has 16 tasks, but one source of benchmark could introduce bias?
R4: Thank you for the thoughtful feedback. We agree that two Qwen3 sizes provide limited model diversity. To address this, we evaluate HASA on IBM's Granite-4.2-8B.
Another model family. We train HASA modules on Granite with the backbone frozen. On LongBench-16, score differences are relative to FA on Granite; brackets give 95% paired-bootstrap confidence intervals.
| Granite-4.2-8B method | Score difference vs. FA [95% CI] | CR |
|---|---|---|
| SnapKV | -0.97 [-1.43, -0.52] | 4.56x |
| HASA | +1.94 [+1.26, +2.62] | 5.02x |
On Granite, HASA improves the average LongBench-16 score by 1.94 points over FA at 5.02x compression. We will document the model settings and per-task results in the revised appendix.
Existing benchmark coverage. The submitted paper already evaluates more than one benchmark:
- LongBench: 16 English tasks, including question answering, summarization, and code completion (Appendix C.3).
- LongBench-v2: 503 questions (Appendix C.4).
- RULER: 13 synthetic tasks testing retrieval and aggregation (Appendix C.5).
- PG-19: next-token prediction on long book passages, measured by perplexity (Appendix C.6).
- Short-context tasks: eleven tasks such as HellaSwag, ARC, MMLU, and GSM8K, on which HASA matches FA because the prompts fit in its uncompressed window (Appendix C.7).
Section 5.2 summarizes these results. We will make this coverage more explicit in the main text.
Q5: Model sizes should be more diverse and scale up a little bit, at least. For this kind of attention mechanism, I believe a 30B model should be tested to prove the performance and scalability.
R5: Thank you for suggesting a larger-model evaluation. We train HASA on Qwen3-32B with the same three-stage pipeline and a frozen backbone. The current checkpoint gives the following preliminary results; score differences are HASA minus FA, with 95% paired-bootstrap confidence intervals.
| Benchmark | FA score | HASA score | Score difference [95% CI] | CR |
|---|---|---|---|---|
| LongBench-v2, 503 questions | 34.79 | 37.97 | +3.18 [+0.20, +3.78] | 6.24x |
These results are consistent with HASA's main quality finding at a larger model scale: on Qwen3-32B, it achieves higher average scores than FA on LongBench-v2 while reading substantially fewer KV entries. We are finalizing the broader evaluation and will include the results and model configuration in the revised manuscript.
Q6: Considering the stability of the long context benchmark, the 1.1 points improvement is not convincing for a reasonable contribution. It is probably worse on the other settings or prompt variants. Stability analysis should be included.
R6: We appreciate the reviewer's thoughtful feedback on the stability of the results. We examine repeated evaluation, a retrained model, and a change in prompt format.
Repeated evaluation. On all 3,750 LongBench-16 examples, each row compares HASA with FA evaluated in the same run. Differences are in points; brackets give 95% paired-bootstrap confidence intervals.
| Evaluation | Score difference vs. FA [95% CI] | CR |
|---|---|---|
| Result in the current submission | +1.06 [+0.50, +1.61] | 4.35x |
| Same checkpoint, evaluated on a second machine | +1.02 [+0.48, +1.55] | 4.35x |
Both intervals exclude zero.
Retrained model. We also retrained Stage 3 of the Qwen3-8B model with a lower compression target, keeping the compressor and Stage-2 router. The new model is +1.03 [+0.54, +1.53] points above FA on LongBench-16, and on RULER-32K it is at parity with FA on the eleven tasks other than the two word-extraction tasks (-0.11 [-0.42, +0.19]), so the gain is consistent across retraining.
Prompt sensitivity. Without Qwen3's chat template, FA's score falls from 49.45 to 36.84 and HASA's from 50.51 to 36.93. HASA's advantage becomes +0.09 [-0.36, +0.54] points. Without the template the model itself collapses, with or without compression: on the eleven tasks that use the template, FA loses 18 points on average, and on question answering it falls from 59 to 12 on HotpotQA, from 43 to 13 on 2WikiMQA, and from 27 to 4 on NarrativeQA. In this setting compression does not make the model worse: HASA remains level with FA. Under the protocol the model is designed for, HASA is 1.06 points above FA. We will report these stability tests in the revised manuscript.
Q7: What's the correspondence between the limitations you identified in the second paragraph of the introduction and the proposed novelties? I didn't see a clear logic for the new ideas based on the previous limitations.
R7: We appreciate the reviewer's constructive suggestion. The second paragraph identifies two limitations, which motivate HASA's hierarchy and query-dependent routing:
| Limitation | HASA's response | Supporting experiment |
|---|---|---|
| Binary selection leaves unselected spans out of the current attention computation | Keep every span represented, using original KV entries for some spans and learned summaries for others | Dropping spans instead of reading their summaries reduces the LongBench-16 gain from +1.06 to +0.18 points (Appendix D). |
| Query-independent compression fixes the amount of detail before seeing the current query | Choose raw entries, L1 summaries, or L2 summaries separately for each query and layer | With the same compressor and matched compression, HASA exceeds recency ranking by 10.25 points and Quest-style ranking by 1.08 points. |
The fused attention kernel addresses the broader goal of reducing decoding time: it reads this mixture of raw entries and summaries directly, yielding the measured speedups with routing included. We will make this mapping explicit in the introduction and link each design choice to its analysis in Section 5.3.
Q1: The architecture figure is too simple to illustrate the core mechanism or training pipeline. For example, the compressor and retrieval projector are hard to locate.
R1: Thank you for the constructive feedback. The revised manuscript includes two detailed diagrams in Appendix H, covering the architecture and the three-stage training pipeline.
The architecture diagram separates the main components:
The training diagram identifies the modules and losses for each stage: Stage 1 learns the summaries, Stage 2 learns span ranking from full attention, and Stage 3 learns how to assign resolutions to balance compression and prediction quality. The backbone remains frozen throughout. We will also revise the main-text figure to make the components easier to locate.
Q2: There are no ablation studies showing which parts contribute to the final performance, including training stages and modules. If these kinds of analyses are in the appendix, they should be moved forward.
R2: We appreciate the suggestion to make the ablations more visible. Section 5.3 of the submitted paper already examines how learned summaries and the intermediate L1 level affect LongBench performance. Further analyses appear in Appendices D and E; below, we add training-stage and routing ablations.
Training stages. We evaluate all 3,750 LongBench-16 examples. Score differences are HASA minus full attention (FA), in points; brackets give 95% paired-bootstrap confidence intervals. Compression ratio (CR) is the ratio of dense KV entries to entries read by attention, averaged over examples.
The last row is the result reported in the submission; the other rows are new evaluations, each against FA from the same run. Stage 2 learns a span ranking; the middle row sets routing thresholds from its scores on a calibration set, without Stage-3 training (Appendix A, "Cutpoint initialization"). The results show that compressors alone are insufficient, while adding trained retrieval and routing recovers quality. Stage 3 increases compression from 4.24x to 4.35x with similar quality.
Routing at matched compression. For each example and layer, we keep the average L0/L1/L2 span counts over decoding fixed and change only the span ranking; the compressor stays unchanged. Higher-ranked spans receive raw KV entries; lower-ranked spans receive L1 or L2 summaries. Each control achieves a CR of about 4.35x, differing from HASA by no more than 0.02x.
The Quest-style ranking orders spans at every decode step by Quest's upper bound on their attention scores. The SnapKV-style ranking orders spans once by the attention they receive from the last 32 prompt tokens. These are ranking controls within HASA, not the complete Quest or SnapKV methods. At matched compression, HASA outperforms all tested alternative rankings, showing that learned span selection improves quality.
Learned summaries and the L1 level. In Section 5.3, replacing learned summaries with selected original token KV entries changes the LongBench-16 difference from +1.06 to -0.04 points. Reading spans assigned to L1 as more compressed L2 summaries instead gives +0.64 points at 4.77x CR. Thus, learned summaries are important to the gain, while L1 provides a finer choice between raw entries and the most compressed summaries.
We will bring the key module and training-stage results into Section 5.3, with the complete tables and protocols in the appendix. We will also clarify the caption of Table 1 and the surrounding text, so that it is clear the table shows what each component adds.