- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper uses the weight of the QK projection of attention (the spectrum of , ) to determine the sparsification strategy for each head. The feature rank for routing and the block granularity are adapted per head, and exact attention is computed on the selected KV.
Critical strengths
This paper shows that we can find a sparsification strategy by analyzing the weight matrix of QK. The idea is simple and training-free, and the rank selection is done offline, so the runtime overhead is small.
The theoretical analysis of the routing error is a nice addition.
Critical weaknesses
-
Tensor parallel (head parallel) imbalance. In HASA, both the rank r_h and the number of regions I_h differ across heads, and I_h is input-dependent. This makes the selection cost different per head, so under tensor parallelism the slowest GPU decides the latency. The paper does not discuss this at all, and the efficiency result (Fig. 4) is only single-kernel self-attention latency, with no hardware, batch size, multi-GPU, or end-to-end throughput.
-
Novelty of head-adaptive sparsity. Allocating different sparsity across heads is not new. SEA (Lee et al., ICLR 2024) already selects top-k jointly across heads, so each head gets a different budget dynamically. FlexPrefill also adjusts the budget per head. The new part of this paper is only where the adaptation signal comes from (the QK weight spectrum). The authors should clarify the difference from these kinds of works.
-
The baseline is too weak. We lack stronger baselines, such as FlexPrefill and MInference, which already allocate different sparsity per head.
Questions for authors
Please check weakness section to answer my concerns.
4 comments
Table R13. Accuracy results on LongBench-Pro using Ministral-3-8B-Instruct-2512.
| Method | Budget | Overall | En | Zh | Extreme | Hard | Moderate | Easy |
|---|---|---|---|---|---|---|---|---|
| Full attention | -- | 37.70 | 35.88 | 38.96 | 31.13 | 33.72 | 31.60 | 48.86 |
| FlexPrefill | 2048 | 28.87 | 28.34 | 29.40 | 17.01 | 30.28 | 23.22 | 38.53 |
| DHSA | 2048 | 32.20 | 30.94 | 33.47 | 23.52 | 32.30 | 24.54 | 41.27 |
| MInference | 2048 | 30.62 | 30.18 | 31.05 | 22.91 | 30.49 | 24.19 | 38.59 |
| UNIQUE | 2048 | 37.11 | 35.57 | 38.15 | 31.28 | 33.18 | 31.03 | 47.66 |
| HASA (Ours) | 2048 | 37.62 | 36.90 | 38.28 | 32.80 | 34.10 | 31.01 | 47.25 |
| FlexPrefill | 1024 | 22.52 | 23.00 | 22.04 | 14.64 | 21.33 | 17.65 | 30.41 |
| DHSA | 1024 | 28.28 | 26.85 | 29.71 | 17.98 | 29.94 | 23.34 | 36.50 |
| MInference | 1024 | 27.30 | 27.72 | 26.88 | 18.71 | 28.58 | 19.41 | 35.88 |
| UNIQUE | 1024 | 36.72 | 35.77 | 37.98 | 31.02 | 32.78 | 30.75 | 48.34 |
| HASA (Ours) | 1024 | 37.53 | 36.66 | 38.74 | 32.17 | 33.83 | 32.27 | 48.63 |
| FlexPrefill | 512 | 13.75 | 14.53 | 12.97 | 10.91 | 12.91 | 10.79 | 17.30 |
| DHSA | 512 | 23.47 | 24.10 | 22.85 | 14.24 | 24.87 | 17.10 | 31.74 |
| MInference | 512 | 25.70 | 25.19 | 26.20 | 17.27 | 27.86 | 19.97 | 32.82 |
| UNIQUE | 512 | 36.58 | 35.22 | 38.23 | 30.86 | 33.75 | 30.37 | 47.67 |
| HASA (Ours) | 512 | 37.43 | 35.02 | 40.13 | 30.91 | 34.27 | 32.94 | 48.46 |
Table R14. Accuracy results on RULER-32K using Ministral-3-8B-Instruct-2512.
| Method | Budget | SG1 | SG2 | SG3 | MK1 | MK2 | MK3 | MV | MQ | VT | CWE | FWE | QA1 | QA2 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full attention | -- | 100.00 | 99.60 | 99.60 | 99.40 | 100.00 | 99.80 | 98.55 | 99.05 | 99.96 | 88.46 | 93.27 | 75.80 | 57.40 | 93.15 |
| FlexPrefill | 2048 | 100.00 | 100.00 | 100.00 | 70.00 | 94.00 | 69.00 | 91.25 | 96.00 | 89.40 | 43.60 | 96.67 | 57.00 | 51.00 | 81.38 |
| DHSA | 2048 | 100.00 | 100.00 | 100.00 | 96.00 | 93.00 | 84.00 | 94.50 | 96.00 | 99.80 | 89.00 | 96.00 | 69.00 | 57.00 | 90.33 |
| MInference | 2048 | 100.00 | 100.00 | 99.00 | 99.00 | 100.00 | 93.00 | 95.50 | 100.00 | 99.60 | 38.50 | 93.67 | 71.00 | 61.00 | 88.48 |
| UNIQUE | 2048 | 100.00 | 99.80 | 99.20 | 99.60 | 100.00 | 99.80 | 99.00 | 99.15 | 99.96 | 82.28 | 91.07 | 75.60 | 57.40 | 92.53 |
| HASA (Ours) | 2048 | 100.00 | 100.00 | 100.00 | 100.00 | 99.00 | 99.80 | 98.51 | 99.15 | 100.00 | 83.86 | 90.43 | 76.53 | 57.40 | 92.67 |
| FlexPrefill | 1024 | 100.00 | 100.00 | 98.00 | 60.00 | 76.00 | 45.00 | 73.50 | 86.50 | 36.40 | 9.50 | 97.00 | 31.00 | 49.00 | 66.30 |
| DHSA | 1024 | 100.00 | 100.00 | 100.00 | 90.00 | 85.00 | 63.00 | 89.75 | 93.75 | 97.20 | 59.50 | 97.67 | 65.00 | 51.00 | 83.99 |
| MInference | 1024 | 100.00 | 100.00 | 99.00 | 98.00 | 97.00 | 76.00 | 89.50 | 99.75 | 93.80 | 27.50 | 95.33 | 70.00 | 52.00 | 84.45 |
| UNIQUE | 1024 | 100.00 | 99.80 | 99.20 | 99.60 | 100.00 | 99.80 | 99.00 | 99.15 | 99.96 | 75.38 | 90.96 | 75.16 | 57.56 | 91.97 |
| HASA (Ours) | 1024 | 100.00 | 100.00 | 100.00 | 100.00 | 99.01 | 99.79 | 98.51 | 99.15 | 100.00 | 78.39 | 90.27 | 76.09 | 57.56 | 92.21 |
| FlexPrefill | 512 | 91.00 | 90.00 | 62.00 | 28.00 | 10.00 | 9.00 | 38.50 | 45.75 | 3.00 | 4.20 | 86.33 | 14.00 | 26.00 | 39.06 |
| DHSA | 512 | 100.00 | 100.00 | 100.00 | 71.00 | 73.00 | 35.00 | 78.00 | 85.75 | 79.00 | 19.30 | 98.67 | 50.00 | 45.00 | 71.98 |
| MInference | 512 | 100.00 | 99.00 | 100.00 | 97.00 | 74.00 | 35.00 | 82.00 | 99.50 | 62.20 | 21.20 | 93.00 | 50.00 | 46.00 | 73.76 |
| UNIQUE | 512 | 100.00 | 99.80 | 99.20 | 99.80 | 100.00 | 99.60 | 99.55 | 99.30 | 99.96 | 62.90 | 88.67 | 73.60 | 57.80 | 90.78 |
| HASA (Ours) | 512 | 100.00 | 100.00 | 100.00 | 99.80 | 100.00 | 99.20 | 100.00 | 100.00 | 100.00 | 66.37 | 87.73 | 74.54 | 57.40 | 91.16 |
Thank you again for your thoughtful review and constructive feedback. As the discussion period is drawing to a close, we wanted to briefly check in and see if our responses and additional results have addressed your concerns.
Please let us know if you have any remaining questions or if there is anything else we can clarify. We sincerely appreciate your time and consideration.
Thank you again for your thoughtful review and constructive feedback. As the discussion period is approaching its end, we would like to briefly highlight that we have provided additional evidence directly addressing the three concerns raised in your original review:
W1 (Tensor-parallel efficiency): We conducted an 8-GPU tensor-parallel evaluation showing only a 0.15% difference in end-to-end decoding latency between adaptive and fixed resolutions. We also reported full-model decoding measurements, demonstrating up to 5.94× speedup over dense attention and approximately 1.19× over UNIQUE.
W2 (Novelty): We clarified that HASA jointly adapts feature and sequence resolutions based on the QK interaction spectrum, distinguishing it from prior methods that primarily adapt sparsity patterns or budgets across heads.
W3 (Competitive baselines): We added comparisons against FlexPrefill, MInference, and DHSA across three KV budgets, with HASA consistently achieving the strongest overall performance among the evaluated sparse-attention methods.
Given these additional analyses and experiments, we would greatly appreciate your reassessment of whether the original concerns have been sufficiently addressed, particularly whether any remaining issues materially affect the technical soundness or overall contribution of our work.
Thank you again for your time and consideration.
We appreciate your positive assessment of HASA's unified head-adaptive design, its clear technical narrative, and its training-free nature, as well as your recognition of the value of our theoretical analysis. We welcome the opportunity to address your questions.
W1a: Workload Imbalance under Tensor Parallelism
Thank you for raising this question. Although HASA's adaptive feature and sequence resolutions introduce variations in routing workload across GPUs, the routing computation accounts for only a small fraction of the overall decoding cost. This is because the decode latency is dominated by costs that are uniform across heads (e.g., kernel launches, fixed-budget attention).
To evaluate its practical impact, we compare adaptive and fixed resolutions under 8-GPU tensor parallelism on NVIDIA V100 GPUs (batch size 16, 32K context length, and 512-token KV budget). As shown in Table R11, despite variations in per-GPU routing workload, the end-to-end decoding latency differs by only 0.15%, demonstrating that the practical impact is negligible under the evaluated configuration.
Table R11. Comparison of routing workload and decode latency between HASA and fixed resolution.
W1b: End-to-End Efficiency
Thank you for your suggestion. In addition to the evaluation protocol in Appendix C.1, all measurements use 8 NVIDIA V100 GPUs, batch size 16, KV budget 512, and FP16 precision.
Regarding end-to-end inference acceleration, we further measure full-model decoding latency and throughput, including kernel-launch overhead, non-attention layers, synchronization, and multi-GPU communication (Table R12). HASA achieves 1.78–5.94× end-to-end decoding speedup over dense attention and approximately 1.19× over UNIQUE from 8K to 32K contexts, while maintaining nearly constant decoding throughput (499–512 tokens/s).
We will incorporate these additional results and evaluation details into the revised manuscript.
Table R12. End-to-end decoding latency and throughput comparison across different context lengths.
W2: Novelty of Head-Adaptive Sparsity
Thank you for the opportunity to clarify our contribution. The key novelty of HASA is the joint adaptation of feature resolution and sequence resolution based on a unified characterization of the query–key interaction structure. By coupling these two dimensions, HASA achieves not merely higher accuracy or lower latency in isolation, but an advancement of the accuracy–latency Pareto frontier.
This fundamentally differs from SEA and FlexPrefill, which primarily adapt the sparse support, pattern, or budget across heads. While these methods adapt the outcome of sparse selection, HASA adapts the computational process of selection itself. Specifically, the QK spectrum determines the compact response space, which in turn changes the distinguishability of tokens and thus governs the appropriate sequence granularity. The QK spectrum is therefore not merely an alternative adaptation signal, but a unified structural basis that intrinsically connects both resolutions. In principle, HASA's resolution adaptation is complementary to adaptive sparsity allocation.
We will further clarify our distinction from FlexPrefill in the existing Related Work discussion and include an explicit discussion of SEA.
W3: Additional Competitive Baselines
Thank you for suggesting these additional baselines. We have expanded our evaluation to include FlexPrefill and MInference, as well as DHSA, in Tables R13 and R14, covering three KV budgets (512, 1024, and 2048). Across all evaluated budgets and model backbones, HASA consistently achieves the strongest overall performance among the considered sparse-attention methods. Among all evaluated baselines, UNIQUE from our original evaluation remains the strongest overall.
We will incorporate these results into the revised manuscript.