- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes adding a pre-scoring step before HyperAttention, using k-means, k-median, or leverage scores to select a subset of important keys before attention is computed. The authors provide a planted-model analysis showing that clustering can recover the same heavy keys as leverage scores, and evaluate the method on long-context perplexity with ChatGLM2/3-6B-32k and on pretrained Vision Transformers on ImageNet.
Critical strengths
The Idea of the paper sounds exciting and practical, and the paper is easy to follow.
Critical weaknesses
-
The headline claim that pre-scoring reduces perplexity from 12 to 8.3 is not supported. Appendix A.1 states that the 8.3 result is obtained with pre-scoring turned off and comes from the min-seq-len setting. In Tables 2–4, plain HyperAttention without residual sampling already reaches 10.41, which matches or beats nearly every pre-scored configuration.
-
The GLM2 results come from code that Appendix F says had three bugs, yet they are still used in the abstract and main text.
-
GLM3 results are shown only as a plot, with no table or runtime, and the gains are modest.
-
In the forward-only test (Fig. 1), HyperAttention reaches about 13.5× over FlashAttention at 131K tokens, while the pre-scored variants reach only about 3–5×. Pre-scoring therefore makes HyperAttention roughly 3× slower, yet the text describes it as giving “a mild acceleration.”
-
The ViT comparison is not apples to apples. The K-means results use pretrained ViTs, while the LevAttention numbers come from models trained from scratch with a lower baseline. Relative to their own baselines, both keep about 98% accuracy.
-
The experimental scope is narrow. Important related work on clustering-based attention (e.g., Routing Transformer, clustered attention) is neither discussed nor compared against, and there are no LongBench task scores. The abstract also mentions a 131K context while both models are 32K.
Minor: the presentation needs polish (Figures and tables are not in a good shape), and the header still says ICLR 2026.
I do not think the paper is ready for acceptance in its current form, and I recommend strong rejection. The central empirical claims are not adequately supported, particularly the attribution of the reported gains to pre scoring.
Questions for authors
Please see Weaknesses section.