- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the relationship between low-dimensional structure, approximation cost, and task performance in long-context attention. On frozen Llama and Qwen models, it compares exact exponential evaluation and kernel approximations using the same local coordinates. A basis built from a small set of high-attention keys captures most of the variance captured by the full-history rank-eight basis, and computing those keys exactly closes most of the gap between the kernel approximation and the exact-exponential diagnostic. The paper also compares allocations under equal storage and arithmetic budgets and evaluates task performance in full models, including Mistral.
Critical strengths
-
The paper shows that the same small set of keys helps recover the local basis and reduce approximation error. I find this result convincing because the exact and approximate computations use the same local coordinates, and the projection and attention-sink controls check whether the improvement can be explained by those factors alone.
-
I found the comparison in Figure 4 useful because the conclusion about the low-dimensional remainder changes when computation, rather than storage, is held fixed. Table 1 also shows that better local approximation does not necessarily preserve task performance. These results give a clearer picture of when the observed structure is useful.
Critical weaknesses
- The main structural analysis uses 20 test problems, and the same problems are used for both Llama and Qwen. I think the range of inputs is rather limited given that these structural observations are the main contribution of the paper. The larger full-model experiments are useful, but they mainly report task scores and do not directly show whether the basis-recovery and exact-set correction results also hold on new inputs. Repeating these checks on a separate set of inputs would make the main findings more convincing.
Questions for authors
- Could you check whether the basis-recovery and exact-set correction results remain similar on a small set of fresh inputs? The core structural analysis uses 20 test problems, while the larger fresh-input evaluation focuses on task scores. A direct check would help clarify how broadly the central observation holds.
3 comments
Thank you for the additional experiments and the detailed response. The basis-recovery and exact-set correction analyses on fresh inputs directly address my main concern about the limited input coverage of the original structural analysis. The checks with unseen filler text, a third model, and all query heads further strengthen the empirical support for the main findings.
I consider my concern addressed and will maintain my rating of 3 (Weak acceptance), with greater confidence in my positive assessment.
Follow-up to Reviewer SorR
We deeply appreciate your follow-up comment. The changes you requested are now part of the paper (Section 4 and Appendix M).
Response to Reviewer SorR
We deeply appreciate your thoughtful feedback. Yes: we repeated the basis-recovery and exact-set correction checks on new inputs and on a third model, and both results remain close to the submitted values.
New experiments (Appendix M). The same analyses, code, two heads per layer, windows and statistics as in the submission, with rules frozen before the first new input was captured, on:
Triples below are Llama-3.2-3B / Qwen3-8B-Base / Mistral-7B-v0.3.
Basis recovery. The rank-eight basis rebuilt from the top N/128 keys (ranked by the attention weight of the block's first query) captures a median .977 / .983 / .985 of the variance captured by the optimal rank-eight basis on the fresh inputs and .980 / .983 / .986 on the unseen-filler inputs (submission .978 / .985). Its error under the exact-exponential diagnostic exceeds that of the full-history basis by .0020 / .0026 / .0018 (submission .0017 / .0021), and it outperforms the frozen global basis in every one of the 240 fresh and 144 unseen-filler inputs.
Exact-set correction. Computing those keys exactly, with P2 elsewhere, closes .949 / .926 / .959 of the gap between the P2 summary and the exact-exponential diagnostic on the fresh inputs and .953 / .943 / .968 on the unseen-filler inputs (submission .954 / .944). The controls keep their order: equal-sized uniform and next-ranked sets close at most .13, sink positions 0–3 alone .36 to .49, and the correction with the projection kept .80 to .90. Input by input, the share closed is at least .75 in 239 of the 240 fresh inputs (minimum .746) and in all 144 unseen-filler inputs (Table 50).
All heads. On every query head of the three models (five tasks), the window-median capture is .977 / .982 / .983 and the share of the gap closed, with all pairs pooled, .932 / .899 / .952 (Tables 52–53).
Section 4 of the revised PDF summarizes these results, Appendix M reports them per input, per task and per head, and Section 2 now states the limits of the original 20 test inputs next to their description. The general response lists the few places where a new result differs from the submission.