acceptodds
Under review as a conference paper at ICLR 2027

The Shadow of a Few Keys: Local Low-Dimensional Structure in Long-Context Attention

Abstract

Efficient long-context attention rests on an implicit chain of assumptions: a low-dimensional query–key interaction should admit a small separable representation, that representation should give cheap and accurate attention, and accurate attention should preserve the model's answers. We test this chain link by link in two frozen language models, using local coordinates adapted to each block of queries. The first link fails: better local directions sharply reduce the error of an exact-exponential diagnostic, yet finite-state kernel summaries on the same coordinates recover only a small part of that gain. We call this discrepancy the geometry–realization gap and then locate it. A small set of high-mass keys, under one percent of the history, suffices to rebuild nearly all of the useful local geometry, and computing exactly on those same keys closes most of the gap, an effect that survives controls for projection error, attention sinks and the choice of support. We call this the few-key shadow. Two boundaries follow. A low-dimensional remainder often pays off under matched storage but never under matched arithmetic in our tests, and the support that reconstructs average geometry is far smaller than what the answer-determining queries need, so a frozen fresh-input evaluation still falls short of task preservation on three models. The results separate geometric compressibility, computational realization and task fidelity, and show that the first does not certify the other two.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention compression, long-context attention, low-rank approximation, kernel attention, sparse attention, KV cache, attention sinks, transformer inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SorROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the relationship between low-dimensional structure, approximation cost, and task performance in long-context attention. On frozen Llama and Qwen models, it compares exact exponential evaluation and kernel approximations using the same local coordinates. A basis built from a small set of high-attention keys captures most of the variance captured by the full-history rank-eight basis, and computing those keys exactly closes most of the gap between the kernel approximation and the exact-exponential diagnostic. The paper also compares allocations under equal storage and arithmetic budgets and evaluates task performance in full models, including Mistral.

Critical strengths
  1. The paper shows that the same small set of keys helps recover the local basis and reduce approximation error. I find this result convincing because the exact and approximate computations use the same local coordinates, and the projection and attention-sink controls check whether the improvement can be explained by those factors alone.

  2. I found the comparison in Figure 4 useful because the conclusion about the low-dimensional remainder changes when computation, rather than storage, is held fixed. Table 1 also shows that better local approximation does not necessarily preserve task performance. These results give a clearer picture of when the observed structure is useful.

Critical weaknesses
  1. The main structural analysis uses 20 test problems, and the same problems are used for both Llama and Qwen. I think the range of inputs is rather limited given that these structural observations are the main contribution of the paper. The larger full-model experiments are useful, but they mainly report task scores and do not directly show whether the basis-recovery and exact-set correction results also hold on new inputs. Repeating these checks on a separate set of inputs would make the main findings more convincing.
Questions for authors
  1. Could you check whether the basis-recovery and exact-set correction results remain similar on a small set of fresh inputs? The core structural analysis uses 20 test problems, while the larger fresh-input evaluation focuses on task scores. A direct check would help clarify how broadly the central observation holds.
3 comments
AuthorsOpenReview ↗

Response to Reviewer SorR

We deeply appreciate your thoughtful feedback. Yes: we repeated the basis-recovery and exact-set correction checks on new inputs and on a third model, and both results remain close to the submitted values.

New experiments (Appendix M). The same analyses, code, two heads per layer, windows and statistics as in the submission, with rules frozen before the first new input was captured, on:

  • 80 fresh inputs per model (16 per task), taken by index from the frozen fresh-input set of Section 6; their NIAH and VT inputs reuse the development filler text (the background text of these synthetic tasks), hence the next set;
  • 48 new synthetic inputs per model whose NIAH and VT filler comes from books never used in this work, a different segment for each input;
  • a third model, Mistral-7B-v0.3;
  • a separate run on every query head of every layer, on 20 fresh inputs per model.

Triples below are Llama-3.2-3B / Qwen3-8B-Base / Mistral-7B-v0.3.

Basis recovery. The rank-eight basis rebuilt from the top N/128N/128 keys (ranked by the attention weight of the block's first query) captures a median .977 / .983 / .985 of the variance captured by the optimal rank-eight basis on the fresh inputs and .980 / .983 / .986 on the unseen-filler inputs (submission .978 / .985). Its error under the exact-exponential diagnostic exceeds that of the full-history basis by .0020 / .0026 / .0018 (submission .0017 / .0021), and it outperforms the frozen global basis in every one of the 240 fresh and 144 unseen-filler inputs.

Exact-set correction. Computing those keys exactly, with P2P_2 elsewhere, closes .949 / .926 / .959 of the gap between the P2P_2 summary and the exact-exponential diagnostic on the fresh inputs and .953 / .943 / .968 on the unseen-filler inputs (submission .954 / .944). The controls keep their order: equal-sized uniform and next-ranked sets close at most .13, sink positions 0–3 alone .36 to .49, and the correction with the projection kept .80 to .90. Input by input, the share closed is at least .75 in 239 of the 240 fresh inputs (minimum .746) and in all 144 unseen-filler inputs (Table 50).

All heads. On every query head of the three models (five tasks), the window-median capture is .977 / .982 / .983 and the share of the gap closed, with all pairs pooled, .932 / .899 / .952 (Tables 52–53).

Section 4 of the revised PDF summarizes these results, Appendix M reports them per input, per task and per head, and Section 2 now states the limits of the original 20 test inputs next to their description. The general response lists the few places where a new result differs from the submission.

Thank you for the additional experiments and the detailed response. The basis-recovery and exact-set correction analyses on fresh inputs directly address my main concern about the limited input coverage of the original structural analysis. The checks with unseen filler text, a third model, and all query heads further strengthen the empirical support for the main findings.

I consider my concern addressed and will maintain my rating of 3 (Weak acceptance), with greater confidence in my positive assessment.

AuthorsOpenReview ↗

Follow-up to Reviewer SorR

We deeply appreciate your follow-up comment. The changes you requested are now part of the paper (Section 4 and Appendix M).

Reviewer_YQb5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates the assumptions behind efficient long-context attention, particularly whether low-dimensional representations of attention can be effectively realized through computationally efficient approximations. The authors conduct experiments on frozen Llama-3.2-3B and Qwen3-8B-Base models using block-local coordinates.

They first show that better local directions substantially reduce the approximation error when using exact exponential attention, but finite-state kernel summaries fail to recover most of this improvement, which the authors refer to as the geometry–realization gap. They further show that selecting only the highest-mass 1/1281/128 of keys recovers approximately 98% of the variance explained by the optimal rank-8 basis, and computing exact attention over these selected keys closes around 95% of the gap.

The authors also investigate combining exact computation with a low-dimensional approximation of the remaining keys, finding that the remainder is beneficial in 78 out of 112 configurations under matched storage budgets, but provides no improvement under matched arithmetic budgets. Finally, they evaluate the approach at the full-model level and show that multi-row selection recovers most of the task loss compared with using a single anchor. However, the resulting representation fails to satisfy the predefined five-point non-inferiority margin on unseen inputs across all three evaluated models, including Mistral-7B.

Critical strengths
  1. The diagnostics are well designed. They separate which directions are retained from how the exponential is realized, using the same coordinates. They also include careful controls for projected-only correction, sink-exact arms, uniform and mass-matched sets, and an all-head replication.
  2. Resources are matched carefully. Comparing at matched storage and at matched MACs reveals a reversal that "sparse vs. low-rank" framing hides, and the cost ledger and traffic model are explicit.
  3. The theory is correct and useful for interpretation. Propositions 1–5 hold, and their measured slack is reported (about 12–15×).
Critical weaknesses
  1. The question-row budget claim is circular. The abstract and Section 6 say the question rows need "about a quarter of the history." But the target ("full-model level") is the question-row error the end-to-end runs reach at m=N/4m = N/4. Appendix G.5 then finds the operator needs about N/4N/4 to reach it (ratios 1.00 and 0.97).

Please use a target set independently of N/4N/4, or report the full error-vs-mm curve and soften the claim.


  1. The headline comparison changes between sections. The introduction and Fig. 1(a) compare global and local bases at fixed r=8r = 8 (.0225 vs .0025 on Llama). Section 3 and Fig. 2(a) compare global r=1r = 1 with local r=8r = 8 (.0496 vs .0111), which changes rank and locality together. The fixed-rank comparison should be the primary one throughout.

  1. Practical significance is limited, and the fast settings are not evaluated.

    • The unfused implementation is 14–22× slower than dense attention.
    • The authors' own traffic model says a fused kernel would move 5.1–5.4× the bytes of a dense kernel.
    • Prefill stays quadratic.
    • The only configurations that ran faster than dense attention (L=128L = 128, m=1024m = 1024; App. G.4) have no measured accuracy. With 32-query blocks, m=1024m = 1024 already gives a relative error of .175.

  1. The statistical base is narrow. Operator-level conclusions rest on 20 items (four per task) and two fixed heads per layer. NIAH and VT largely share filler text (8-gram overlap of .83–1; App. F.2), and the fresh inputs reuse the development filler books. The intervals are therefore narrower than the number of sequences suggests. This should be stated next to the main claims, not only in the appendix.

  1. Some key comparisons use only one error measure. The gate-vs-P2P_2 and gate-vs-per-query-selection rankings are reported only in L2/Vmax⁡L_2/V_{\max} (Table 27 says so). The gaps are large, so the ranking will probably hold, but please confirm it under output-relative error.

  1. The paper is hard to follow. At 80 pages, many qualifications needed to read the main claims sit in the appendices. Notation is overloaded: CH\mathcal{C}_H, ε\varepsilon, MM, TT/TnT_n, and ww each have several meanings. A table linking each claim to its conditions and caveats would help a lot.
Questions for authors

see weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer YQb5

We deeply appreciate your detailed and insightful feedback. We agree with points 1, 2, 4 and 6 and revised the paper; for points 3 and 5 we performed the measurements you requested. Triples are Llama-3.2-3B / Qwen3-8B-Base / Mistral-7B-v0.3 (the third model is new).

1. Question-row budget. You are right: the level was the error reached at m=N/4m=N/4, so reaching it near N/4N/4 was not evidence. The statement is removed from Sections 1 and 6 and qualified in Appendix G.5. We now report the curve up to m=Nm=N and the budget for fixed targets from .20 to .03 (Section 6; Tables 59–60). With four anchor rows, the question-row relative error (‖ô − o‖/‖o‖) reaches .10 at .10N / .09N / .18N, .05 at .29N / .24N / .40N and .03 at .43N / .36N / .55N on fresh synthetic inputs (original inputs, Llama / Qwen: .10N / .09N, .29N / .23N, .43N / .36N). At m=N/4m=N/4 it is .058 / .048 / .082; a single anchor row needs .57N / .68N / .72N for .05. The curve falls smoothly; N/4N/4 is the operating point of our full-model runs, not a threshold. The revision also states that the .05 target of our earlier sweep had been set near the error at N/4N/4.

2. Headline comparison. Adopted. Section 3 and Fig. 2(a) now begin with the fixed-rank (r=8r=8) comparison of the introduction and Fig. 1(a): the local basis lowers the exact-exponential diagnostic error by .0225 and .0219 (Llama, Qwen) but the P2P_2 error by only .0025 and .0055. The comparison that changes rank and locality together is presented second. On the new inputs and on Mistral, the P2P_2 reduction is .08 to .27 of the diagnostic reduction (Tables 47–49).

3. Practical significance. We agree that the paper shows no speedup at a usable accuracy, and it claims none (Scope, Section 1). We measured the setting that ran faster than dense attention: Llama-3.2-3B, 128K tokens, 128-query blocks, one anchor row, m=1024m=1024. Its relative error is .197 [.195, .200] on blocks inside the filler text at the longest history and .267 on the question rows (.172 and .267 with 32-query blocks at the same mm), far above .05. At 16K, on the last 128 rows of the fresh inputs, the same setting gives .130 / .132 / .193, against .122 / .120 / .164 with 32-query blocks. Qwen3-8B-Base has 32,768 positions without YaRN, so its 128K timing has no accuracy measurement. We therefore claim no configuration that is both faster than dense attention and accurate; this is stated in Table 2 and Appendices G.4 and M.8.

4. Statistical base. Agreed. Section 2 now states, next to the test inputs, that they are four per task and shared by both models, that NIAH and VT share filler text, and that the intervals describe these items rather than a population. We also widened the base (general response). It includes 48 new synthetic inputs per model whose NIAH and VT filler comes from three books never used before, a different segment for each input, which removes the sharing for these tasks (share of the gap closed by the exact set .953 / .943 / .968, variance capture .980 / .983 / .986). Heads: the restricted-basis comparison already covered all 1,824 head–layer pairs (Appendix D.3). On fresh inputs we now also measure capture and closure on every head (five tasks): window-median capture .977 / .982 / .983, closure with all pairs pooled .932 / .899 / .952, and 68% / 66% / 76% of pairs within .003 of the full-history basis and below the global basis (73% / 71% on the original inputs; Tables 52–53).

5. One error measure. Confirmed. At m=N/128m=N/128 we recomputed both rankings under five measures (L2/Vmax; output-relative error as a row mean, pooled, and as a window median; error over the RMS value norm), on the original, fresh and unseen-filler inputs of three models (Table 57). The gate outperforms the r=8r=8 P2P_2 summary, and per-query selection outperforms the gate at the same mm and remainder, in all 165 comparisons; every interval excludes zero. Two secondary statements are not reproduced (Appendix M.6, Table 58). The budget at which the gate matches per-query selection (N/16N/16 to N/8N/8 on the test items) varies with the measure and the model, and on Mistral the gate at N/8N/8 is still worse under L2/Vmax. A comparison that mixes two remainder treatments changes sign with the measure on Llama.

6. Readability. Agreed. New Table 2 in the main text links each main claim to its evidence, conditions and caveats. Table 63 lists each symbol you named with its meanings by section; in the main text the capture share is renamed and ε is no longer used. The qualifications the main claims need (inputs, heads, filler, the question-row budget) are now in Sections 2, 4 and 6.

Thank you for the thorough response and for answering all my concerns. The following points remain

3. Practical significance — request answered; limitation remains. Appendix M.8 measures the configuration that ran faster than dense attention and states that no tested configuration is both faster and accurate. I see this as an inherent limit of a diagnostic paper rather than a flaw, but it constrains the paper's significance.

6. Readability. Table 2 and the notation table address my main concerns. The paper has grown to 98 pages, so I'm keeping my presentation score, and many edits seem AI-generated, which makes the presentation and readability even harder; a few examples:

  • The caption of Table 10 still refers to arms "requested in the second-round review"; Appendix A.1 ("Section 5 no longer quotes it") and Appendix K.1 ("legacy analysis variable") also refer to earlier versions.
  • Section 6 calls the progressive-selector results "development comparisons rather than non-inferiority tests", while Appendix I reports a registered non-inferiority test of that arm, which was not met. Please make the main text consistent.
  • In Table 4 (development split), the Llama r=8r = 8 local-exact entry (.0205, 82.0%) is identical to the test-split value in Table 5, while all other entries differ between splits. Please confirm it is correct.

My soundness concerns are resolved, and I am raising Soundness to 3. The central findings now hold on new inputs, new filler text, a third model, and all heads, and the authors withdrew the claims that did not hold. The remaining limits are the absence of a usable speedup and the modest conceptual step over prior sparse-plus-low-rank analyses, as Reviewer etLQ also notes. I am retaining my overall rating.

AuthorsOpenReview ↗

Follow-up to Reviewer YQb5

We deeply appreciate your careful reading of the revision and your updated assessment. We addressed the three items you list in an updated PDF; changes are marked in red, as before.

References to earlier versions. Corrected. The caption of Table 10 no longer contains that sentence. Appendix A.1 now reads "Section 5 does not quote it", and Appendix K.1 now reads "The result files name this endpoint E4, but it contains five or three tasks and is distinct from the four-task development endpoint." We searched the whole paper for wording of this kind and reworded four further places (Section 6 and Appendices G.4, L.1 and M.9). None of these edits changes a number or a claim.

Progressive selector (Section 6 and Appendix I). You are right, and Appendix I gives the correct account. The comparison of the progressive selector with exact attention was a prespecified non-inferiority test with a five-point margin, and the margin was not met on either model: −5.1-5.1 [−11.4-11.4, +1.0+1.0] and −8.0-8.0 [−12.5-12.5, −3.7-3.7] points on Llama and Qwen (pni=.51p_{\mathrm{ni}} = .51 and .91.91; row "4 rows, causal" of Table 1(a)). Section 6 now states this result in place of the sentence you quote.

Dimension sweep on the development split (Table 5 in the revised PDF, Table 4 in the submission). The entry is correct. This table and the test-split table that follows it use disjoint inputs (four development and four test sequences per task), and the Llama r=8r=8 diagnostic values agree only after rounding: .02050.02050 and 82.01%82.01\% on the development split, .02055.02055 and 82.00%82.00\% on the test split. We recomputed the development values from the per-window results (2,240 windows), and the caption now notes this.

Point 3 and length. We agree with your reading of point 3: this is a diagnostic study, and it shows no speedup at a usable accuracy (Scope in Section 1, Table 2 and Section 8). On length, the added pages are the tenth page of the main text and Appendix M (17 pages), which holds the analyses requested in the reviews; Table 2 points from each main claim to the appendix that supports it. We will continue to edit the appendices for readability in the final version.

Reviewer_etLQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether local low-dimensional query–key structure enables compact and accurate long-context attention. It constructs anchor-weighted local coordinates and compares exact exponentiation with finite-state kernel approximations on the same projected interactions. In two frozen language models, a small high-mass key set reconstructs most of the variance captured by the optimal rank-eight basis. Computing those keys exactly also removes most of the gap between a quadratic summary and the projected exponential diagnostic. Resource-matched comparisons distinguish storage savings from arithmetic savings. Full-model evaluations, including fresh inputs on a third model, show that the tested representation does not establish task preservation.

Critical strengths
  • S1: The controlled separation of geometry and kernel realization provides a useful empirical insight. The top 1/128 of history keys recover about 98% of the variance captured by the optimal rank-eight basis. Exact correction on the same keys closes about 95% of the quadratic-to-diagnostic error gap. Keeping the projection fixed still closes about 89% and 82%, which shows that restoring omitted directions does not explain the entire improvement. Support, centering, and sink controls make this more informative than simply observing sparse attention.
  • S2: The study carefully separates different notions of successful compression. A richer remainder wins in 78 of 112 configurations at matched storage but none at matched counted arithmetic. The authors also retain negative task results, test a causal selector, and explicitly distinguish analytical costs from measured runtime. These checks provide useful guidance on why geometric compressibility, economical computation, and preserved model behavior must be evaluated separately.
Critical weaknesses
  • W1: Limited contribution. Scatterbrain already analyzes the complementarity of sparse and low-rank attention and uses exact corrections to reduce kernel approximation error. LoLA studies finite-rank kernel limitations and selective exact retention, and shows that selecting keys by kernel error need not improve retrieval. The main increment here is the controlled observation that the same small key set captures local geometry and explains much of the kernel approximation gap. This is informative, but the conceptual advance over these existing analyses is limited. The extensive diagnostics do not yet show how this connection yields a new prediction or broadly useful design principle.
  • W2: Limited independent validation of the central findings. The core geometric and kernel-approximation analyses use 20 test inputs per model, only four per task, with the same inputs shared across the two architectures. The all-head study adds heads but no independent inputs. Resampling and sensitivity checks help assess robustness within this original set. However, the fresh-input evaluation measures downstream task performance without repeating the central geometry and kernel-approximation analyses. It therefore remains unclear whether the same small key sets consistently recover local geometry and close the approximation gap on new inputs.

References:

Questions for authors
  • Q1: What prediction or design implication follows from the shared key support that existing sparse-correction and finite-rank analyses do not already provide? Evidence that this connection predicts when a compact approximation will succeed on independent inputs would strengthen the contribution.
  • Q2: Can the restricted-basis and projected-only correction results be checked on an independent subset of the fresh inputs, ideally including Mistral? Reporting input-level effects and variation across tasks would help establish whether the central findings generalize beyond the original 20 test inputs.
3 comments
AuthorsOpenReview ↗

Response to Reviewer etLQ

We deeply appreciate your insightful feedback. In brief: the restricted-basis and exact-set results, including the projected-only correction, hold on new inputs and on Mistral-7B-v0.3, input by input and task by task (Q2); and the shared support gives a prediction, tested on such inputs, and design rules that the two cited papers do not report (Q1).

Terms: the exact set is the top N/128N/128 keys by the attention weight of the block's first query (the anchor). The gap is the error of the P2P_2 summary minus that of the exact-exponential diagnostic on the same coordinates; closure is the share of the gap removed by computing the exact set exactly. The restricted basis is the rank-8 basis rebuilt from the exact set; capture is the share of the optimal rank-8 variance it captures. Triples are Llama-3.2-3B / Qwen3-8B-Base / Mistral-7B-v0.3.

W2 / Q2: independent validation

We used the same heads, windows, code and statistics, with rules frozen before the first new capture, on 80 fresh inputs per model and on 48 new synthetic inputs per model whose NIAH and VT filler (the background text of these tasks) comes from books never used before (general response; Appendix M). On the fresh inputs:

  • Closure: .949 / .926 / .959 (submission .954 / .944). Projected-only (only the exponential made exact on the set): .895 / .810 / .870 (submission .890 / .815).
  • Restricted basis: median capture .977 / .983 / .985; error above the full-history basis by .0020 / .0026 / .0018 (submission .0017 / .0021) and below the global basis by .0239 / .0211 / .0156.

Input level (Table 50): closure is at least .75 in 80, 79 and 80 of the 80 fresh inputs (minimum .882 / .746 / .832); the restricted basis outperforms the global basis in every input.

Tasks (Table 51): closure is lowest on FWE (.902 / .807 / .915) and highest on HotpotQA (1.009 / 1.032 / 1.028; above 1, the corrected arm is more accurate than the diagnostic). Median capture is .970 to .988 on every task and model.

All heads, now on new inputs (five tasks; Tables 52–53): window-median capture .977 / .982 / .983 and, with all pairs pooled, closure .932 / .899 / .952.

W1 / Q1: contribution and prediction

We agree that both ingredients have precedents; revised Section 7 states that our exact-set intervention has the form of a sparse correction to a kernel estimate. Scatterbrain corrects a data-independent random-feature estimate at positions found by locality-sensitive hashing. LoLA combines a sliding window, a linear-attention state and an exact cache of the key–value pairs that this state recalls worst, scored without the query. Neither reports which support rebuilds a data-adapted low-dimensional basis, whether the kernel error lies on that same support under projected-only, sink and support controls, or how allocations rank under matched storage versus matched arithmetic.

What follows, with evidence on new inputs:

  1. The anchor row predicts where the few-key basis suffices, before any approximation. A threshold on the anchor's attention mass on its top N/128N/128 keys (at least .878), fitted on the original inputs only, was applied unchanged to all query heads on fresh inputs from the tasks without shared filler, and to Mistral, which took no part in the fit. Of the head–layer pairs it selects, a share of .890 / .809 / .927 have a restricted basis within .003 of the full-history basis, against base rates of .671 / .645 / .760 (AUROC .826 / .738 / .790; Table 55). The relation is graded, not an accurate classifier.
  2. The pattern over heads is stable. Per head–layer pair, original and fresh inputs give rank correlations of .82 and .82 for the restricted-minus-full difference, .94 and .94 for the capture and .80 and .84 for the closure (Llama, Qwen; all heads, tasks without shared filler; Table 54).
  3. Make high-mass keys exact before enlarging the kernel. In the submission the exact set closes .954 / .944 of the gap, more than a degree-eight polynomial with 12,870 states (.873 / .746); on the new inputs it closes .93 to .97, while equal-sized uniform or next-ranked sets close at most .13 and sink positions 0–3 alone .36 to .49.
  4. Whether a low-dimensional remainder helps depends on the constrained resource. It is better in 78 of 112 configurations at matched storage and in none at matched arithmetic. Scatterbrain compares under one parameter budget and LoLA at a matched cache size, so neither reports this reversal.
  5. Geometric sufficiency does not certify the answer. The support that rebuilds the basis is far smaller than what the question rows (the rows of the prompt's question tokens) need (Appendix M.7), so selection should be scored on the rows that determine the answer. This agrees with LoLA's observation that selecting keys by kernel error need not improve retrieval.

Items 1 and 2 answer Q1 on new inputs; the claims behind items 3 to 5 are listed with their conditions in Table 2.

Thank you for the thorough response. The additional analyses on independent inputs and Mistral address my concern about the validation of the central findings. The prediction using a fixed threshold also clarifies the contribution beyond the original observations. These results strengthen my positive assessment, and I would like to keep my current score, which is already positive.

AuthorsOpenReview ↗

Follow-up to Reviewer etLQ

We deeply appreciate your follow-up and your positive assessment. Your two questions led to the analyses on independent inputs and on Mistral-7B-v0.3 and to the fixed-threshold prediction, which are now part of the paper (Appendix M).

Comments on the paper

1 comment
AuthorsOpenReview ↗

New results and revised PDF (to all reviewers and the AC)

We deeply appreciate the insightful feedback from the reviewers. All three reviewers asked whether the operator-level findings, which are based on 20 test inputs shared by two models, hold on new inputs. We repeated the analyses on new inputs and on a third model (Mistral-7B). The central quantities remain close to the submitted values. The revised PDF marks changed text in red (citations and cross-references are blue links, as in the submission) and reports the new results in Appendix M.

New experiments. The same analyses, code, windows and statistics as in the submission (two query heads per layer), with input selection and decision rules frozen before the first new capture. A control re-capture of an original test input reproduced the archived tensors bit for bit.

  • Fresh inputs: 80 per model (16 per task), chosen by index from the frozen fresh-input set of Section 6 before any operator-level analysis of them. Their NIAH and VT inputs reuse the development filler text (the background text of these synthetic tasks), so Appendix M also reports the other 48 inputs alone.
  • Unseen-filler inputs: 48 new synthetic inputs per model. The 32 NIAH and VT inputs take their filler from three books never used before, a different segment for each input (8-gram overlap with earlier inputs at most .02); the 16 FWE inputs contain no filler.
  • Third model: Mistral-7B-v0.3, not part of the original analysis (per-head statistics fitted on 10 separate fresh inputs).
  • All query heads: a separate run on every head of every layer, 20 fresh inputs per model.

Main result (m=N/128m=N/128 exact keys, rank r=8r=8, L2/Vmax error; intervals in Tables 47–49):

ModelInputsGap closed by exact setSame, projection keptVariance captureRestricted − full
Llama-3.2-3Bsubmission (20).954.890.978.0017
fresh (80).949.895.977.0020
unseen filler (48).953.894.980.0019
Qwen3-8B-Basesubmission (20).944.815.985.0021
fresh (80).926.810.983.0026
unseen filler (48).943.802.983.0021
Mistral-7B-v0.3fresh (80).959.870.985.0018
unseen filler (48).968.878.986.0016

Gap: error of the P2P_2 summary minus that of the exact-exponential diagnostic on the same eight local directions. Exact set: the N/128N/128 keys with the largest attention weight from the block's first query, computed exactly, with P2P_2 elsewhere. Projection kept: only the exponential is made exact on them. Capture: median share of the optimal rank-eight variance captured by the basis rebuilt from them (the restricted basis). Last column: its error minus that of the full-history basis.

On every model and new input set, equal-sized uniform or next-ranked sets close at most .13 of the gap and positions 0–3 (the attention sinks) alone .36 to .49. The exact set closes at least .75 of the gap in 239 of the 240 fresh inputs (minimum .746) and in all 144 unseen-filler inputs, and the restricted basis outperforms the frozen global basis in every input (Table 50). On all query heads (five tasks; Llama / Qwen / Mistral), the window-median capture is .977 / .982 / .983 and the share of the gap closed, with all pairs pooled, is .932 / .899 / .952; 68% / 66% / 76% of head–layer pairs are within .003 of the full-history basis and below the global basis, against 73% / 71% on the original inputs (Tables 52–53).

What differs. The decision rules frozen for the original analysis give the original verdicts on all three models, except one descriptive reading: in 4 of 18 cells the sink positions account for slightly more than half of the exact set's error reduction (.50 to .54) rather than at most half (Table 56). Two secondary statements about the anchor gate versus per-query selection are not reproduced (response to YQb5, point 5).

Claims. The main claims are unchanged. One statement is withdrawn: that the question rows need "about a quarter of the history" (response to YQb5, point 1).

Revision. (1) Appendix M: the results above, per input, per task and per head; summary in Section 4. (2) Section 3 and Fig. 2(a) begin with the fixed-rank comparison. (3) Section 6 and Appendix M.7: question-row error against the budget up to m=Nm=N, with budgets for fixed targets. (4) The configuration that ran faster than dense attention is measured and is not accurate (Appendix M.8). (5) The rankings of Section 5 hold under five error measures (Appendix M.6). (6) New Table 2: each main claim with its evidence and conditions; Table 63: notation; Section 2: limits of the original test inputs. (7) Section 7 discusses Scatterbrain and LoLA directly. The main text remains within 10 pages; appendix table numbers shift by one (Table 27 of the submission is Table 28).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.