- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks how linear-attention models keep in-context facts in their recurrent state, using donor/recipient interventions in GLA-340M, GDN-340M, and GDN-1.3B. It reports that one development-selected "hub" head per model writes content-specific information and mediates the later read, that four facts stay selectively editable despite cross-fact coupling, and that adding facts hurts recall more than adding filler. In hybrid models, swapping the full-attention KV cache transfers nearly all of the donor effect, while swapping the recurrent state transfers almost none.
Critical strengths
-
The hybrid result is clean and well replicated. The KV-versus-recurrent factorial holds across a near-matched GDN pair, two Qwen3.5 sizes, PG19 filler, and RULER. It shows which substrate carries the recalled fact at query time.
-
Selection is kept separate from confirmation. In both 340M models, the frozen development-selected head ranks first on held-out sources (App. C.1), and control heads test content specificity. This guards against selection circularity, so I accept the concentration finding, if not yet its mechanistic reading.
Critical weaknesses
-
Two mechanism claims in the abstract lack tests that could fail. For the read pathway, the paper specifies single-token queries and answers only for the development task (App. B). In that setting, a head's pre-query state reaches the logits only through its query-time output (Eq. 2). Restoring the recipient's output, as in Fig. 4, therefore cancels any state swap by construction. The output patch, state-swap screens, and logit lens also confound the read with stored content. The coupling result has a similar problem. The part of fact A's edit that points along fact B's edit moves fact B whenever the two edits overlap (Fig. 5c). Because the two parts add nearly linearly and the full edit moves B less than that part alone, the remaining orthogonal part has to push B the other way. Any nonzero overlap thus produces the opposing effects the paper reads as coupling. Norm-matched Gaussian controls are a weak null here, since random directions are nearly orthogonal to any fixed direction in a high-dimensional state. As reported, neither test rules out a simpler reading: stored content alone, or ordinary non-orthogonal storage. Please add a test of the read that could fail, such as swapping only the query vector between bindings. For coupling, please compare the cosine between the two edits with a null from unrelated contexts and, if it does not exceed that null, restate the abstract claim as non-orthogonal storage with small cross-talk.
-
The closest mechanistic prior work is not cited. Bick et al. (ICML 2025) anticipate the concentration finding and the hybrid result's direction: they find a Gather-and-Aggregate mechanism concentrated in a few heads of Transformers and Mamba-based SSMs, with hybrids handing the aggregation role to attention. The paper does cite Schlag et al. (2021), but it does not engage their capacity analysis, which predicts key-overlap interference and decline under load. Without a comparison to these works, I cannot tell which of the concentration, hybrid, load, and overlap findings are new; please position the paper against both and say what it adds.
Questions for authors
-
Every filler token is also a write, so the load-versus-filler comparison contrasts template-matched fact writes with prose writes, not competition with passive loss. The ranking also depends on the chosen ranges: GLA loses more recall from 128 to 1024 filler tokens than from one to four facts (Fig. 7a-b). At fixed length, how does recall compare across same-template bindings, different-template bindings, and token-matched filler?
-
Do the four candidates include values of the other stored facts? If they do, one-fact and multi-fact recall are different tasks.
-
Fig. 7c shows only GDN-340M-colored markers, and neither the caption nor the text names the model. Was the key-overlap test also run on GLA-340M?
-
Why does GLA's clean four-fact recall differ across App. D.1, Fig. 7a, and Fig. 8a (about 92%, 96%, and 76%)?
-
Zero-ablating the hub's write drives GLA recall below chance (Fig. 3a). Under mean or resample ablation, does blocking the hub alone still hurt more than blocking all heads (Fig. 3c)?
-
Which two Qwen3.5 conditions were excluded (p.7 l.344-345), and when was the exclusion rule fixed?
-
Near-zero recovery from the recurrent swap also fits a recurrent state that still holds the fact but loses to the recipient's KV entries. The paper mostly limits its claim to the direct runtime carrier (App. E.1), but the Sec. 4.1 title ("The memory substrate shifts") invites the reading that the recurrent state no longer stores the fact. Masking attention to the fact span in every full-attention layer would test recall from the recurrent state alone.
-
Despite the Limitations caveat, the title and Conclusion speak about linear attention in general. The full analysis covers three checkpoints from one release, none above 1.5B parameters, and single-component dominance does not hold in GLA-7B (App. E.2). Please scope both to the tested setting.
7 comments
W2: Distinction from Prior Mechanistic Studies
We thank the reviewer for identifying these important connections.
Bick et al. (ICML 2025) identify Gather-and-Aggregate mechanisms in Transformers and Mamba-based SSMs, showing concentrated retrieval contributions and attention-dominated aggregation in hybrid architectures. Schlag et al. (ICML 2021) establish the fast-weight interpretation of linear attention and analyze finite-memory limitations, motivating delta-rule updates to reduce interference. We acknowledge that head concentration, hybrid attention dominance, and finite-capacity interference are not novel findings in isolation.
Our contribution is to causally connect the writing, retention, and query-dependent retrieval of individual in-context associations in pretrained linear-attention models. We link these stages through fact-time state-update interventions, pre-query state transplantation, and fixed-memory query-vector interventions. This runtime memory-lifecycle analysis complements prior work on retrieval-critical components and fast-weight memory. We further characterize factual competition, retention distance, and write-key geometry, interpreting cross-fact effects as non-orthogonal storage with measurable interference rather than a distinct coupling mechanism.
Q1: Distinguishing Memory Load from Intervening Context
We agree that filler tokens also update recurrent memory. We therefore conducted a length- and position-matched study comparing same-template bindings, different-template bindings, and prose filler while controlling the target association, answer candidates, and retention distance.
At a 512-token target-to-query distance, four total bindings (one target and three same-template competitors) reduced correct-answer margins relative to matched filler by 1.55, 2.39, and 2.24 logits in GLA-340M, GDN-340M, and GDN-1.3B, respectively. Different-template bindings also reduced margins, although differences in competitor retrievability prevent attributing this effect specifically to template similarity. We confirmed that prose filler induces nonzero recurrent-state updates. Thus, these results distinguish competing factual writes from other active memory updates rather than passive decay.
We also extended the original Fig. 7(b) experiment without changing its protocol. GLA-340M recall declined from 89.06% at 1K to 37.50% at 3,950 filler tokens. Across 128–3,950 tokens, recall declined by 60.16 pp, compared with 38.28 pp when increasing memory load from 1 to 16 facts. In contrast, GDN-340M recall declined by only 3.91 pp over 128–1,902 filler tokens, compared with 31.25 pp under increased load. These comparisons reflect separate experimental sweeps rather than matched causal estimates.
Together, these results show that neither memory load nor retention distance universally dominates recall degradation. Their relative effects depend on architecture, evaluated range, and metric. Notably, GLA's accuracy-based reversal does not extend to correct-answer margins, where increased load produces a larger decline (4.79 versus 3.97 logits).
Q2: Memory Load, Candidate Construction, and Associative Competition
We thank the reviewer for highlighting this distinction. The original evaluation uses four fixed answer candidates (red, blue, green, and yellow), but their roles change with memory load. At K=1, incorrect candidates are unbound distractors, at K=4, they correspond to other stored bindings, and at K=8/16, values are reused across bindings. Thus, increasing memory load also changes associative competition, preventing the accuracy decline from being attributed solely to limited memory capacity.
To distinguish competing associations from candidate exposure, we conducted a frequency- and position-matched unbound-foil control. Alternative answer values appeared either without competing key associations or bound to other keys, while their frequencies and positions, the target association, answer candidates, and retention distance remained fixed.
At a 512-token distance, competing-binding minus unbound-foil margin differences were −0.647 logits for GLA-340M (95% CI [−0.861, −0.436]), −1.390 logits for GDN-340M ([−1.534, −1.245]), and −0.571 logits for GDN-1.3B ([−0.715, −0.425]). These results show interference from competing factual associations beyond alternative-value exposure alone.
Nevertheless, this control neither equates the original K=1 and K=4 task difficulty nor isolates intrinsic memory capacity. We therefore interpret memory-load effects as load-dependent associative interference, potentially arising during storage and/or retrieval, rather than a direct measurement of intrinsic memory-capacity limits.
Q3: Extending Key-Overlap Analysis to GLA
We thank the reviewer for pointing out this omission. The original Figure 7(c) experiment was conducted only on GDN-340M, which was not explicitly stated in the caption.
To examine whether the key-overlap effects extend to GLA, we conducted additional experiments on GLA-340M using its frozen selected head (L21H3), following the original K=4 and K=16 settings. Specifically, we aligned or orthogonalized a later distractor's write key relative to the target-fact write key while preserving its norm, using the same 64 sources per memory-load condition.
At K=4, alignment changed the correct-answer margin by −0.222 logits (95% CI [−0.273, −0.173]), while orthogonalization changed it by +0.373 logits (95% CI [0.278, 0.476]). These results provide causal evidence that write-key geometry affects factual recall in GLA as well as GDN. However, because GLA uses outer-product updates whereas GDN employs key-dependent delta corrections, we interpret the findings as shared sensitivity to key geometry rather than evidence of identical interference mechanisms.
Q4: Discrepancies in Clean Recall Across Evaluation Protocols
We thank the reviewer for pointing out this inconsistency. The three reported GLA-340M clean recall accuracies were obtained using different evaluation protocols, which were not sufficiently distinguished in the original manuscript.
Clean recall across evaluation protocols.
| Experiment | Clean recall (%) | Evaluation unit | Full input length |
|---|---|---|---|
| Appendix D.1 | 91.80 | 470/512 queried bindings | 562 tokens |
| Figure 7(a) | 96.09 | 123/128 selected targets | 658 tokens |
| Figure 8(a) | 76.17 | 390/512 queried bindings | 898 tokens |
Although all three experiments use the same checkpoint, they differ in fact-slot layouts (4/16/32), presentation counts (4/3/4), source windows, fact ordering, value mappings, and evaluation aggregation. In particular, Figure 7(a) evaluates one selected target per source, whereas Appendix D.1 and Figure 8(a) aggregate recall across queried bindings.
We verified the reported accuracies against the saved clean logits and reproduced the original measurements, finding no numerical or plotting errors. Thus, the differences reflect distinct evaluation protocols rather than inconsistent measurements under identical conditions. Comparisons within each experiment use consistent settings, but the baselines should not be directly compared, and the accuracy gaps cannot be attributed to any single experimental factor.
Q5: Robustness of Hub-Only versus All-Head Write Ablation
We thank the reviewer for this suggestion. We repeated the hub-only versus all-head write-ablation comparison using zero, mean, and donor-resampled replacements on 128 held-out GLA-340M sources.
GLA-340M recall under different write-ablation replacements.
| Replacement | Hub-only recall (%) | All-head recall (%) | Hub − All (pp) |
|---|---|---|---|
| Zero | 14.1 | 22.7 | −8.6 |
| Mean | 14.1 | 22.7 | −8.6 |
| Resample | 17.5 | 24.1 | −6.6 |
Hub-only blocking produces lower recall than all-head blocking under all three replacement strategies. For resampling, the accuracy difference is not statistically conclusive (95% CI [−13.75, +0.31] pp), although the paired correct-answer margin difference remains significant (−0.869 logits, 95% CI [−1.112, −0.640]).
Although mean replacements have approximately 61% of the native update norm, they are behaviorally similar to zero ablation: only 2.34% of hub-only predictions changed, all between incorrect answers, leaving accuracy unchanged. Thus, mean ablation provides limited independent robustness evidence. In contrast, donor-resampled replacements have approximately 103% of the native update norm and change 23.28% of predictions, providing stronger evidence beyond zero ablation.
Together, these results support nonadditive causal effects among recurrent components that persist under a behaviorally distinct replacement strategy, while the uncertainty in resampled accuracy differences prevents stronger conclusions about a unique circuit or compensatory mechanism.
Q6: Excluded Qwen3.5 Conditions and Exclusion Criteria
We initially considered sequential state transition and symbolic accumulation to evaluate memory behavior on more complex tasks. However, preliminary evaluations showed that Qwen3.5-4B achieved only 25% clean accuracy on both tasks, equivalent to chance in our four-choice setting. Moreover, none of the cache interventions improved accuracy beyond 25%.
We therefore excluded these tasks from the primary memory-attribution analysis because the model could not reliably perform the underlying tasks, making causal attribution uninformative. This decision was based on baseline task competence rather than favorable attribution outcomes.
Note. W and Q denote Critical Weaknesses and Questions for Authors, respectively, following the original review's numbering.
We thank the reviewer for their constructive feedback. We conducted fixed-memory query-vector interventions demonstrating binding-sensitive readout beyond output restoration. Empirical null comparisons did not consistently support a distinct cross-fact coupling mechanism. Additional experiments address write-ablation robustness, factual-binding competition, GLA key overlap, hybrid KV masking, recall-baseline discrepancies, and Qwen3.5 exclusion criteria.
Planned Manuscript Revisions: We will clarify output restoration in Fig. 4 and add query-vector controls to the Appendix. We will refine the coupling claims in the Abstract and Fig. 5(c) using empirical-null and interference analyses. We will expand the Fig. 3 ablation analysis, add candidate-exposure and GLA key-overlap controls to the Fig. 7 analysis, and clarify the protocols behind Appendix D.1 and Figs. 7(a)/8(a). We will also revise Section 4.1 and Appendix E.1 to clarify KV dominance, recurrent-state contributions, and Qwen exclusions, and expand Related Work to discuss Bick et al. (2025) and Schlag et al. (2021).
Query-dependent readout. We agree that output restoration after a state swap does not independently establish selective reading, since the state effect necessarily propagates through that output. We therefore performed query-vector interventions on the original four-fact task using the same checkpoints and frozen hub heads. An auxiliary read used the same pre-query recurrent matrix for the selected head while substituting its query vector for fact A with that for fact B. Native query-time updates and returned caches were preserved, isolating query selection at the intervened readout.
Fixed-prequery intervention results (B-answer selection, %).
We evaluated 128, 128, and 64 held-out sources, respectively, averaging 12 ordered binding pairs per source and using source-level bootstrap confidence intervals.
Random queries were norm-matched Gaussian vectors, unrelated queries came from independent contexts, and the control-head intervention targeted L6H1. Values represent absolute B-answer selection among four candidates. Selected-head interventions exceeded unrelated-query controls by 69.27, 68.62, and 44.66 percentage points, with source-paired 95% confidence intervals excluding zero. Fixed-prequery state–query interactions were 7.94, 5.71, and 4.68 logits, providing causal evidence for binding-sensitive readout from the same stored memory.
Cross-fact interference. We agree that the opposing parallel/orthogonal edit effects in Fig. 5(c) do not fully establish specialized coupling. Against role- and position-matched unrelated-context nulls, within-context absolute cosines were 0.075, 0.067, and 0.100, versus 0.089, 0.093, and 0.076, respectively. Alignment was lower than the null in both 340M models but higher in GDN-1.3B. Similar opposing effects were also observed with unrelated-context edits, suggesting that this pattern alone does not establish a distinct coupling mechanism.
Removing fact A's edit component aligned with fact B's readout gradient reduced B's absolute collateral margin changes by 77.3%, 85.5%, and 88.5%, while preserving 98.7%, 100.3%, and 99.1% of A's intended margin effect. Matched-norm random removal produced negligible reductions (within ±0.2 %p). Since gradient-aligned removal suppresses first-order readout sensitivity by construction, these results characterize functional interference rather than establish a specialized learned coupling mechanism.
Together, these results support query-dependent readout and a more conservative interpretation of cross-fact effects as non-orthogonal recurrent storage with measurable interference. We will revise the Abstract accordingly. We sincerely thank the reviewer for these insightful and important comments, which have helped us critically reassess and strengthen our mechanistic claims.