acceptodds
Under review as a conference paper at ICLR 2027

How Linear Attention Remembers

Abstract

Linear attention replaces the growing key–value (KV) cache of standard attention with a fixed-size recurrent state, substantially reducing memory growth with context length. This efficiency, however, changes how past information is stored: many tokens must share and repeatedly update the same memory. We study how this recurrent state functions as a memory system. Using an analytical decomposition together with controlled causal interventions in pretrained GLA and GDN models, we trace how recalled information is written, retained, and later accessed. We find that fact-specific information enters recurrent memory through concentrated, content-dependent writes and is later accessed through concentrated query-time read pathways. Multiple facts can remain selectively accessible within the same state, yet their internal representations exhibit cross-fact causal coupling rather than independent KV-like storage. As memory load increases, both recall and targeted editability degrade, whereas elapsed context alone has a substantially smaller effect within the tested regime. Causal interventions on subsequent writes further show that interference is shaped by their overlap with existing memory. Finally, in hybrid architectures that combine recurrent layers with full attention, the runtime memory directly supporting recall shifts predominantly to the full-attention KV state. Together, these results reveal how fixed-size recurrent memory supports selective recall despite shared storage, while exposing the interference and capacity limits that distinguish it from token-addressable KV memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5qHfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks how linear-attention models keep in-context facts in their recurrent state, using donor/recipient interventions in GLA-340M, GDN-340M, and GDN-1.3B. It reports that one development-selected "hub" head per model writes content-specific information and mediates the later read, that four facts stay selectively editable despite cross-fact coupling, and that adding facts hurts recall more than adding filler. In hybrid models, swapping the full-attention KV cache transfers nearly all of the donor effect, while swapping the recurrent state transfers almost none.

Critical strengths
  1. The hybrid result is clean and well replicated. The KV-versus-recurrent factorial holds across a near-matched GDN pair, two Qwen3.5 sizes, PG19 filler, and RULER. It shows which substrate carries the recalled fact at query time.

  2. Selection is kept separate from confirmation. In both 340M models, the frozen development-selected head ranks first on held-out sources (App. C.1), and control heads test content specificity. This guards against selection circularity, so I accept the concentration finding, if not yet its mechanistic reading.

Critical weaknesses
  1. Two mechanism claims in the abstract lack tests that could fail. For the read pathway, the paper specifies single-token queries and answers only for the development task (App. B). In that setting, a head's pre-query state reaches the logits only through its query-time output (Eq. 2). Restoring the recipient's output, as in Fig. 4, therefore cancels any state swap by construction. The output patch, state-swap screens, and logit lens also confound the read with stored content. The coupling result has a similar problem. The part of fact A's edit that points along fact B's edit moves fact B whenever the two edits overlap (Fig. 5c). Because the two parts add nearly linearly and the full edit moves B less than that part alone, the remaining orthogonal part has to push B the other way. Any nonzero overlap thus produces the opposing effects the paper reads as coupling. Norm-matched Gaussian controls are a weak null here, since random directions are nearly orthogonal to any fixed direction in a high-dimensional state. As reported, neither test rules out a simpler reading: stored content alone, or ordinary non-orthogonal storage. Please add a test of the read that could fail, such as swapping only the query vector between bindings. For coupling, please compare the cosine between the two edits with a null from unrelated contexts and, if it does not exceed that null, restate the abstract claim as non-orthogonal storage with small cross-talk.

  2. The closest mechanistic prior work is not cited. Bick et al. (ICML 2025) anticipate the concentration finding and the hybrid result's direction: they find a Gather-and-Aggregate mechanism concentrated in a few heads of Transformers and Mamba-based SSMs, with hybrids handing the aggregation role to attention. The paper does cite Schlag et al. (2021), but it does not engage their capacity analysis, which predicts key-overlap interference and decline under load. Without a comparison to these works, I cannot tell which of the concentration, hybrid, load, and overlap findings are new; please position the paper against both and say what it adds.

Questions for authors
  1. Every filler token is also a write, so the load-versus-filler comparison contrasts template-matched fact writes with prose writes, not competition with passive loss. The ranking also depends on the chosen ranges: GLA loses more recall from 128 to 1024 filler tokens than from one to four facts (Fig. 7a-b). At fixed length, how does recall compare across same-template bindings, different-template bindings, and token-matched filler?

  2. Do the four candidates include values of the other stored facts? If they do, one-fact and multi-fact recall are different tasks.

  3. Fig. 7c shows only GDN-340M-colored markers, and neither the caption nor the text names the model. Was the key-overlap test also run on GLA-340M?

  4. Why does GLA's clean four-fact recall differ across App. D.1, Fig. 7a, and Fig. 8a (about 92%, 96%, and 76%)?

  5. Zero-ablating the hub's write drives GLA recall below chance (Fig. 3a). Under mean or resample ablation, does blocking the hub alone still hurt more than blocking all heads (Fig. 3c)?

  6. Which two Qwen3.5 conditions were excluded (p.7 l.344-345), and when was the exclusion rule fixed?

  7. Near-zero recovery from the recurrent swap also fits a recurrent state that still holds the fact but loses to the recipient's KV entries. The paper mostly limits its claim to the direct runtime carrier (App. E.1), but the Sec. 4.1 title ("The memory substrate shifts") invites the reading that the recurrent state no longer stores the fact. Masking attention to the fact span in every full-attention layer would test recall from the recurrent state alone.

  8. Despite the Limitations caveat, the title and Conclusion speak about linear attention in general. The full analysis covers three checkpoints from one release, none above 1.5B parameters, and single-component dominance does not hold in GLA-7B (App. E.2). Please scope both to the tested setting.

7 comments

Note. W and Q denote Critical Weaknesses and Questions for Authors, respectively, following the original review's numbering.

Summary of Responses and Planned Revisions

We thank the reviewer for their constructive feedback. We conducted fixed-memory query-vector interventions demonstrating binding-sensitive readout beyond output restoration. Empirical null comparisons did not consistently support a distinct cross-fact coupling mechanism. Additional experiments address write-ablation robustness, factual-binding competition, GLA key overlap, hybrid KV masking, recall-baseline discrepancies, and Qwen3.5 exclusion criteria.

Planned Manuscript Revisions: We will clarify output restoration in Fig. 4 and add query-vector controls to the Appendix. We will refine the coupling claims in the Abstract and Fig. 5(c) using empirical-null and interference analyses. We will expand the Fig. 3 ablation analysis, add candidate-exposure and GLA key-overlap controls to the Fig. 7 analysis, and clarify the protocols behind Appendix D.1 and Figs. 7(a)/8(a). We will also revise Section 4.1 and Appendix E.1 to clarify KV dominance, recurrent-state contributions, and Qwen exclusions, and expand Related Work to discuss Bick et al. (2025) and Schlag et al. (2021).

W1: Causal Evidence for Query-Dependent Readout and Cross-Fact Interference

Query-dependent readout. We agree that output restoration after a state swap does not independently establish selective reading, since the state effect necessarily propagates through that output. We therefore performed query-vector interventions on the original four-fact task using the same checkpoints and frozen hub heads. An auxiliary read used the same pre-query recurrent matrix for the selected head while substituting its query vector for fact A with that for fact B. Native query-time updates and returned caches were preserved, isolating query selection at the intervened readout.

Fixed-prequery intervention results (B-answer selection, %).

ModelSelected-head q_BRandom queryUnrelated queryControl-head q_B
GLA-340M91.3424.0222.072.73
GDN-340M82.4915.7613.871.04
GDN-1.3B58.2013.4113.542.34

We evaluated 128, 128, and 64 held-out sources, respectively, averaging 12 ordered binding pairs per source and using source-level bootstrap confidence intervals.

Random queries were norm-matched Gaussian vectors, unrelated queries came from independent contexts, and the control-head intervention targeted L6H1. Values represent absolute B-answer selection among four candidates. Selected-head interventions exceeded unrelated-query controls by 69.27, 68.62, and 44.66 percentage points, with source-paired 95% confidence intervals excluding zero. Fixed-prequery state–query interactions were 7.94, 5.71, and 4.68 logits, providing causal evidence for binding-sensitive readout from the same stored memory.

Cross-fact interference. We agree that the opposing parallel/orthogonal edit effects in Fig. 5(c) do not fully establish specialized coupling. Against role- and position-matched unrelated-context nulls, within-context absolute cosines were 0.075, 0.067, and 0.100, versus 0.089, 0.093, and 0.076, respectively. Alignment was lower than the null in both 340M models but higher in GDN-1.3B. Similar opposing effects were also observed with unrelated-context edits, suggesting that this pattern alone does not establish a distinct coupling mechanism.

Removing fact A's edit component aligned with fact B's readout gradient reduced B's absolute collateral margin changes by 77.3%, 85.5%, and 88.5%, while preserving 98.7%, 100.3%, and 99.1% of A's intended margin effect. Matched-norm random removal produced negligible reductions (within ±0.2 %p). Since gradient-aligned removal suppresses first-order readout sensitivity by construction, these results characterize functional interference rather than establish a specialized learned coupling mechanism.

Together, these results support query-dependent readout and a more conservative interpretation of cross-fact effects as non-orthogonal recurrent storage with measurable interference. We will revise the Abstract accordingly. We sincerely thank the reviewer for these insightful and important comments, which have helped us critically reassess and strengthen our mechanistic claims.

W2: Distinction from Prior Mechanistic Studies

We thank the reviewer for identifying these important connections.

Bick et al. (ICML 2025) identify Gather-and-Aggregate mechanisms in Transformers and Mamba-based SSMs, showing concentrated retrieval contributions and attention-dominated aggregation in hybrid architectures. Schlag et al. (ICML 2021) establish the fast-weight interpretation of linear attention and analyze finite-memory limitations, motivating delta-rule updates to reduce interference. We acknowledge that head concentration, hybrid attention dominance, and finite-capacity interference are not novel findings in isolation.

Our contribution is to causally connect the writing, retention, and query-dependent retrieval of individual in-context associations in pretrained linear-attention models. We link these stages through fact-time state-update interventions, pre-query state transplantation, and fixed-memory query-vector interventions. This runtime memory-lifecycle analysis complements prior work on retrieval-critical components and fast-weight memory. We further characterize factual competition, retention distance, and write-key geometry, interpreting cross-fact effects as non-orthogonal storage with measurable interference rather than a distinct coupling mechanism.

Q1: Distinguishing Memory Load from Intervening Context

We agree that filler tokens also update recurrent memory. We therefore conducted a length- and position-matched study comparing same-template bindings, different-template bindings, and prose filler while controlling the target association, answer candidates, and retention distance.

At a 512-token target-to-query distance, four total bindings (one target and three same-template competitors) reduced correct-answer margins relative to matched filler by 1.55, 2.39, and 2.24 logits in GLA-340M, GDN-340M, and GDN-1.3B, respectively. Different-template bindings also reduced margins, although differences in competitor retrievability prevent attributing this effect specifically to template similarity. We confirmed that prose filler induces nonzero recurrent-state updates. Thus, these results distinguish competing factual writes from other active memory updates rather than passive decay.

We also extended the original Fig. 7(b) experiment without changing its protocol. GLA-340M recall declined from 89.06% at 1K to 37.50% at 3,950 filler tokens. Across 128–3,950 tokens, recall declined by 60.16 pp, compared with 38.28 pp when increasing memory load from 1 to 16 facts. In contrast, GDN-340M recall declined by only 3.91 pp over 128–1,902 filler tokens, compared with 31.25 pp under increased load. These comparisons reflect separate experimental sweeps rather than matched causal estimates.

Together, these results show that neither memory load nor retention distance universally dominates recall degradation. Their relative effects depend on architecture, evaluated range, and metric. Notably, GLA's accuracy-based reversal does not extend to correct-answer margins, where increased load produces a larger decline (4.79 versus 3.97 logits).

Q2: Memory Load, Candidate Construction, and Associative Competition

We thank the reviewer for highlighting this distinction. The original evaluation uses four fixed answer candidates (red, blue, green, and yellow), but their roles change with memory load. At K=1, incorrect candidates are unbound distractors, at K=4, they correspond to other stored bindings, and at K=8/16, values are reused across bindings. Thus, increasing memory load also changes associative competition, preventing the accuracy decline from being attributed solely to limited memory capacity.

To distinguish competing associations from candidate exposure, we conducted a frequency- and position-matched unbound-foil control. Alternative answer values appeared either without competing key associations or bound to other keys, while their frequencies and positions, the target association, answer candidates, and retention distance remained fixed.

At a 512-token distance, competing-binding minus unbound-foil margin differences were −0.647 logits for GLA-340M (95% CI [−0.861, −0.436]), −1.390 logits for GDN-340M ([−1.534, −1.245]), and −0.571 logits for GDN-1.3B ([−0.715, −0.425]). These results show interference from competing factual associations beyond alternative-value exposure alone.

Nevertheless, this control neither equates the original K=1 and K=4 task difficulty nor isolates intrinsic memory capacity. We therefore interpret memory-load effects as load-dependent associative interference, potentially arising during storage and/or retrieval, rather than a direct measurement of intrinsic memory-capacity limits.

Q3: Extending Key-Overlap Analysis to GLA

We thank the reviewer for pointing out this omission. The original Figure 7(c) experiment was conducted only on GDN-340M, which was not explicitly stated in the caption.

To examine whether the key-overlap effects extend to GLA, we conducted additional experiments on GLA-340M using its frozen selected head (L21H3), following the original K=4 and K=16 settings. Specifically, we aligned or orthogonalized a later distractor's write key relative to the target-fact write key while preserving its norm, using the same 64 sources per memory-load condition.

At K=4, alignment changed the correct-answer margin by −0.222 logits (95% CI [−0.273, −0.173]), while orthogonalization changed it by +0.373 logits (95% CI [0.278, 0.476]). These results provide causal evidence that write-key geometry affects factual recall in GLA as well as GDN. However, because GLA uses outer-product updates whereas GDN employs key-dependent delta corrections, we interpret the findings as shared sensitivity to key geometry rather than evidence of identical interference mechanisms.

Q4: Discrepancies in Clean Recall Across Evaluation Protocols

We thank the reviewer for pointing out this inconsistency. The three reported GLA-340M clean recall accuracies were obtained using different evaluation protocols, which were not sufficiently distinguished in the original manuscript.

Clean recall across evaluation protocols.

ExperimentClean recall (%)Evaluation unitFull input length
Appendix D.191.80470/512 queried bindings562 tokens
Figure 7(a)96.09123/128 selected targets658 tokens
Figure 8(a)76.17390/512 queried bindings898 tokens

Although all three experiments use the same checkpoint, they differ in fact-slot layouts (4/16/32), presentation counts (4/3/4), source windows, fact ordering, value mappings, and evaluation aggregation. In particular, Figure 7(a) evaluates one selected target per source, whereas Appendix D.1 and Figure 8(a) aggregate recall across queried bindings.

We verified the reported accuracies against the saved clean logits and reproduced the original measurements, finding no numerical or plotting errors. Thus, the differences reflect distinct evaluation protocols rather than inconsistent measurements under identical conditions. Comparisons within each experiment use consistent settings, but the baselines should not be directly compared, and the accuracy gaps cannot be attributed to any single experimental factor.

Q5: Robustness of Hub-Only versus All-Head Write Ablation

We thank the reviewer for this suggestion. We repeated the hub-only versus all-head write-ablation comparison using zero, mean, and donor-resampled replacements on 128 held-out GLA-340M sources.

GLA-340M recall under different write-ablation replacements.

ReplacementHub-only recall (%)All-head recall (%)Hub − All (pp)
Zero14.122.7−8.6
Mean14.122.7−8.6
Resample17.524.1−6.6

Hub-only blocking produces lower recall than all-head blocking under all three replacement strategies. For resampling, the accuracy difference is not statistically conclusive (95% CI [−13.75, +0.31] pp), although the paired correct-answer margin difference remains significant (−0.869 logits, 95% CI [−1.112, −0.640]).

Although mean replacements have approximately 61% of the native update norm, they are behaviorally similar to zero ablation: only 2.34% of hub-only predictions changed, all between incorrect answers, leaving accuracy unchanged. Thus, mean ablation provides limited independent robustness evidence. In contrast, donor-resampled replacements have approximately 103% of the native update norm and change 23.28% of predictions, providing stronger evidence beyond zero ablation.

Together, these results support nonadditive causal effects among recurrent components that persist under a behaviorally distinct replacement strategy, while the uncertainty in resampled accuracy differences prevents stronger conclusions about a unique circuit or compensatory mechanism.

Q6: Excluded Qwen3.5 Conditions and Exclusion Criteria

We initially considered sequential state transition and symbolic accumulation to evaluate memory behavior on more complex tasks. However, preliminary evaluations showed that Qwen3.5-4B achieved only 25% clean accuracy on both tasks, equivalent to chance in our four-choice setting. Moreover, none of the cache interventions improved accuracy beyond 25%.

We therefore excluded these tasks from the primary memory-attribution analysis because the model could not reliably perform the underlying tasks, making causal attribution uninformative. This decision was based on baseline task competence rather than favorable attribution outcomes.

Reviewer_A6C9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how pretrained linear-attention models store, retain, and retrieve factual information in fixed-size recurrent states. The authors combine an analytical framework with causal interventions to trace how facts are written, how the state evolves, and how information is read out at query time. Experiments on GLA and GDN models show selective recall from shared recurrent states and examine how memory load and overlapping writes affect retention and editability. In hybrid architectures, full-attention KV caches become the dominant direct memory substrate for the recall tasks tested.

Critical strengths

(1)The intervention framework traces information through writing, retention, and retrieval. Combining write blocking, state replacement, and output interventions provides stronger evidence than a purely correlational analysis and helps explain how recurrent memory supports factual recall. (2)The experiments show that multiple facts can remain selectively accessible within a shared recurrent state, even as they interact measurably with one another. The analyses of memory load, overlapping writes, and hybrid architectures help clarify both the capabilities and the limitations of recurrent memory.

Critical weaknesses

(1)The main findings on writing, retention, and readout come primarily from controlled key-value association tasks with constrained answer spaces. Appendix E.1 extends the hybrid memory analysis to additional retrieval settings, but it does not test the complete memory lifecycle in pure recurrent models on more naturalistic language tasks. The prose-retrieval experiment in Appendix F.2 also could not support causal analysis because baseline accuracy was too low. The evidence therefore leaves open whether the complete mechanism extends beyond controlled associative recall. (2)The full causal lifecycle analysis focuses primarily on three relatively small pure recurrent models. Appendix E.2 extends the readout analysis to larger checkpoints, where the distribution of causal contributions across components varies substantially. For example, the leading component accounts for 94.7% of the full intervention effect in GLA-340M, compared with 12.3% in GLA-7B. These experiments do not establish whether the complete write-retain-read mechanism holds in larger models with more distributed readout pathways. This leaves the scope of the mechanistic conclusions uncertain.

Questions for authors

(1)Can the authors provide evidence for the complete write-retain-read mechanism in a more naturalistic language retrieval setting with sufficiently high baseline accuracy? This would help establish whether the findings extend beyond controlled associative recall. (2)Given the variation in readout concentration across checkpoints in Appendix E.2, how far does the evidence support generalizing the complete causal memory lifecycle to larger models, where the relevant information is distributed across multiple components?

3 comments

Note. W and Q denote points raised under Critical Weaknesses and Questions for Authors, respectively, with numbering corresponding to the original review.

Summary of Responses and Planned Revisions

We thank the reviewer for the constructive feedback on the generalizability of our causal memory analysis. We extended the experiments to a natural-language preference retrieval setting, where GDN-1.3B achieves over 82% joint accuracy across two question paraphrases, with causal interventions supporting factual writing, persistent-state contributions, and binding-sensitive readout (W1/Q1). We also extended the complete lifecycle analysis to GLA-7B, where joint interventions on 16 native recurrent groups recover 94.93% of the full-donor effect, providing evidence for a distributed causal pathway in a larger checkpoint (W2/Q2).

Planned Manuscript Revisions: We will expand Appendix F.2 with the natural-language retrieval experiments (W1/Q1) and extend Appendix E.2 with the GLA-7B joint group-level analysis, clarifying the distinction between native recurrent groups and query-head cache copies (W2/Q2). We will also refine the accompanying Discussion and Conclusion to distinguish concentrated and distributed causal contributions and clarify the scope of the natural-language findings (W1/Q1, W2/Q2).

W1/Q1: Generalization to Naturalistic Language Retrieval

We thank the reviewer for highlighting the limited evidence beyond controlled associative recall. To extend our analysis, we evaluated a natural-language preference retrieval task with two factual bindings, two question paraphrases, and 128 held-out sources. Unlike the original four-choice task, this experiment evaluates unrestricted next-token recall. Under a controlled layout where the target precedes the competing fact, appears twice, and is queried after 128 intervening tokens, GDN-1.3B achieves 82.81% recipient and 82.03% donor joint accuracy across both paraphrases, providing a sufficiently competent baseline for causal analysis.

We applied the original causal intervention principles to this setting. Blocking target fact-time net persistent-state updates at the frozen head L20H3 reduced individual-question recall accuracy by 50.00 and 51.56 percentage points for the two paraphrases, respectively. Restoring the corresponding clean recurrent state recovered the matched clean performance. Pre-query transplantation of the selected recurrent state also recovered 33.41–39.43% of the full-donor effect on the donor-versus-recipient logit contrast, demonstrating a persistent causal contribution from recurrent memory.

To examine binding-sensitive retrieval, we performed query-vector interventions at the selected head. Under native state updates, substituting the query vector associated with the competing binding increased its answer selection by 9.38–14.84 percentage points, with positive binding-specific content–query interactions (0.317–0.560 logits). An additional controlled intervention using the same pre-question recurrent matrix for the selected head's auxiliary read yielded consistent results, increasing alternative-answer selection by 10.16–15.63 percentage points. Together, these findings support query-dependent readout beyond output-path restoration alone.

These experiments provide evidence for causal write, retention, and binding-sensitive read contributions in a controlled natural-language retrieval setting, rather than a direct replication of the original four-choice protocol. However, retrieval of the competing binding remains less reliable and performance depends on fact ordering. We therefore do not claim robust retrieval of arbitrary associations or generalization to open-domain question answering.

W2/Q2: Complete Causal Lifecycle in Larger Models

We agree that the lower concentration of causal contributions in GLA-7B raises an important question about extending the complete memory lifecycle beyond smaller recurrent models. We therefore conducted additional joint causal interventions on GLA-7B, evaluating 16 native recurrent groups on 64 held-out sources.

Unlike the original Appendix E.2 analysis, which measured individual query-head cache-copy contributions, the new experiments use a separately frozen native-group ranking obtained from 32 development sources. Each native recurrent group corresponds to four query-head cache copies, so jointly intervening on 16 groups examines a broader set of components rather than directly extending the original single-component analysis.

Blocking target fact-time net persistent-state updates reduced the correct-answer margin by 4.75 logits relative to a matched tokenwise-prefill baseline. Restoring the clean selected-group states recovered 100.92% of the write-blocking effect, while pre-query state transplantation recovered 94.93% of the full-donor change in answer logit contrast. Furthermore, a 5.70-logit binding-specific state–query interaction under fixed-memory interventions provides separate evidence of query-dependent readout. Matched-size random and low-ranked groups showed substantially smaller effects on factual writing, state transplantation, and binding-sensitive readout.

Together, these results support a linked causal write–retain–read pathway in the evaluated GLA-7B checkpoint, even when memory contributions are distributed across multiple recurrent groups. We note that we do not interpret output-path mediation alone as evidence of selective reading, nor assume that the same distribution or retrieval mechanism generalizes to other larger architectures at this point.


We sincerely thank the reviewer for thoughtful and constructive feedback. The suggestions motivated valuable additional experiments that strengthened the evidence for our causal memory analysis, particularly its generalizability across retrieval settings and model scales. We believe these results have helped make the paper more rigorous and its contributions more clearly defined.

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review authors' responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

Reviewer_XZe7OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how linear-attention models use their fixed-size recurrent state as memory. Using causal interventions on pretrained GLA and GDN models, as well as hybrid GDN and Qwen3.5 models, it traces how facts are written, retained, and later retrieved in a controlled recall task.

The authors find that memory writing and retrieval occur through content-dependent pathways. Multiple facts can remain selectively retrievable from the same recurrent state, but they are not stored independently and can interfere with one another. Recall degrades mainly as memory load increases, while elapsed context (fact-to-query distance) has a smaller effect. In hybrid architectures, recalled information is largely dependent on the full-attention KV cache (rather than the recurrent state).

Critical strengths
  • The experimental design is carefully thought of. Heads are selected separately and frozen before evaluation and the controls and interventions are used consistently across the manuscript.

  • The analytic approach in Section 2.1 is simple, yet it captures the important directions that will be empirically tested in later sections.

  • Clear result for hybrid models. Swapping only the recurrent state transfers essentially none of the donor answer, while swapping only the KV cache transfers nearly all of it (96–100%).

  • Strong causal methodology. The write, state-swap, output-patch, and output-restore interventions cleanly separate memory storage (parameters) from memory readout.

  • Transparent reporting. The paper includes negative or mixed findings, such as weak write-read correlations or GLA failing its own collateral-damage criterion.

Critical weaknesses
  • The related work is expected to include more pointers in "Causal analysis of model memory" which is core in this effort. For example, closely related causal analyses of recurrent-state memory.

  • The context-length analysis is limited. The comparison between memory load and elapsed context only covers 128-1024 tokens; longer distances could be considered. (minor)

  • It does not test whether alternative head-selection procedures would identify different but equally effective circuits. (minor)

Questions for authors
  • How does your framework relate to prior causal analyses of recurrent-state memory, such as Sharma et al. (2024) on Mamba ( https://arxiv.org/abs/2404.03646 )? What aspects of the memory mechanism are uniquely revealed by your intervention methodology?

  • Do the conclusions regarding memory load versus elapsed context continue to hold at substantially longer retention intervals than those tested in Figure 7?

3 comments

Note. W and Q denote points raised under Critical Weaknesses and Questions for Authors, respectively, with numbering corresponding to the original review.

Summary of Responses and Planned Revisions

We thank the reviewer for their constructive suggestions and comments. We extended the original retention experiment to the models' native context limits (4K for GLA and 2K for GDN), finding that the relative effects of memory load and elapsed context depend on the architecture, context range, and evaluation metric (W2/Q2). We also evaluated four alternative head-selection criteria, which consistently identified the same dominant hub in all three smaller recurrent models (W3).

Planned Manuscript Revisions: We will extend Fig. 7(b) with the long-context retention results and refine the accompanying interpretation of memory load versus elapsed context (W2/Q2). We will add the head-selection robustness (W3) and length-matched interference analyses (W2/Q2) to the Appendix. We will also clarify the relationship to Sharma et al. (2024) in Related Work, distinguishing their analysis of pretrained factual knowledge from our investigation of in-context recurrent memory (W1/Q1).

W1/Q1: Distinction from Prior Causal Analyses of Recurrent Memory

We thank the reviewer for highlighting Sharma et al. (2024), Locating and Editing Factual Associations in Mamba. Their work employs causal tracing, activation patching, and adapted attention-knockout interventions to investigate the localization and information flow of pretrained factual knowledge in Mamba, alongside parameter-level knowledge editing.

Our work addresses a complementary question: how newly introduced factual associations are written into, retained in, and selectively retrieved from runtime recurrent memory in linear-attention models. While Sharma et al. investigate how pretrained factual knowledge is represented and retrieved, we trace the causal influence of in-context associations across distinct stages of the memory lifecycle.

Specifically, we intervene on fact-time recurrent-state updates to examine memory formation, transplant persistent states to test retention, and independently modify query vectors while holding recurrent memory fixed to examine binding-sensitive reading. We further characterize how competing factual writes, retention distance, and key overlap affect recall and interference.

Thus, our contribution is not causal localization in recurrent architectures itself, but connecting the formation, persistence, and query-dependent retrieval of individual in-context associations through causal interventions on runtime recurrent states. This complements Sharma et al.'s analysis of pretrained factual knowledge while addressing a distinct aspect of recurrent memory behavior.

W2/Q2: Memory Load versus Retention Distance at Longer Contexts

We thank the reviewer for this suggestion. We extended the original Figure 7(b) experiment to the native context limits of GLA-340M (4K tokens) and GDN-340M (2K tokens), with GDN-1.3B included as an additional evaluation. We preserved the original protocol and exactly reproduced its reported results before extending the retention intervals.

Long-context retention results.

ModelMax. contextRecall at 1K (%)Recall at max. (%)Δ Recall (pp)
GLA-340M4K89.0637.50−51.56
GDN-340M2K96.0993.75−2.34
GDN-1.3B2K100.0096.88−3.12

The results show that the relative importance of memory load and retention distance depends on the model and evaluated range. For GLA-340M, extending filler from 128 to 3,950 tokens reduces recall by 60.16 percentage points, exceeding the 38.28-point decrease observed when increasing memory load from 1 to 16 facts. In contrast, GDN-340M shows only a 3.91-point decline over its extended filler range, compared with a 31.25-point decline from increasing memory load. These comparisons describe the respective experimental sweeps rather than fully matched causal effects. Moreover, GLA's accuracy-based reversal does not extend to correct-answer margins, where the load increase produces a larger decrease (4.79 versus 3.97 logits).

We additionally conducted a separate length-matched analysis controlling target positions, answer candidates, and retention distances. At the longest evaluated distances, competing factual bindings reduced correct-answer margins relative to non-factual filler by 0.74, 2.10, and 1.50 logits in GLA-340M, GDN-340M, and GDN-1.3B, respectively. Notably, under this separate protocol, GLA-340M retained 72.66% donor-answer transfer and approximately 96% recovery of the full-donor effect at 4K tokens, despite native recall declining to 67.19%. These results distinguish interference from competing factual writes from the persistence of causal memory contributions.

Overall, both factual memory load and intervening context influence recall, but neither universally dominates the other. Their relative effects depend on architecture, retention range, and evaluation metric.

W3. Robustness to Alternative Head-Selection Procedures

We thank the reviewer for raising this concern. To assess the robustness of head selection, we evaluated all recurrent heads (96 in GLA-340M and GDN-340M, and 192 in GDN-1.3B) on the same 64 development examples using four causal scoring criteria:

  1. State transplantation: The increase in donor-versus-recipient answer logit contrast after transplanting an individual head's pre-query recurrent state.
  2. Write blocking: The decrease in correct-answer margin when an individual head's target fact-time net persistent-state updates are blocked.
  3. Readout intervention: The decrease in correct-answer margin when an individual head's query-time output is zeroed.
  4. Binding-sensitive query intervention: The content–query interaction measuring how substituting query vectors associated with alternative stored keys changes the donor-versus-recipient content effect.

All four criteria identified the same top-ranked head in each model: L21H3 (GLA-340M), L20H2 (GDN-340M), and L20H3 (GDN-1.3B). These rankings were frozen before held-out evaluation, supporting robust identification of the dominant hub across different causal criteria.

However, agreement at the top rank does not extend to the complete rankings, as lower-ranked heads and the coverage of jointly intervened component sets differ across criteria. We therefore interpret these findings as evidence of robust dominant-hub identification, rather than a unique or exhaustive memory circuit.


We thank the reviewer again for the constructive feedback and positive assessment of our work. We appreciate the suggestions, which have helped us strengthen the experimental validation and clarify the scope of our contributions.

I want to thank the authors for their clarification comments, additional experiments and most specifically for their pragmatic, detailed plans on the integration of their updates into the original version of the manuscript. Confirming and keeping my initial positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers and the Area Chair for their careful evaluation and constructive feedback.

In response to the reviews, we have conducted extensive additional experiments to examine the robustness, generalizability, and causal validity of our findings. Given the scope of the requested evaluations, we prioritized completing and validating these experiments and sharing the results promptly through our rebuttal. The responses below summarize our new findings and explain how they address the reviewers' concerns.

While the additional results are now available, not all of them have yet been incorporated into the manuscript. We plan to upload an updated version early next week, including the new experiments and corresponding clarifications, so that the reviewers and Area Chair can examine the complete revisions.

We appreciate the opportunity to clarify and strengthen our work and look forward to further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.