acceptodds
Under review as a conference paper at ICLR 2027

A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Abstract

Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, which makes them efficient but leaky: once many key–value associations compete, earlier facts are overwritten and exact recall of distant tokens fails. The brain faces the same trade-off and resolves it with complementary learning systems: a slowly consolidating neocortex compresses experience, while a fast hippocampal store keeps the surprising episodes the neocortex cannot absorb. Many efficient hybrids already give the recurrent state such a store — a small softmax-attention cache that holds exact key–value pairs — but fill it with the most recent tokens, as if the state could not tell what it has lost. We show that it can: in a delta-rule model, the magnitude of the update a token commits to the state measures exactly how poorly that token fits the compressed memory, and it is already computed at every step. HOLA (Hippocampal Linear Attention) adds to every Gated DeltaNet layer a small exact key–value memory that keeps the top-scoring tokens by this signal, however far back they are. The mechanism is visible directly: on a 32k-token sequence, the trained model's 64-slot memory still holds the needle 29k tokens later, whereas a recency cache of the same size has long dropped it. At 1B parameters trained on 50B tokens, HOLA surpasses published Gated DeltaNet, KDA, and their preconditioned variants on in-context retrieval (four-task average 35.4 vs. at most 32.7) while staying on par with them in language modeling and commonsense reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, state-space models, associative memory, KV cache, long-context retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZYxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors motivate the use of an additional long-term storage in the case of linear models (linear attention or linear recurrent models) that stores a discrete memory outside of the fixed-size recurrent state and which can be used to retrieve long-term information. The resulting model, HOLA, is shown to be more effective at a set of standard language modeling downstream tasks relative to a number of linear models while also demonstrating improved keypass retrieval skills on the needle-in-a-haystack task.

Critical strengths

The neuroscience motivation is interesting and a good way of motivating the structure introduced in the model.

Critical weaknesses

While the architecture is interesting, it ultimately will come with a number of bottlenecks or issues. The additional memory is not particularly novel, as it simply resembles the use of an additional cached state that can always be appended or conditoned on when generating the next token; one can simply imagine this as simply having a parallel fixed-sized attention operation that goes through a gated mix-in with the recurrent output. This has been explored in the past (ex. Hymba from ICLR2025) and thus the major contribution here should be stated. While the eviction policy for this memory is interesting, it would appear to still act as a non-contiguous KV cache which ultimately has been explored in some capacity in the past.

Additionally, it appears that the approach is designed specifically for linear recurrent models, but even with a bounded memory the issue of possible difficulties of information retention still exist. One can consider an extremely information dense setting, for example code generation, where the model would have to store wide amounts of information that could still exceed the additional memory allocation, thus the model can still be expected to have issues. This is in comparison to attention, which doesn't have such memory issues but is computationally more expensive, but there still exists a middle-ground that is sparse attention approaches or hybrid models (NSA, Hymba, etc.) that should be compared against.

Next, during pre-filling. Since the exact memory is fixed-size and operates using a token-eviction policy, the actual determination of what information goes in and is evicted would naturally follow a serial process where the surprisal would be compared against each token in the memory. However, when this is parallelized, there would require a method of knowing which tokens exist in the cache at all time-steps, then computing the similarity scores/softmax. From my intuition, the way this would function would lead to essentially a large, sparse-attention computation that is difficult to optimize for HBM/SRAM memory loads due to the sparsity, alongside a foward pass of the linear model which in this case cannot actually be fully parallelized since the correct order to determining the correct token to evict requires actually knowing St−1\mathbf{S}_{t-1} before being able to compute it at step tt, as otherwise neither the eviction nor the softmax and additative gated update can be done exactly.

In terms of results, performance increases appear modest at best while there is still a computational expense incurred relative to the linear backbone that is being used. Justifying this computational expense would be rather important, particuarly for actual long-context tasks for which this model appears to be designed for. In this case, beyond just passkey retrieval tasks, there should be actual long-context performance results (i.e. ones where context length significantly exceeds the training lengths).

Questions for authors
  • Could the authors directly clarify how the actual parallelization of the method is computed, particularly in terms of handling the memory eviction?
  • Could the authors provide more conrete evidence that the method is beneficial on long-context settings that resemble real-world tasks, i.e. LongBench, Bamboo, etc.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZYxg (Part 1/2)

W1. Novelty relative to hybrid architectures and KV-cache methods Adding bounded attention memory alongside a recurrent state resembles existing hybrid architectures such as Hymba, while selective non-contiguous KV caching has also been explored. Could the authors clarify the main contribution beyond these existing designs?

Response:

Existing hybrid architectures. Lines 78–88 of the Introduction already discuss related hybrid architectures. Common choices include retaining recent tokens in a sliding-window cache and retaining tokens with high cumulative attention.

Our core contribution. The central question we address is which associations to preserve under a limited exact-memory budget. As emphasized by the title, “An Exact Memory for What the Recurrent State Forgets,” we aim to complement the recurrent state with exact memory for information that it may lose. Our key insight is that delta-rule models already compute an innovation term for each input relative to the current state; the actual write magnitude, β∥e∥\beta\lVert e\rVert, provides an intrinsic surprise signal. HOLA directly uses this signal to decide which key/value associations to preserve exactly.

Controlled eviction-policy experiments. We compare HOLA+recency (sliding-window selection) in Tables 1 and 2 and cumulative-attention selection in Table 3. These controls test our core strategy: allocating exact memory according to the recurrent state's intrinsic write signal, rather than solely by position or cumulative attention. The results support the effectiveness of this strategy in improving overall retrieval performance.

W2. Capacity limits and alternative attention architectures A bounded cache may be insufficient for information-dense tasks such as code generation. How does HOLA address this limitation, and how does it compare with sparse-attention or hybrid approaches such as NSA and Hymba, which offer alternatives between recurrent memory and full attention?

Response:

Choosing bounded memory. The motivation for this bounded exact memory is that, with fixed model dimensions and cache budgets, HOLA maintains only the recurrent state, a bounded exact cache, and a local context of fixed size. Its persistent inference state is therefore O(1)O(1) with respect to context length, preserving the complexity advantages of linear attention for long sequences.

Resource constraints of sparse attention. Sparse attention and bounded recurrent memory operate under different resource constraints. Methods such as NSA dynamically select historical blocks, reducing the KV accessed by each query, but historical KV must still be retained so that later queries can select them again. Total storage therefore still grows with context length. Storing these KV in tiers across GPU and host memory changes where they reside, but does not make total storage independent of length. The global-attention branches in Hymba likewise retain KV caches that grow with context length.

Real-world long-context evaluation. We have added a LongBench-E comparison of 1B HOLA and GDN after matched continued pretraining at 8K context length. The settings and results are provided in our response to W4/Q2 below.

AuthorsOpenReview ↗

Response to Reviewer ZYxg (Part 2/2)

W3 / Q1. Prefill parallelism and hardware memory access Cache eviction appears sequential because scoring depends on the preceding recurrent state, and parallel attention requires knowing the correct cache membership at each step. Could the authors explain how prefill parallelizes these operations and handles non-contiguous KV access efficiently across HBM and SRAM?

Response:

Parallel cache construction by chunk. Appendix E (lines 718–720) already explains that HOLA's prefill uses fixed write-time scores to construct the prefix-top-ww cache of preceding tokens for each chunk in a batch. Queries within a chunk share the same historical cache, so there is no need to simulate cache insertion and eviction token by token. Write scores are computed through GDN's chunkwise computation path.

Compact caches and shared reads. For each head, the selected KV are first gathered by chunk into compact w×dw\times d arrays. Although the selected tokens occupy noncontiguous positions in the original sequence, the resulting attention inputs have a regular layout: the CC queries in a chunk share this cache in a C×wC\times w historical-attention computation, while a C×CC\times C causal local-attention computation handles the current chunk. The gathering cost is shared across the chunk, without each query independently gathering KV from the historical sequence.

HBM/SRAM access. This layout allows attention to use regular matrix tiles: once a KV tile is loaded from HBM into on-chip SRAM, it can be reused by multiple queries. As stated in lines 729–731, we use two FlashAttention varlen calls for the historical cache and chunk-local causal attention, respectively, and merge the results with the appropriate normalization. Thus, noncontiguous source access is concentrated in cache construction, while subsequent reads can use standard tiled attention kernels instead of irregular per-query sparse access over the entire history.

W4 / Q2. Real-world long-context benefits and computational cost The performance gains appear modest relative to the additional computational cost. Could the authors provide results on realistic long-context tasks, such as LongBench or Bamboo, with inputs substantially longer than the training context, to demonstrate that the gains extend beyond synthetic retrieval and justify the overhead?

Response:

Real-world long-context experiments. We have added a LongBench-E comparison of 1B GDN and 1B HOLA, covering real-world document QA, multi-document summarization, classification, and code completion. Because the original maximum training length of 2,048 is substantially shorter than the maximum input length of 32,768 used in the LongBench-E evaluation, we first extended the models' long-context capabilities through a standard continued-pretraining procedure. Only the final checkpoints after continued pretraining are compared below.

Continued pretraining settings. Starting from their respective original checkpoints pretrained on 50B tokens at a context length of 2,048, both models receive an additional 5B tokens (approximately 10% of the original budget) at a training length of 8,192. The additional data come from training shard 24 and subsequent shards of SlimPajama-627B Reupload.

Evaluation settings. We use the full 13-task LongBench-E suite with 3,668 examples, generating 7,336 new predictions across the two models. Actual input lengths range from 277 to 32,768 tokens. We use greedy decoding without a chat template.

TaskGDN (after) ↑HOLA (after) ↑HOLA−GDN
Qasper5.916.05+0.14
MultiFieldQA-en14.7515.55+0.80
HotpotQA6.847.46+0.62
2WikiMQA9.6110.83+1.22
GovReport5.8812.91+7.03
MultiNews1.6811.69+10.01
TREC23.6712.33-11.34
TriviaQA30.8531.98+1.13
SAMSum28.1323.74-4.39
PassageCount5.094.50-0.59
PassageRetrieval-en6.145.08-1.06
LCC44.2445.19+0.95
RepoBench-P36.4536.88+0.43

Results. HOLA outperforms GDN on 9 of the 13 tasks, with advantages spanning QA, summarization, and code completion. In particular, HOLA/GDN score 12.91/5.88 (+7.03 points) on GovReport and 11.69/1.68 (+10.01 points) on MultiNews. These results provide additional evidence for the usefulness of HOLA's bounded exact memory on real-world long-context tasks, particularly summarization tasks that require integrating information across long documents.

Dear Reviewer ZYxg,

Thank you again for your time and thoughtful review of our paper! As the author–reviewer discussion period is approaching its end tomorrow, we wanted to gently follow up to see if you have had a chance to read our response.

In brief, we have addressed your main concerns by clarifying the intrinsic delta-rule O(1) memory mechanism, detailing the chunkwise FlashAttention execution, and adding a 32k LongBench-E evaluation. We would be very grateful to know whether our response resolves your concerns, or if there are any remaining questions we can quickly clarify before the deadline.

Best regards, Authors

Reviewer_nDYuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HOLA, a hybrid memory architecture for linear-attention language models. Motivated by complementary learning systems, it augments each Gated DeltaNet layer with a bounded explicit key–value cache alongside its compressed recurrent state. Cache entries are selected by the magnitude of the delta-rule update, β||e||, rather than recency, allowing important historical associations to be preserved regardless of distance. A separate cache-reading pathway sharpened with RMSNorm improves retrieval selectivity. Evaluated at 340M and 1B scales on retrieval, long-context, language modeling, and commonsense tasks, HOLA improves retrieval over Gated DeltaNet and recency-based caching while keeping inference memory bounded.

Critical strengths

S1. A simple, well-motivated approach to selective memory preservation.

HOLA reuses the delta-rule update magnitude already available in the recurrent model to decide what to cache exactly, avoiding a separate learned importance estimator. The decoupled sharpened-read normalization also addresses a real limitation of applying softmax attention directly to normalized linear-attention keys and queries. The combination of selective storage and sharpened retrieval is technically appealing, with small parameter overhead.

S2. Strong controlled experiments and efficiency analysis.

The 340M experiments match backbone and training configuration: the four-task retrieval average improves from 22.83 (GDN) and 23.68 (HOLA+recency) to 27.75 (Table 1), and on RULER S-NIAH-1 at 32K, HOLA reaches 0.58 versus 0.24 (recency) and 0.14 (GDN). Cache-content analysis supports that important earlier tokens remain accessible after long delays, and the paper reports training, prefill, decoding, and memory costs.

Critical weaknesses

W1. The 1B comparisons are not sufficiently controlled.

HOLA's 1B four-task average of 35.42 is compared against a published result of 32.66, but Appendix D acknowledges no same-backbone 1B GDN control was trained and evaluation protocols were not fully matched; sample selection, generation length, truncation, and scoring can all differ, and DROP uses answer containment rather than standard EM/F1. The reproduced 340M GDN average (within ~0.3 points) is reassuring, but the 1B claim remains less conclusive than the matched 340M experiments.

W2. The link between update magnitude and future forgetting is not fully demonstrated.

β||e|| measures a token's update at write time, not how much information will later be lost: a large update may mark an initially surprising but stably stored association, while a small one may become unretrievable after interference. Since scores are fixed at insertion, the rule does not track later changes. The practical usefulness of the heuristic is supported by cache-content visualizations and recency comparisons, but the stronger interpretation — that the recurrent state directly identifies what it will forget — needs more evidence. Comparisons with alternative importance measures (e.g., H2O) are also mainly at 46M with a flat cache read, not the full 340M configuration.

Questions for authors

Q1. (W1) Could the authors provide a 1B evaluation against existing GDN or linear-attention checkpoints under matched prompts, sample sets, truncation, and scoring — a unified re-evaluation of available checkpoints would suffice if retraining is impractical?

Q2. (W2) Have the authors measured the correlation between write-magnitude scores and actual retrieval error of a token's association at later positions, and does the criterion still beat alternative selection policies (e.g., attention-based importance) in the full sharpened-read architecture at 340M?

2 comments

W1 / Q1. Controlled 1B comparisons The 1B results rely on published baselines without a same-backbone GDN control, while sample selection, truncation, generation length, and scoring may differ, including DROP answer containment rather than standard EM/F1. Could the authors provide a matched 1B comparison? A unified re-evaluation of available checkpoints would suffice if retraining is impractical.

Response:

Matched training and evaluation. We have trained an additional 1B GDN from scratch to match HOLA, using the same backbone, Swish output gate, tokenizer, training data and data order, shared backbone initialization, and optimization settings. We evaluate it using the same task templates, sample sets, truncation, generation, and scoring protocols as HOLA in the original manuscript.

Results across all task categories. The table below retains HOLA's results and adds the newly trained GDN. HOLA outperforms GDN on all four retrieval tasks, significantly improving the average by 3.81 percentage points. The CS6 average improves by 0.94 percentage points. These conclusions are consistent with Table 1 in the manuscript.

ModelFDA ↑SWDE ↑SQuAD ↑DROP ↑Retrieval Avg. ↑WikiText-2 PPL ↓LAMBADA PPL ↓CS6 Avg. ↑
GDN (ours, new)15.1552.5736.2922.4231.6117.9612.8750.34
HOLA (original manuscript)23.9654.1940.7822.7635.4217.6612.8851.28

W2 / Q2. Write magnitude and future forgetting Write-time update magnitude does not necessarily predict later forgetting, and fixed scores cannot track subsequent interference. Have the authors measured its correlation with later retrieval error, and does this criterion outperform alternatives such as attention-based importance in the full 340M sharpened-read architecture, beyond the current 46M flat-read comparison?

Response:

Comparison in the full 340M architecture. We have extended the cumulative-attention (H2O-style) selection policy to the full 340M model. Apart from the cache selection policy, all architectural and training settings remain the same as in the original HOLA-340M. See the results below.

Selection policyFDA ↑SWDE ↑SQuAD ↑DROP ↑Retrieval Avg. ↑WikiText-103 PPL ↓LAMBADA PPL ↓CS6 Avg. ↑
Cumulative attention (H2O-style, new)18.1031.4030.7018.1024.5823.3232.9842.80
β∥e∥\beta\lVert e\rVert (HOLA, original manuscript)20.2035.9034.1020.8027.7522.9230.2742.85

Retrieval, PPL, and commonsense results. HOLA outperforms cumulative attention on all four retrieval tasks, improving the average by approximately 3.18 percentage points. WikiText-103 and LAMBADA PPL decrease by 0.40 and 2.71, respectively. This justify the effectiveness of HOLA's eviction policy design.

Dear Reviewer nDYu,

Thank you again for your time and thoughtful review of our paper! As the author–reviewer discussion period is approaching its end tomorrow, we wanted to gently follow up to see if you have had a chance to read our response.

In brief, we have addressed your main concerns by training a matched 1B GDN baseline under identical evaluation protocols and extending the H2O baseline to the full 340M architecture. We would be very grateful to know whether our response resolves your concerns, or if there are any remaining questions we can quickly clarify before the deadline.

Best regards, Authors

Reviewer_DK2yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HOLA incorporates a precise KV cache of 64 slots into every Gated DeltaNet layer. This cache retains tokens that have the highest delta-rule update magnitude β·||e|| rather than holding onto the latest tokens, and retrieves them using a refined softmax method. It is evaluated at 1B against established benchmarks and at 340M against comparable GDN and recency-cache controls, focusing primarily on in-context retrieval and RULER.

Critical strengths
  1. The eviction score represented by β·||e|| is a valuable insight: it entails no cost, has already been calculated, and is straightforward to understand. This is supported by the cache analysis shown in Fig. 3, as needle tokens remain in the cache even after an increase of 29K tokens.

  2. The experiment involving the 340M HOLA+recency control is appropriate, given that only the selection rule is altered. The improvements are significant (retrieval averages increase from 23.68 to 27.75, while S-NIAH-1 at 32K rises from 0.24 to 0.58).

  3. The paper transparently identifies the source of the perplexity improvement (the BOS problem in App. G). The additional memory usage is minimal.

Critical weaknesses
  1. The 1B outcome is assessed alongside figures from other studies, lacking a GDN that was trained by the authors at an equivalent scale. Therefore, the only clear evidence available is found at 340M.

  2. The score remains constant at the time of writing, and slots are not re-evaluated. Initial tokens experience significant residuals simply due to the empty state, causing the rule to have an inherent bias towards early positions. The results from the multi-needle RULER are also variable.

  3. The latency during decoding is approximately 35% greater, while the throughput is about 27% less than that of GDN, and this should be articulated as explicitly as the memory figure. There is a lack of comparisons with LTE and a GDN+SWA hybrid at similar costs.

Questions for authors

See weaknesses

2 comments

W1. Matched 1B GDN baseline The 1B outcome is assessed alongside figures from other studies, lacking a GDN that was trained by the authors at an equivalent scale.

Response:

Matched 1B GDN control. We have trained an additional 1B GDN control from scratch to match HOLA, using the same backbone, Swish output gate, tokenizer, training data and data order, shared backbone initialization, and optimization settings.

The table below compares the newly trained 1B GDN with HOLA:

ModelFDA ↑SWDE ↑SQuAD ↑DROP ↑Retrieval Avg. ↑WikiText-2 PPL ↓LAMBADA PPL ↓CS6 Avg. ↑
GDN (ours, new)15.1552.5736.2922.4231.6117.9612.8750.34
HOLA (original manuscript)23.9654.1940.7822.7635.4217.6612.8851.28

HOLA outperforms GDN on all four retrieval tasks, improving the average by 3.81 percentage points. This conclusion is consistent with Table 1 in the manuscript.

W2. Fixed scores, early-position preference The score remains constant at the time of writing, and slots are not re-evaluated. Initial tokens experience significant residuals simply due to the empty state, causing the rule to have an inherent bias towards early positions.

Response:

Fixed scoring versus re-evaluation. Fixing each score at write time is designed to maintain efficiency and the bounded memory of linear attention.

It is useful to distinguish the problem that re-evaluation is intended to address. To identify historical associations that were not forgotten at write time but were subsequently lost under interference, the candidates for re-evaluation must include historical tokens that have already left the exact cache. Rescoring all such historical tokens requires retaining their keys and values, causing memory usage to grow with context length and sacrificing the bounded memory of linear attention.

HOLA instead reuses the intrinsic write signal already available in the delta rule, maintaining an exact memory with a fixed budget through a parameter-free selection rule. Its persistent inference state is O(1)O(1) with respect to context length, preserving the complexity advantages of linear attention for long sequences.

Early-position preference bias. We want to clarify that even for initial tokens, their value representation and write strength β\beta are learnable. The tokens' selection scores are therefore not assigned a fixed high value solely because of their positions. Whether it is retained is a learned outcome, rather than being enforced by the eviction rule. Any position preference can be modulated through learning. We acknowledge that the initial conditions introduced by zero initialization may still affect the final score distribution. Further analysis of the respective effects of initialization, data, and learning, and improvements to these components, are left to future work.

W3. Decoding cost and comparisons with LTE and GDN+SWA The latency during decoding is approximately 35% greater, while the throughput is about 27% less than that of GDN, and this should be articulated as explicitly as the memory figure. There is a lack of comparisons with LTE and a GDN+SWA hybrid at similar costs.

Response:

Decoding cost. We agree that the decoding overhead should be stated alongside the memory advantage.

GDN+SWA control. We already report HOLA+recency in Tables 1 and 2. HOLA+recency is precisely the GDN+SWA hybrid requested by the reviewer. HOLA achieves a substantial improvement of 4.07 percentage points in the four-task retrieval average over GDN+SWA (HOLA+recency). This supports the effectiveness of HOLA's memory design for retrieval tasks.

Selection relative to recurrent memory. We have discussed LTE in the related work. LTE uses a CNN with a limited 13-token receptive field to compute retention scores, primarily aggregating short-range interactions, instead of long-term token representations.

Dear Reviewer DK2y,

Thank you again for your time and thoughtful review of our paper! As the author–reviewer discussion period is approaching its end tomorrow, we wanted to gently follow up to see if you have had a chance to read our response.

In brief, we have addressed your main concerns by training a matched 1B GDN baseline from scratch, clarifying the O(1) write-time scoring and decoding overhead, and highlighting our GDN+SWA and LTE comparisons. We would be very grateful to know whether our response resolves your concerns, or if there are any remaining questions we can quickly clarify before the deadline.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.