- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors motivate the use of an additional long-term storage in the case of linear models (linear attention or linear recurrent models) that stores a discrete memory outside of the fixed-size recurrent state and which can be used to retrieve long-term information. The resulting model, HOLA, is shown to be more effective at a set of standard language modeling downstream tasks relative to a number of linear models while also demonstrating improved keypass retrieval skills on the needle-in-a-haystack task.
Critical strengths
The neuroscience motivation is interesting and a good way of motivating the structure introduced in the model.
Critical weaknesses
While the architecture is interesting, it ultimately will come with a number of bottlenecks or issues. The additional memory is not particularly novel, as it simply resembles the use of an additional cached state that can always be appended or conditoned on when generating the next token; one can simply imagine this as simply having a parallel fixed-sized attention operation that goes through a gated mix-in with the recurrent output. This has been explored in the past (ex. Hymba from ICLR2025) and thus the major contribution here should be stated. While the eviction policy for this memory is interesting, it would appear to still act as a non-contiguous KV cache which ultimately has been explored in some capacity in the past.
Additionally, it appears that the approach is designed specifically for linear recurrent models, but even with a bounded memory the issue of possible difficulties of information retention still exist. One can consider an extremely information dense setting, for example code generation, where the model would have to store wide amounts of information that could still exceed the additional memory allocation, thus the model can still be expected to have issues. This is in comparison to attention, which doesn't have such memory issues but is computationally more expensive, but there still exists a middle-ground that is sparse attention approaches or hybrid models (NSA, Hymba, etc.) that should be compared against.
Next, during pre-filling. Since the exact memory is fixed-size and operates using a token-eviction policy, the actual determination of what information goes in and is evicted would naturally follow a serial process where the surprisal would be compared against each token in the memory. However, when this is parallelized, there would require a method of knowing which tokens exist in the cache at all time-steps, then computing the similarity scores/softmax. From my intuition, the way this would function would lead to essentially a large, sparse-attention computation that is difficult to optimize for HBM/SRAM memory loads due to the sparsity, alongside a foward pass of the linear model which in this case cannot actually be fully parallelized since the correct order to determining the correct token to evict requires actually knowing before being able to compute it at step , as otherwise neither the eviction nor the softmax and additative gated update can be done exactly.
In terms of results, performance increases appear modest at best while there is still a computational expense incurred relative to the linear backbone that is being used. Justifying this computational expense would be rather important, particuarly for actual long-context tasks for which this model appears to be designed for. In this case, beyond just passkey retrieval tasks, there should be actual long-context performance results (i.e. ones where context length significantly exceeds the training lengths).
Questions for authors
- Could the authors directly clarify how the actual parallelization of the method is computed, particularly in terms of handling the memory eviction?
- Could the authors provide more conrete evidence that the method is beneficial on long-context settings that resemble real-world tasks, i.e. LongBench, Bamboo, etc.
5 comments
Response to Reviewer ZYxg (Part 2/2)
W3 / Q1. Prefill parallelism and hardware memory access Cache eviction appears sequential because scoring depends on the preceding recurrent state, and parallel attention requires knowing the correct cache membership at each step. Could the authors explain how prefill parallelizes these operations and handles non-contiguous KV access efficiently across HBM and SRAM?
Response:
Parallel cache construction by chunk. Appendix E (lines 718–720) already explains that HOLA's prefill uses fixed write-time scores to construct the prefix-top- cache of preceding tokens for each chunk in a batch. Queries within a chunk share the same historical cache, so there is no need to simulate cache insertion and eviction token by token. Write scores are computed through GDN's chunkwise computation path.
Compact caches and shared reads. For each head, the selected KV are first gathered by chunk into compact arrays. Although the selected tokens occupy noncontiguous positions in the original sequence, the resulting attention inputs have a regular layout: the queries in a chunk share this cache in a historical-attention computation, while a causal local-attention computation handles the current chunk. The gathering cost is shared across the chunk, without each query independently gathering KV from the historical sequence.
HBM/SRAM access. This layout allows attention to use regular matrix tiles: once a KV tile is loaded from HBM into on-chip SRAM, it can be reused by multiple queries. As stated in lines 729–731, we use two FlashAttention varlen calls for the historical cache and chunk-local causal attention, respectively, and merge the results with the appropriate normalization. Thus, noncontiguous source access is concentrated in cache construction, while subsequent reads can use standard tiled attention kernels instead of irregular per-query sparse access over the entire history.
W4 / Q2. Real-world long-context benefits and computational cost The performance gains appear modest relative to the additional computational cost. Could the authors provide results on realistic long-context tasks, such as LongBench or Bamboo, with inputs substantially longer than the training context, to demonstrate that the gains extend beyond synthetic retrieval and justify the overhead?
Response:
Real-world long-context experiments. We have added a LongBench-E comparison of 1B GDN and 1B HOLA, covering real-world document QA, multi-document summarization, classification, and code completion. Because the original maximum training length of 2,048 is substantially shorter than the maximum input length of 32,768 used in the LongBench-E evaluation, we first extended the models' long-context capabilities through a standard continued-pretraining procedure. Only the final checkpoints after continued pretraining are compared below.
Continued pretraining settings. Starting from their respective original checkpoints pretrained on 50B tokens at a context length of 2,048, both models receive an additional 5B tokens (approximately 10% of the original budget) at a training length of 8,192. The additional data come from training shard 24 and subsequent shards of SlimPajama-627B Reupload.
Evaluation settings. We use the full 13-task LongBench-E suite with 3,668 examples, generating 7,336 new predictions across the two models. Actual input lengths range from 277 to 32,768 tokens. We use greedy decoding without a chat template.
| Task | GDN (after) ↑ | HOLA (after) ↑ | HOLA−GDN |
|---|---|---|---|
| Qasper | 5.91 | 6.05 | +0.14 |
| MultiFieldQA-en | 14.75 | 15.55 | +0.80 |
| HotpotQA | 6.84 | 7.46 | +0.62 |
| 2WikiMQA | 9.61 | 10.83 | +1.22 |
| GovReport | 5.88 | 12.91 | +7.03 |
| MultiNews | 1.68 | 11.69 | +10.01 |
| TREC | 23.67 | 12.33 | -11.34 |
| TriviaQA | 30.85 | 31.98 | +1.13 |
| SAMSum | 28.13 | 23.74 | -4.39 |
| PassageCount | 5.09 | 4.50 | -0.59 |
| PassageRetrieval-en | 6.14 | 5.08 | -1.06 |
| LCC | 44.24 | 45.19 | +0.95 |
| RepoBench-P | 36.45 | 36.88 | +0.43 |
Results. HOLA outperforms GDN on 9 of the 13 tasks, with advantages spanning QA, summarization, and code completion. In particular, HOLA/GDN score 12.91/5.88 (+7.03 points) on GovReport and 11.69/1.68 (+10.01 points) on MultiNews. These results provide additional evidence for the usefulness of HOLA's bounded exact memory on real-world long-context tasks, particularly summarization tasks that require integrating information across long documents.
Dear Reviewer ZYxg,
Thank you again for your time and thoughtful review of our paper! As the author–reviewer discussion period is approaching its end tomorrow, we wanted to gently follow up to see if you have had a chance to read our response.
In brief, we have addressed your main concerns by clarifying the intrinsic delta-rule O(1) memory mechanism, detailing the chunkwise FlashAttention execution, and adding a 32k LongBench-E evaluation. We would be very grateful to know whether our response resolves your concerns, or if there are any remaining questions we can quickly clarify before the deadline.
Best regards, Authors
Response to Reviewer ZYxg (Part 1/2)
W1. Novelty relative to hybrid architectures and KV-cache methods Adding bounded attention memory alongside a recurrent state resembles existing hybrid architectures such as Hymba, while selective non-contiguous KV caching has also been explored. Could the authors clarify the main contribution beyond these existing designs?
Response:
Existing hybrid architectures. Lines 78–88 of the Introduction already discuss related hybrid architectures. Common choices include retaining recent tokens in a sliding-window cache and retaining tokens with high cumulative attention.
Our core contribution. The central question we address is which associations to preserve under a limited exact-memory budget. As emphasized by the title, “An Exact Memory for What the Recurrent State Forgets,” we aim to complement the recurrent state with exact memory for information that it may lose. Our key insight is that delta-rule models already compute an innovation term for each input relative to the current state; the actual write magnitude, β∥e∥, provides an intrinsic surprise signal. HOLA directly uses this signal to decide which key/value associations to preserve exactly.
Controlled eviction-policy experiments. We compare HOLA+recency (sliding-window selection) in Tables 1 and 2 and cumulative-attention selection in Table 3. These controls test our core strategy: allocating exact memory according to the recurrent state's intrinsic write signal, rather than solely by position or cumulative attention. The results support the effectiveness of this strategy in improving overall retrieval performance.
W2. Capacity limits and alternative attention architectures A bounded cache may be insufficient for information-dense tasks such as code generation. How does HOLA address this limitation, and how does it compare with sparse-attention or hybrid approaches such as NSA and Hymba, which offer alternatives between recurrent memory and full attention?
Response:
Choosing bounded memory. The motivation for this bounded exact memory is that, with fixed model dimensions and cache budgets, HOLA maintains only the recurrent state, a bounded exact cache, and a local context of fixed size. Its persistent inference state is therefore O(1) with respect to context length, preserving the complexity advantages of linear attention for long sequences.
Resource constraints of sparse attention. Sparse attention and bounded recurrent memory operate under different resource constraints. Methods such as NSA dynamically select historical blocks, reducing the KV accessed by each query, but historical KV must still be retained so that later queries can select them again. Total storage therefore still grows with context length. Storing these KV in tiers across GPU and host memory changes where they reside, but does not make total storage independent of length. The global-attention branches in Hymba likewise retain KV caches that grow with context length.
Real-world long-context evaluation. We have added a LongBench-E comparison of 1B HOLA and GDN after matched continued pretraining at 8K context length. The settings and results are provided in our response to W4/Q2 below.