acceptodds
Under review as a conference paper at ICLR 2027

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Abstract

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds 0.025% to analytical FLOPs, with persistent context storage approximately 1/2600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to post-trained Mamba-2-8B and Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear attention, Long-context language modeling, Recurrent language models, Deep Architecture, Episodic memory, Memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_Y5GqOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NoteBank, a training-free memory mechanism for improving the long-context capabilities of linear attention models. It decouples recurrent computation from long-term information storage by archiving selected text spans, retrieving relevant notes using the backbone's native features, and replaying the retrieved text through the original model. Experiments on RULER, LongBench, and MemoryAgentBench demonstrate substantial improvements in long-context retrieval, with limited additional computational overhead.

Critical strengths
  1. The paper presents a simple and practical approach to extending recurrent models' long-context capabilities. Its training-free, plug-and-play design does not require modifying the backbone or introducing a separately trained retriever, making it easy to apply to existing models.

  2. The experimental results demonstrate substantial improvements on million-token retrieval tasks across multiple recurrent architectures. The paper also provides detailed analyses of computational overhead, memory usage, and the respective contributions of evidence retrieval and model adaptation.

Critical weaknesses
  1. The improvements on real-world long-context tasks are relatively limited compared with the substantial gains on RULER. For GDN, training-free NoteBank improves the LongBench-v1 score only from 15.49 to 16.07, while most of the gains with LoRA are also observed without NoteBank. This raises questions about how effectively improved evidence access translates into better downstream task performance.

  2. The native retrieval mechanism is not consistently effective. BM25 substantially outperforms native routing on some experiments, particularly for Mamba-2, suggesting that the proposed routing strategy leaves considerable room for improvement. This weakens the case for the specific retrieval mechanism as a key technical contribution.

  3. Although the recurrent state remains fixed-size, the external archive and address storage grow linearly with context length. Thus, NoteBank substantially reduces memory overhead compared with a full Transformer KV cache, but does not achieve constant total memory usage.

Questions for authors
  1. How can the authors explain the substantial gap between the improvements on RULER and those on LongBench? Can authors offer evidence that demonstrates NoteBank improves downstream reasoning beyond retrieving relevant information?

  2. Given that BM25 outperforms native routing in several experiments, what are the key limitations of the proposed routing mechanism, and why should it be preferred over conventional retrieval methods?

Reviewer_d4ttOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method named NoteBank, which aims to address the memory limitations of linear attention. I think it is essentially a RAG-variant with native features. It first selects text spans from the context and stores them as notes; each note keeps raw token IDs plus per-layer-head compact native-query addresses (similar to a group of text embeddings). At query time, the question's native query features are used to retrieve the most relevant notes by per-channel matching and voting. The retrieved notes are then replayed through the frozen backbone before the question is answered.

Critical strengths
  1. I agree with the direction of addressing the memory limitations of linear attention. Fixed-size memory is indeed a practical constraint, and NoteBank provides a reasonable way to circumvent it by storing and retrieving relevant text instead of forcing the recurrent state to retain everything.
  2. The experiments cover RULER, LongBench v1/v2, and MemoryAgentBench, including retrieval at 1M tokens. The RULER result is quite strong: GDN improves from 1.7% to 70.3% at 1M tokens.
  3. The paper provides detailed analysis and experiment settings, which helps in understanding the effectiveness and limitations of the proposed method.
Critical weaknesses
  1. Mismatch of baselines. I think the paper should compare with more retrieval-based methods, such as a RAG baseline and similar methods for linear attention [1]. The current comparison is limited. Table 5 includes Memory Caching and MARCH, but they use different checkpoints and pretraining protocols, so they are reference points rather than matched controls. In addition, for Mamba-2-8B, the main table reports Plain 0.0 to NB-TF 14.7, while the appendix shows that a clean-start baseline already reaches 13.0; the actual gain of NoteBank over clean start is only +1.7. This baseline mismatch significantly weakens the headline claim.

  2. The explanation is somewhat weak. For example, why NoteBank is better than other retrieval-based methods, and why the native addresses are better than normal text embeddings, is not clearly justified. The paper's own Table 8 shows that BM25 routing is stronger than native NB routing on RULER (GDN 64.0 vs. 59.8; Mamba-2 52.7 vs. 37.6), and on LongBench for Mamba-2 (18.66 vs. 14.25).

  3. Related work is placed in the appendix. This makes it less accessible for readers who want to quickly understand the differences between this paper and prior work. This is mainly a presentation issue, but I think it should be addressed.

Questions for authors

See weakness

Reviewer_rL1yOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a lightweight "note bank" to store interesting spans in a long-context block, which can then be retrieved when processing the question. It shows promising results on a variety of long-context benchmarks, including surprising performance gains on multi-doc QA.

Critical strengths
  • Clear presentation, nice experiment setup, easy to read, some surprising results (e.g., multi-doc QA) given the simplicity of the technique.
  • I like that it involves no further training.
  • The inclusion of the oracle baseline is a nice touch and it's interesting that GDN performs so much better than Mamba. The presentation of this baseline in Figure 5 is excellent.
  • The method is nice, and I think that extensions to this work could be a valuable playground for other academic researchers.
  • The ablations in the appendix are quite thorough and informative. Good presentation.
Critical weaknesses
  • My main concern is that many of the tasks studied involve unusual/surprising strings inserted inside filler text. I believe the method is particularly suited to picking up and storing these strings. I think that if the total length of the concatenated spans retrieved exceeds the model's actual memory within its hidden state, the method will fall apart. This would possibly require modifying RULER slightly to make the NIAH tasks substantially harder (many keys, many values). I would be curious to see at what point the method definitively breaks on (multi-)NIAH.
  • This type of explicit / brittle memory store might not be so relevant given the trend of giving models access to tool calls. I could imagine that lots of these gains could be achieved by instrumenting the linear attention models with grep. That said, I still find the tool-free (single-shot) setting to be interesting and important.
Questions for authors

Could you provide examples of the retrieved spans for various settings (particularly QA)?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and reviewers for their careful and constructive evaluation of our work. We are encouraged that all reviewers are excited about the paper and recognize the paper’s strong long-context results and broader potential to inspire further research. We would like to use this final message to emphasize what we believe is the paper’s most important scientific implication.

Our central finding is that the apparent long-context collapse of recurrent LLMs is not a limitation of recurrent computation; it arises from forcing a single fixed-size state to serve simultaneously as the model’s computational state and its long-term memory. This distinction changes how we interpret the failure of linear models at extreme context lengths. A frozen Gated DeltaNet achieves only 1.7% mean retrieval accuracy at 1 million tokens, yet reaches 70.3% once archival memory is separated from recurrent computation, without changing a single backbone weight. The model already possessed the computation needed to use distant information; what it lacked was continued access to that information. In this sense, our results suggest that collapse of recurrent LLMs in long-context settings is primarily an access problem.

We believe this finding changes the research landscape for linear and recurrent LLMs. Much of the field has treated the fixed recurrent state as both the source of linear models’ efficiency and the fundamental ceiling on their long-context capability, motivating increasingly sophisticated ways to compress more history into that state. Our results reveal a different path: the recurrent state need not carry the entire burden of history at all. Once recurrent computation is freed from the requirement to preserve all potentially useful past information and history can instead be selectively restored, models that appear to fail catastrophically at extreme context lengths can recover strong million-token capability while retaining the efficiency advantages of recurrence.

NoteBank is one realization of this principle, but we believe the broader implication is a new research direction: frontier linear models need not solve long context solely by learning ever more powerful recurrent memories; they can instead preserve access to past context outside the evolving computational state and restore it through the model’s own computation when needed. If this principle continues to scale, the long-standing trade-off between efficient recurrence and strong long-context access may not be fundamental after all.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.