- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes NoteBank, a training-free memory mechanism for improving the long-context capabilities of linear attention models. It decouples recurrent computation from long-term information storage by archiving selected text spans, retrieving relevant notes using the backbone's native features, and replaying the retrieved text through the original model. Experiments on RULER, LongBench, and MemoryAgentBench demonstrate substantial improvements in long-context retrieval, with limited additional computational overhead.
Critical strengths
-
The paper presents a simple and practical approach to extending recurrent models' long-context capabilities. Its training-free, plug-and-play design does not require modifying the backbone or introducing a separately trained retriever, making it easy to apply to existing models.
-
The experimental results demonstrate substantial improvements on million-token retrieval tasks across multiple recurrent architectures. The paper also provides detailed analyses of computational overhead, memory usage, and the respective contributions of evidence retrieval and model adaptation.
Critical weaknesses
-
The improvements on real-world long-context tasks are relatively limited compared with the substantial gains on RULER. For GDN, training-free NoteBank improves the LongBench-v1 score only from 15.49 to 16.07, while most of the gains with LoRA are also observed without NoteBank. This raises questions about how effectively improved evidence access translates into better downstream task performance.
-
The native retrieval mechanism is not consistently effective. BM25 substantially outperforms native routing on some experiments, particularly for Mamba-2, suggesting that the proposed routing strategy leaves considerable room for improvement. This weakens the case for the specific retrieval mechanism as a key technical contribution.
-
Although the recurrent state remains fixed-size, the external archive and address storage grow linearly with context length. Thus, NoteBank substantially reduces memory overhead compared with a full Transformer KV cache, but does not achieve constant total memory usage.
Questions for authors
-
How can the authors explain the substantial gap between the improvements on RULER and those on LongBench? Can authors offer evidence that demonstrates NoteBank improves downstream reasoning beyond retrieving relevant information?
-
Given that BM25 outperforms native routing in several experiments, what are the key limitations of the proposed routing mechanism, and why should it be preferred over conventional retrieval methods?
General Response
We thank the Area Chair and reviewers for their careful and constructive evaluation of our work. We are encouraged that all reviewers are excited about the paper and recognize the paper’s strong long-context results and broader potential to inspire further research. We would like to use this final message to emphasize what we believe is the paper’s most important scientific implication.
Our central finding is that the apparent long-context collapse of recurrent LLMs is not a limitation of recurrent computation; it arises from forcing a single fixed-size state to serve simultaneously as the model’s computational state and its long-term memory. This distinction changes how we interpret the failure of linear models at extreme context lengths. A frozen Gated DeltaNet achieves only 1.7% mean retrieval accuracy at 1 million tokens, yet reaches 70.3% once archival memory is separated from recurrent computation, without changing a single backbone weight. The model already possessed the computation needed to use distant information; what it lacked was continued access to that information. In this sense, our results suggest that collapse of recurrent LLMs in long-context settings is primarily an access problem.
We believe this finding changes the research landscape for linear and recurrent LLMs. Much of the field has treated the fixed recurrent state as both the source of linear models’ efficiency and the fundamental ceiling on their long-context capability, motivating increasingly sophisticated ways to compress more history into that state. Our results reveal a different path: the recurrent state need not carry the entire burden of history at all. Once recurrent computation is freed from the requirement to preserve all potentially useful past information and history can instead be selectively restored, models that appear to fail catastrophically at extreme context lengths can recover strong million-token capability while retaining the efficiency advantages of recurrence.
NoteBank is one realization of this principle, but we believe the broader implication is a new research direction: frontier linear models need not solve long context solely by learning ever more powerful recurrent memories; they can instead preserve access to past context outside the evolving computational state and restore it through the model’s own computation when needed. If this principle continues to scale, the long-standing trade-off between efficient recurrence and strong long-context access may not be fundamental after all.