acceptodds
Under review as a conference paper at ICLR 2027

MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

Abstract

Long-term memory is fundamental to intelligence, yet enabling AI to process lifetime-scale information on the order of 100M tokens remains challenging. Full-attention architectures limit LLMs to roughly 1M tokens, and existing extensions such as linear attention, fixed-size memory, or external storage methods suffer from accuracy loss, growing latency, limited memory adaptability, or poor end-to-end optimization. These limitations restrict scalability and hinder applications including large‑corpus summarization, stable digital twins, and long‑history agent reasoning. We present Memory Sparse Attention (MSA), an end-to-end trainable, efficient, and massively scalable memory model framework. Through core innovations including scalable sparse attention architecture and document‑wise RoPE, MSA achieves linear complexity in both training and inference while maintaining exceptional precision stability, exhibiting less than 9% degradation when scaling from 16K to 100M tokens. Furthermore, KV cache compression, combined with Memory Parallel during inference, enables 100M tokens inference on GPUs. In addition, we propose a Memory Interleave mechanism that effectively facilitates complex multi‑hop reasoning across scattered memory segments. MSA significantly surpasses frontier language models, state-of-the-art (SOTA) RAG systems, and leading memory agents in long-context benchmarks. These results demonstrate that by decoupling memory capacity from reasoning, MSA provides a scalable foundation to endow general-purpose models with intrinsic, lifetime-scale memory. Model available at https://huggingface.co/Anoy123423123/MSA-4B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, memory model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZWQGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Memory Sparse Attention (MSA). MSA is implemented by post training the latter half of a model to use a different type of attention. In MSA a new Router Q projector is trained to be able to index over the (precomputed) KV’s for the documents in the database. This means that the regular attention in the model and the retrieval use the same backbone. The model is post trained via two objectives, the first a simple language modelling loss and the second an auxiliary loss for the top k selection of documents. MSA is compared to many baselines including regular RAG and MemAgent. MSA outperforms both almost all of the time, on MS Marco MSA achieves good extreme context generalisation.

Critical strengths
  1. Comprehensive baselines
  2. Good results on extremely long contexts
  3. Good explanation of architecture modifications.
Critical weaknesses
  1. Some baselines are a little old
  2. I am unsure on how novel the document wise RoPE is.
  3. The motivation of the capacity of human memory in the introduction is a little odd, considering the number of pretraining tokens used for language models

Score moved 3 to 4 after rebuttal

Questions for authors
  1. The scores are calculated per layer. Do this mean different documents are queried per layer?
If so, how much overhead does this cause?
  2. How fast is MSA compared to baselines? Especially for memory interleave, does this slow generation down?
3 comments

We thank the reviewer for the clear summary of the architecture and for the positive assessment. We address the three weaknesses and the two questions below.


W1: Some baselines are a little old

The baselines were chosen to be contemporaneous with our backbone, Qwen3-4B-Instruct-2507, so that comparisons isolate the architecture rather than the generation of the underlying model.

W2: Novelty of document-wise RoPE

Our setting imposes two requirements: the model must train on short contexts and infer on long ones, and, in a memory bank, the relative position between two documents carries no meaning. Under both constraints, assigning independent position IDs per document is a natural choice. We arrived at it independently from these requirements of the memory setting.

W3: The human-memory motivation

The analogy we intend is between the capacity of a human lifetime of memory and the capacity of the memory bank — not the amount of training data. Training establishes the model's capability; inference then operates over a memory bank approaching the scale of a human lifetime of memory, with no further training required. We will revise the wording so that this is stated accurately.


Q1: Scores are computed per layer — are different documents queried per layer, and what is the overhead?

Yes. Each MSA layer performs its own top-kk selection, so the set of documents admitted to attention can differ from layer to layer.

We see this as one reason MSA outperforms RAG. Representations differ across depth: shallow layers weigh textual similarity more heavily, while deeper layers weigh higher-level semantic association. Routing at every layer therefore draws on richer information than a single retrieval step over one fixed representation can.

The overhead is reported in Appendix K.5, which gives router latency, throughput and peak GPU memory at three memory-bank sizes. On 2 GPUs, router latency is 1.09 ms at 1M tokens, 8.9 ms at 10M, and 85.5 ms at 100M — close to linear in bank size, consistent with the O(L)\mathcal{O}(L) routing cost in Appendix C. Sharding the routing keys over 8 GPUs reduces this to 22.31 ms at 100M tokens and lowers per-device memory.

Q2: How fast is MSA compared to baselines, and does memory interleave slow generation down?

Against long-context LLMs, MSA is faster, and increasingly so as the memory bank grows: our cost is linear in bank size, theirs is not.

Against RAG, we benchmarked the same-backbone pipeline on one A800 over the same query set. Of the RAG configurations, RAG + rerank top-10 is the one whose accuracy comes closest to MSA's, so we take it as the reference point and report the timing results. On HotpotQA:

SystemFirst output tokenEnd-to-end p50Throughput, C=16 (QPS/GPU)
RAG top-5 (no rerank)71 ms0.73 s7.62
RAG + rerank top-102,462 ms3.05 s0.579
MSA (P=64P=64)196 ms2.58 s1.87

Against that configuration, MSA's first output token arrives 12× sooner and its single-GPU throughput at C=16 is 2.7× higher: RAG + rerank spends 75–93% of its GPU time scoring 100 candidate pairs per query, whereas MSA's routing is part of the forward pass. RAG without reranking is faster still, but less accurate. End-to-end p50 is the median wall-clock time from receiving the question to returning the complete answer, with the memory bank already built; for RAG it covers retrieval, reranking and generation, and for MSA all rounds of routing and generation.

Against the same reference configuration on the other benchmarks (C=1):

BenchmarkSystemFirst output tokenEnd-to-end p50
Natural QuestionsRAG + rerank top-101,714 ms2.43 s
MSA (P=64P=64)193 ms2.05 s
TriviaQA (10M)RAG + rerank top-106,561 ms7.27 s
MSA (P=64P=64)406 ms8.46 s

The TriviaQA row shows the two systems scaling along different axes. RAG's retrieval is an ANN lookup that barely moves with corpus size (22 ms); its cost is dominated by reranking 100 candidates, which scales with document length — TriviaQA's documents average ~833 tokens against ~140 on the other benchmarks, so reranking alone costs 6,139 ms there against 1,625 ms on Natural Questions. MSA's cost scales with corpus size instead, because routing scores the whole compressed bank, and TriviaQA's 10.49M-token corpus is the largest of the three.

MSA's first output token is an order of magnitude earlier on all three benchmarks. TriviaQA-10M is the one case where its end-to-end latency is higher (8.46 s vs. 7.27 s), and the gap comes from decoding rather than from retrieval: the LLM in the RAG pipeline is served with vLLM, whereas our inference prototype is not, so it decodes more slowly — roughly 41 ms per output token against vLLM's 8.5 ms.

Memory construction is a separate one-time cost, not included above: 141 GPU·s for MSA on HotpotQA against 59 GPU·s for RAG's embedding pass. It is paid once per memory-bank version and amortised over every query served from it, whereas a long-context LLM pays its quadratic prefill on every query.

For memory interleave, we compare the default multi-round setting against a single-round variant on a two-hop dataset. Three rounds are used: the first retrieves the first-hop evidence, the second the second-hop evidence, and the third confirms that the evidence is complete.

Multi-round (default)Single-round
Total retrieval time170.0 s52.2 s
Per sample170 ms52 ms
generate calls per sample31
Retrieval precision / recall0.851 / 0.8510.882 / 0.441

Retrieval costs roughly 3× as much under memory interleave, as expected from the three generate calls, and recall rises from 0.441 to 0.851 at comparable precision.

The rebuttal is comprehensive and addresses my concerns. Overall, I think this paper should probably be accepted.

Reviewer_T2b2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces Memory Sparse Attention (MSA), a retrieval-oriented architecture intended to extend language-model memory to 100M tokens. Documents are encoded separately into compressed KV representations, and a learned router selects documents whose KVs are attended to during generation. Memory Interleave further allows the model to emit document identifiers and fetch the corresponding source text for later reasoning. The authors evaluate MSA on document-centric QA and retrieval benchmarks with memory banks up to 100M tokens, report gains over baselines, and provide ablations on continual pre-training, Memory Interleave, and original-text loading.

Critical strengths
  1. An interesting combination of retrieval and attention for large-scale memory access. MSA couples learned document selection with compressed KV memory and sparse attention. By encoding documents independently and retrieving only relevant memory at generation time, it offers a workable route to large document collections without attending over the entire memory bank. This is promising for retrieval-heavy use cases and suggests a useful direction for coupling external memory to language models.
Critical weaknesses
  1. The generality claims exceed the evidence, which remains within document-centric retrieval.
    MSA is framed as a scalable sparse attention architecture supporting 100M-token memory, but its mechanism relies on per-document encoding and document-level retrieval. The 100M-token figure mainly reflects addressable memory-bank capacity, not demonstrated joint modeling of arbitrary dependencies over a 100M-token context. The benchmarks are also mostly document-centric QA and retrieval tasks that fit the architecture’s assumptions. It remains unclear how MSA would perform on general long-context workloads—such as long-horizon reasoning or multi-turn agent interaction—where evidence is spread across a continuous history and segments must be tightly coupled.

  2. The contribution of latent document-KV attention beyond retrieval is unclear.
    Table 4 shows that disabling original-text loading lowers the average QA score from 3.694 to 2.325, suggesting that access to retrieved source text remains essential. This raises the concern that MSA may largely amount to a stronger RAG pipeline with better retrieval rather than gaining substantial benefit from latent KV attention. A retrieval-only control is missing: it would keep the learned router, document-ID generation, and original-text loading while disabling attention to compressed document KVs.

Questions for authors
  1. Please evaluate MSA on long-context tasks without clear document boundaries or with substantial cross-segment dependencies.

  2. Please compare against a retrieval-only variant that preserves the learned router, document-ID generation, and original-text loading but disables attention over compressed document KVs.

2 comments

We thank the reviewer for a careful and well-informed review. Both weaknesses turn on where MSA sits relative to general long-context models and to RAG, and on what latent KV attention contributes beyond retrieval. We address each below.


W1 / Q1: Scope of the generality claims; tasks without clear document boundaries

Our memory context is indeed not a general long context, and we will describe it more precisely — but it is not a pure addressable memory bank either

The reviewer is right that MSA's memory context is not the same thing as a general long context, and we will state this more precisely in the paper rather than leaving the two to be conflated. MSA is best matched to a setting with a fixed, large external memory bank and a large volume of queries against it, but it also handles ordinary long-context tasks normally — the NIAH evaluation in the paper is one such case. The fairest characterisation is that this is a memory context optimised for memory settings rather than a context in the general sense, and we agree with that.

Cross-document interaction

MSA does support interaction across memory units, by two mechanisms:

  1. Across layers. The compressed KVs of the Top-kk documents routed at one layer enter the computation of the query-side hidden states at subsequent layers. Evidence selected early therefore conditions what the model represents, and what it routes to, later in the stack — the documents are not processed in isolation from one another.
  2. Across rounds. Memory interleaving lets the model issue a further retrieval conditioned on evidence it has already gathered, which is what supports multi-hop chains.

Tasks without clear document boundaries, or with substantial cross-segment dependencies

Two evaluations already in the paper cover these two cases, one each.

NIAH — no clear document boundaries. RULER's haystack is a continuous stream of text that comes with no document boundaries at all; nothing in the data marks where one unit ends and the next begins, so the units MSA routes over are segments we impose rather than self-contained documents. Accuracy holds from 32K to 1M tokens (94.84% at 1M) and extends to 100M in Appendix K.6. The memory unit is a granularity of segmentation, not a semantic property the data has to supply.

LoCoMo — substantial cross-segment dependencies. Multi-turn dialogue with interrelated content, where the evidence for a single question regularly spans several documents rather than sitting inside one (Appendix K.4). The distribution appears in neither our continual pre-training nor our post-training data, and MSA-4B is evaluated zero-shot, without fine-tuning:

MethodJudge score
RAG@254.8
RAG@361.5
Mem066.9
MSA-4B (zero-shot)63.4

MSA scores above RAG at comparable depth, i.e. above a pipeline that treats each retrieved passage as an independent document, and below Mem0, a system built specifically for conversational memory.


W2 / Q2: The contribution of latent document-KV attention beyond retrieval

The reviewer is right about the configuration used in the submission

Under the setting used in the submission, the compressed KVs are indeed used mainly for retrieval. We use a large pooling kernel there, so the representation is lossy, and for QA the lost detail is exactly what the answer consists of. This is why source text is fetched, and why Table 4 drops when it is not. What differs below is the configuration, not the model.

This is not a necessary consequence of the architecture

We have run some exploration on this point, which we think addresses part of the reviewer's concern. We tried a pooling size of 16 while reducing the number of routing layers to a quarter of the original, so that total complexity is close to unchanged, and added a partial reconstruction task during pre-training. Two variants were evaluated: one in which the model attempts to reconstruct the source text itself, and one in which it answers directly. LLM-as-a-judge, 0–5 scale:

BenchmarkSubmitted setting (doc-ID + source text)w/o source text (paper Table 4)Exploration: reconstruct source textExploration: answer directly
DuReader4.1552.1864.1374.017
NarrativeQA3.3951.5473.0402.792
HotpotQA4.0612.2973.9113.742

Both variants come close to the setting that fetches source text, and both are far above the w/o source text ablation in the paper, which uses the same router. This indicates that at a smaller compression ratio, and with appropriate training, MSA does not depend on fetching source text the way RAG does.

We state the limits of this plainly: the result depends on the compression ratio of the latent KVs, and because information is genuinely lost in compression, these variants still do not fully match fetching the source text — they approach it.

Thank you for your response. The rebuttal has addressed some of my concerns, particularly regarding the source-text ablation, and I have raised my score to 3.

Reviewer_xHKEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on one of the core challenges in large language models: the scalability of long-term memory. To tackle this challenge, the authors proposed a memory architecture, so-called MSA, that enables LLMs to answer questions using a document collection as large as 100 million tokens, yet limits expensive attention to selected memory. It appears that the authors’ main contribution is to integrate learned memory retrieval into the model’s attention layers by relying on reusable, compressed document representations. The authors illustrate that their innovative architecture has four main components: compressed document memory, learned sparse routing, document-wise positional encoding, and memory interleaving for multi-hop reasoning.

For validation, the authors test MSA through comparison with existing systems, ablation studies, memory-scaling experiments, efficiency measurements, and generalization checks. MSA is compared with RAG systems and long-context models on question-answering and retrieval benchmarks. Ablation studies examine the contributions of iterative retrieval, continual pretraining, original document text, and longer-context training. They also test memory scaling up to 100 million tokens, measure latency and memory usage, and conduct generalization and data-contamination checks. Overall, the results support MSA’s effectiveness and scalability, although some architectural choices are not evaluated independently.

Critical strengths
  • Focuses on a core challenge: scaling LLMs while preserving efficient access
  • Combines learned retrieval, compressed memory, and iterative evidence gathering in one model.
  • Demonstrates strong QA performance and retrieval from memory banks up to 100M tokens.
  • Provides ablations, efficiency measurements, and additional generalization checks.
Critical weaknesses
  • Some key choices are not independently validated, including document-wise RoPE and compression settings
  • Substantial training resources are still needed, and baseline comparisons neglect a bit the training budgets.
Questions for authors
  • Could you please separately ablate document-wise RoPE, learned routing, and KV compression? How sensitive are performance and efficiency to compression size and routing depth?
  • Beyond the reported complexity analysis and inference measurements, how do MSA’s total training, memory-construction, and serving cost compare with standard LLMs and RAG systems under comparable conditions?

I am not an expert in this field, but I would like to increase my rating if the authors can properly address my concerns and questions.

7 comments

We thank you warmly for the careful reading, and for two questions that land exactly on what the submission left unverified — these are the right things to ask of this paper. We are glad to answer both: with the ablations we ran while designing the architecture, and with a set of cost measurements made for this response. We hope the answers below resolve your concerns, and we would be most grateful for any further response.

W1 / Q1: Independent ablations of the architectural components

Learned routing and KV compression were each ablated during our design phase, before the full-scale training run, in order to fix these two hyperparameters; those experiments were not reported in the submission, and we give them in full below. Document-wise RoPE is a component of a different kind, and (c) explains why it cannot be ablated in the same sense.

(a) Learned routing

We compare two ways of producing the routing key KˉR\bar{K}^R:

  • Shared — routing reuses the backbone's attention projection weights, i.e. no dedicated routing parameters are learned;
  • Decoupled — routing uses an independent, lightweight router projector with learnable per-head weights, trained with the auxiliary routing loss.

Each variant requires its own continual pre-training, so both were trained end-to-end on a Qwen3-0.6B backbone under an identical budget. We report gt_hit_rate, the fraction of queries for which the ground-truth document is selected by the router.

RouterRetrieval64KCMTEB-DuRetrieval
Shared (attention weights, no learned router)0.70200.5926
Decoupled (learned router projector)0.84340.6753

The learned router improves routing accuracy by 20.1% relative on Retrieval64K and 14.0% on CMTEB-DuRetrieval.

(b) KV compression size

We ablate the chunk-wise pooling kernel size PP, which sets the compression ratio of the cached memory, on NQ320k at two scales: Qwen3-0.6B (32k memory) and the 4B model (64k memory).

ModelPPgt_hit_rateAR retrieval recall@1Layer routing recall@10
0.6B320.43670.16350.5481
0.6B640.41630.11560.5498
4B320.73650.60390.8028
4B640.73580.60000.7937

At the 4B scale the gap between P=32P=32 and P=64P=64 is within noise (gt_hit_rate 0.7365 vs. 0.7358; recall@1 0.6039 vs. 0.6000), so P=64P=64 sits on a flat region of the quality curve rather than at a tuned optimum. The smaller 0.6B model is more sensitive (−4.7% in gt_hit_rate), which confirms that the compression is lossy: the 0.6B model carries less usable information to begin with, so compressing it further costs proportionally more. On the MSA-4B used in the paper, this parameter is not sensitive.

On the efficiency side, PP enters every cost term through 1/P1/P (Appendix C): the cached memory footprint and the online routing cost are both O(L/P)\mathcal{O}(L/P), and the sparse-generation context is M+kG/PM + kG/P tokens. Moving from P=32P=32 to P=64P=64 therefore halves storage and routing compute and shortens the attended context by 2×, at no measurable cost in retrieval quality at the 4B scale.

(c) Document-wise RoPE

This one we did not ablate, and we are glad of the chance to say why, because the reason is not that we overlooked it. Document-wise RoPE is not a tunable component of MSA — it is what makes the training recipe possible at all. Three things rest on it:

  • Training on few documents, inference over many. Compute limits training to a handful of documents per sequence, while inference runs over banks of up to 100M tokens. Global position IDs increase monotonically across the concatenated bank, so a document's positional semantics would depend on how many documents happen to precede it. Giving every document its own IDs starting from 0 decouples those semantics from bank size: a document sitting at the ten-millionth token is encoded exactly as its counterparts were during training.
  • Length extrapolation. For the same reason, no position index ever exceeds the length of a single document, however large the bank grows. The inference context stays inside the range the backbone was trained on, which is what lets accuracy hold to within 9% as memory scales from 16K to 100M tokens.
  • A memory bank that can actually be edited. Because each document is encoded independently, its entries carry no dependency on any other document, and the bank supports insertion, deletion and replacement at document granularity without touching a single existing entry (Appendix J). Under a global RoPE, every entry's representation is a function of its offset in the concatenation, so inserting or deleting one document invalidates the cached KV of everything after it and the bank has to be rebuilt.

Removing document-wise RoPE is therefore not a simple component ablation for MSA: it would make the extrapolation ability hard to train in the first place. The choice is also fixed at the very start of continual pre-training — the router keys, the compressed memory and the backbone's adaptation to reading parallel documents are all learned under that convention — so a global-RoPE variant would mean repeating the entire 158.95B-token stage, which cost and time make difficult to carry out within the discussion period. We hope you will understand.

(d) Routing depth

We would like to answer this one properly, and want to be sure we have the intended sense. "Routing depth" could refer to (i) which layers carry MSA routing, (ii) the number of documents kk selected per layer, or (iii) the number of memory-interleaving rounds. We would be grateful if you could indicate which was meant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.