- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Memory Sparse Attention (MSA). MSA is implemented by post training the latter half of a model to use a different type of attention. In MSA a new Router Q projector is trained to be able to index over the (precomputed) KV’s for the documents in the database. This means that the regular attention in the model and the retrieval use the same backbone. The model is post trained via two objectives, the first a simple language modelling loss and the second an auxiliary loss for the top k selection of documents. MSA is compared to many baselines including regular RAG and MemAgent. MSA outperforms both almost all of the time, on MS Marco MSA achieves good extreme context generalisation.
Critical strengths
- Comprehensive baselines
- Good results on extremely long contexts
- Good explanation of architecture modifications.
Critical weaknesses
- Some baselines are a little old
- I am unsure on how novel the document wise RoPE is.
- The motivation of the capacity of human memory in the introduction is a little odd, considering the number of pretraining tokens used for language models
Score moved 3 to 4 after rebuttal
Questions for authors
- The scores are calculated per layer. Do this mean different documents are queried per layer? If so, how much overhead does this cause?
- How fast is MSA compared to baselines? Especially for
memory interleave, does this slow generation down?
3 comments
Q2: How fast is MSA compared to baselines, and does memory interleave slow generation down?
Against long-context LLMs, MSA is faster, and increasingly so as the memory bank grows: our cost is linear in bank size, theirs is not.
Against RAG, we benchmarked the same-backbone pipeline on one A800 over the same query set. Of the RAG configurations, RAG + rerank top-10 is the one whose accuracy comes closest to MSA's, so we take it as the reference point and report the timing results. On HotpotQA:
| System | First output token | End-to-end p50 | Throughput, C=16 (QPS/GPU) |
|---|---|---|---|
| RAG top-5 (no rerank) | 71 ms | 0.73 s | 7.62 |
| RAG + rerank top-10 | 2,462 ms | 3.05 s | 0.579 |
| MSA () | 196 ms | 2.58 s | 1.87 |
Against that configuration, MSA's first output token arrives 12× sooner and its single-GPU throughput at C=16 is 2.7× higher: RAG + rerank spends 75–93% of its GPU time scoring 100 candidate pairs per query, whereas MSA's routing is part of the forward pass. RAG without reranking is faster still, but less accurate. End-to-end p50 is the median wall-clock time from receiving the question to returning the complete answer, with the memory bank already built; for RAG it covers retrieval, reranking and generation, and for MSA all rounds of routing and generation.
Against the same reference configuration on the other benchmarks (C=1):
| Benchmark | System | First output token | End-to-end p50 |
|---|---|---|---|
| Natural Questions | RAG + rerank top-10 | 1,714 ms | 2.43 s |
| MSA () | 193 ms | 2.05 s | |
| TriviaQA (10M) | RAG + rerank top-10 | 6,561 ms | 7.27 s |
| MSA () | 406 ms | 8.46 s |
The TriviaQA row shows the two systems scaling along different axes. RAG's retrieval is an ANN lookup that barely moves with corpus size (22 ms); its cost is dominated by reranking 100 candidates, which scales with document length — TriviaQA's documents average ~833 tokens against ~140 on the other benchmarks, so reranking alone costs 6,139 ms there against 1,625 ms on Natural Questions. MSA's cost scales with corpus size instead, because routing scores the whole compressed bank, and TriviaQA's 10.49M-token corpus is the largest of the three.
MSA's first output token is an order of magnitude earlier on all three benchmarks. TriviaQA-10M is the one case where its end-to-end latency is higher (8.46 s vs. 7.27 s), and the gap comes from decoding rather than from retrieval: the LLM in the RAG pipeline is served with vLLM, whereas our inference prototype is not, so it decodes more slowly — roughly 41 ms per output token against vLLM's 8.5 ms.
Memory construction is a separate one-time cost, not included above: 141 GPU·s for MSA on HotpotQA against 59 GPU·s for RAG's embedding pass. It is paid once per memory-bank version and amortised over every query served from it, whereas a long-context LLM pays its quadratic prefill on every query.
For memory interleave, we compare the default multi-round setting against a single-round variant on a two-hop dataset. Three rounds are used: the first retrieves the first-hop evidence, the second the second-hop evidence, and the third confirms that the evidence is complete.
| Multi-round (default) | Single-round | |
|---|---|---|
| Total retrieval time | 170.0 s | 52.2 s |
| Per sample | 170 ms | 52 ms |
generate calls per sample | 3 | 1 |
| Retrieval precision / recall | 0.851 / 0.851 | 0.882 / 0.441 |
Retrieval costs roughly 3× as much under memory interleave, as expected from the three generate calls, and recall rises from 0.441 to 0.851 at comparable precision.
The rebuttal is comprehensive and addresses my concerns. Overall, I think this paper should probably be accepted.
We thank the reviewer for the clear summary of the architecture and for the positive assessment. We address the three weaknesses and the two questions below.
W1: Some baselines are a little old
The baselines were chosen to be contemporaneous with our backbone, Qwen3-4B-Instruct-2507, so that comparisons isolate the architecture rather than the generation of the underlying model.
W2: Novelty of document-wise RoPE
Our setting imposes two requirements: the model must train on short contexts and infer on long ones, and, in a memory bank, the relative position between two documents carries no meaning. Under both constraints, assigning independent position IDs per document is a natural choice. We arrived at it independently from these requirements of the memory setting.
W3: The human-memory motivation
The analogy we intend is between the capacity of a human lifetime of memory and the capacity of the memory bank — not the amount of training data. Training establishes the model's capability; inference then operates over a memory bank approaching the scale of a human lifetime of memory, with no further training required. We will revise the wording so that this is stated accurately.
Q1: Scores are computed per layer — are different documents queried per layer, and what is the overhead?
Yes. Each MSA layer performs its own top-k selection, so the set of documents admitted to attention can differ from layer to layer.
We see this as one reason MSA outperforms RAG. Representations differ across depth: shallow layers weigh textual similarity more heavily, while deeper layers weigh higher-level semantic association. Routing at every layer therefore draws on richer information than a single retrieval step over one fixed representation can.
The overhead is reported in Appendix K.5, which gives router latency, throughput and peak GPU memory at three memory-bank sizes. On 2 GPUs, router latency is 1.09 ms at 1M tokens, 8.9 ms at 10M, and 85.5 ms at 100M — close to linear in bank size, consistent with the O(L) routing cost in Appendix C. Sharding the routing keys over 8 GPUs reduces this to 22.31 ms at 100M tokens and lowers per-device memory.