acceptodds
Under review as a conference paper at ICLR 2027

MonoRoute: One Symmetric Low-Rank Search Scales Attention Memory to 1B Tokens

Abstract

We introduce MonoRoute to combine the answer quality of attention-based retrieval with high question-answering throughput. Memory Sparse Attention (MSA) integrates document selection and multi-hop reasoning within a transformer, but repeatedly searches the corpus in 18 layers using 1024-dimensional routing keys. MonoRoute replaces these searches with a single routing layer per hop; later layers reuse its selection while attending to their own document key-value states. Projection symmetrization constructs a shared query and key orthogonal projection, enabling rank reduction to 128 dimensions without retraining and cutting routing-key storage eightfold. Together, these changes reduce leading-order similarity-scoring memory and compute by 144×, allowing us to construct a corpus index at only 4 bytes per token, while preserving iterative retrieval. Across five QA datasets, MonoRoute exceeds FlashRAG on all four reported answer-quality metrics and achieves 3.1-6.5× its throughput on HotpotQA and 2WikiMultiHopQA at batch size 16, including 3.1× at one billion tokens. These results support attention-based retrieval as an accurate, high-throughput alternative to retrieval-augmented generation (RAG).

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, long-context inference, sparse attention, approximate nearest neighbor search, efficient llm inference, retrieval-augmented generation, kv cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z9CDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MonoRoute starts from the observation that Memory Sparse Attention (MSA) mechanism is inefficient especially in memory consumption and computation. The authors introduce two ways to mitigate it; diminishing size of routing key from 1024 to 128 without retraining, and reuse 18th layer’s search result in latter layers instead of re-compute them repeatedly. For dimension reduction, they use a symmetrized method instead of singular value decomposition (SVD), so that hidden state and routing key share projection to common space. To reuse the search result, they suggest this based on the observation that searched indexes from latter layer are similar to the first (18th layer) one.

Compare to the previous works like FlashRAG, MonoRoute outperforms both in quality and throughput (especially in batch mode). They compare MonoRoute with FlashRAG on five QA datasets, and ablate each change against the original MSA-4B.

Critical strengths
  • The main strength of this work is clarity. The authors address the concrete problem of MSA and suggest practical solutions without training. They observe that routing dominates MSA’s cost (88% of generation time on multi-hop QA, with routing keys occupying 41 of 92 GiB of GPU memory), and that the top-ranked documents of later layers are mostly already selected by the first routing layer. Then they propose the straightforward solutions for each problem; dimension reduction for routing key by symmetric projection, and reuse the searched indexes rather than recompute for all layers. This is consequential because routing latency drops by 280x and 1B-token served on a 2 GPU node. It makes MSA more practical.
Critical weaknesses
  • The authors propose symmetrization over SVD, but they haven’t showed sound result that symmetrization outperforms SVD in term of search quality. They state plain SVD already preserves full accuracy, and both symmetrization and SVD have the same cost (both storage and compute). Symmetrization requires shared projection matrix, so it is compatible with ANN indices, but in the experiment, there is no result on it because all experiments use exact flat search.

  • All their observations and experiments only based on single checkpoint (MSA-4B). For instance, similarity of searched indexed among layers clearly happens in 4B model. However, MSA trains routing layer with the the same contrastive loss on the same dataset, so this agreement may be caused by MSA’s training rather than the general property. Besides, there is no evidence that it holds when model size, architecture, and training dataset vary.

Questions for authors
  • There is no comparison between the symmetrized projection and a rank-128 SVD factorization, although the authors address that the latter already preserves accuracy. Could you add an experiment to compare 128 rank SVD and the symmetrized projection both under the same flat search and ANN search? The latter one is because they claim the advantage of symmetrization is compatibility with graph-based search.

  • MonoRoute and FlashRAG have different number of documents. According to Sec. 4.1 and Table 10, FlashRAG approximately refers 10 documents (3 documents per round, and about 3 rounds) while MonoRoute refers about 16 per hop, about 24 ~ 32 candidates per question. Since MonoRoute selects documents from 16 candidates, it is still unclear the actual number it reads. Could the authors report the number of documents that MonoRoute refers and compare against a RAG baseline with similar document budgets such as FlashRAG with 16 retrieved candidates and rerank?

4 comments

We thank the reviewer for their thorough review. Please find our response to the critical weaknesses and to your 2 questions (Q1, Q2) below:

Symmetrization versus SVD under flat search (Q1). The reviewer is right, and the revision reverses the emphasis (item 1 in our common response above). Under the same flat search, rank and key memory, the truncated SVD keeps the quality of the full-width router on both ablation benchmarks (both calibrated: judge within 0.02, citation recall within 0.003), while the symmetric projection loses 0.38 judge points to it on KILT-HotpotQA. MonoRoute-SVD is therefore the default, the title no longer names symmetry, and the sentence that SVD preserves accuracy is now backed by the ablation. The symmetric projection stays as a variant for single-passage questions, where it cites fewer documents and is faster.

Symmetrization versus SVD under ANN search (Q1). We did not add this comparison, and the revision says why (Section 4.2): an earlier version of the system routed through in-memory DiskANN graphs over the routing keys of every searching layer, one CPU graph walk per query token, and gained little beyond a batch of 8 questions, whereas the single-layer rank-128 scan is one batched matrix product on the GPU and keeps scaling with the batch. At 100M tokens that scan costs 5-9 ms per routing pass, a few percent of a question's time, so an index could not change the reported throughput measurably, and an approximate search cannot raise a router's quality above its exact result. The compatibility of the shared projection with graph or quantized indexes is stated as an expectation for corpora where the scan, linear in the corpus size, comes to dominate (Section 5.5), not as a result.

Single checkpoint; agreement across layers. We agree that the agreement of the routing layers may follow from MSA training every routing layer with the same objective on the same data rather than from a general property, and we cannot test other sizes or architectures: MSA-4B is the only released MSA checkpoint. The revision therefore presents the coverage table as motivation only (Appendix A.5) and rests the single-layer claim on the end-to-end ablation: on 1,000 KILT-HotpotQA questions the single-layer router matches MSA within the confidence interval, and on MS MARCO it loses 0.12 judge points (Figures 2-3). Section 5.5 lists the single checkpoint as a limitation.

Document budgets (Q2). MonoRoute attends to the pooled KV states of 16 candidates per retrieval round and injects the full text only of the documents it cites: 1.7 documents per question on MS MARCO, where it retrieves once, and 1.6 on KILT-HotpotQA over 1.6 rounds (Appendix A.9, "Per-question work", and Figures 7-8); it takes 2 rounds on 2WikiMultiHopQA and MuSiQue. FlashRAG-RR, the baseline the reviewer asked for, retrieves 16 candidates, reranks them and reads 3 per IRCoT round, over 2.6 to 3.0 rounds on the multi-hop benchmarks; HippoRAG-2 reads its 5 best passages (item 4 in our common response above; Appendix A.11). The new Recall@16 column compares the 16-candidate pools of all systems: FlashRAG-RR's pool holds the gold documents more often on KILT-HotpotQA and MuSiQue, and it keeps the higher judge score on KILT-HotpotQA, while MonoRoute-SVD leads in judge score on the other 5 benchmarks.

We hope that we have properly addressed the reviewer concerns, and will be glad to answer additional questions if needed.

Dear Reviewer, as the discussion period closes in 2 days, we wanted to check whether our SVD pivot, 16-candidate FlashRAG-RR budget comparison, and single-checkpoint framing address your concerns. Please let us know if any questions remain. The complete reproducible code is also now available in the supplementary material. Thank you.

Thank you for the substantial revisions and additional experiments. The SVD comparison and stronger RAG baselines address some of my concerns. While I appreciate the clearer acknowledgment of the single-checkpoint limitation, my concern about generalizability remains.

Thank you for the quick reply. We agree with your diagnosis: the layers agree because MSA trains every routing layer with the same contrastive loss on the same positive documents (MSA paper, Sec. 3.3.1). We only draw a different conclusion from it. MonoRoute is not proposed for transformers in general, but only to models trained with MSA's recipe, so a property that this recipe produces should be present wherever our method can be used as is. What may differ is which layer is the first sufficient one, and our released code measures that in a few GPU hours. We cannot show this on a second checkpoint, because none has been released, but our revised paper now states this scope in the abstract and explains the point in Section 4.3.

Since our SVD comparison and our stronger baselines addressed your other concerns, and the remaining one concerns the scope of our claims and not their correctness, we would be grateful if you would consider updating your assessment.

Reviewer_AwMGOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper aims to optimize efficiency of Memory Sparse Attention (MSA), an emerging technology that conducts attention-based memory retrieval inside the transformer model. The paper identifies the routing performance bottlenecks of MSA: (1) Router repeats corpus-wide similarity searches in multiple routing layers, which the paper proves to be redundant. (2) High-dimensional routing keys makes searching process more expensive (both memory and computation). The paper proposes Monoroute, which includes two training-free techniques: (1) Single-layer retrieval: only the shallowest routing layer conducts routing and later routing layers directly uses its results. (2) Projection symmetrization: the paper observes that the routing score is a bilinear form of the hidden states that can be well approximated by a low-rank symmetric matrix, significantly reducing routing key dimensionality. Evaluation results show that Monoroute outperforms the standard RAG method in both throughput and answer quality.

Critical strengths
  • It is a timely issue to accelerate documental retrieval. Attention-based retrieval technology is a promising and emerging direction that deserves more optimization efforts.
  • The The paper achieves significant computation and memory cost reduction compared with MSA.
Critical weaknesses
  • It is hard to judge the novelty of this paper. The paper does not clearly elaborate the limtations of existing methods (no realistic measurement study is provisioned). The paper also argues that a shared query/key space avoids the query-key distribution mismatch of ANN indexes, but no ANN index is used. Moreover, the paper does not explicitly summarize the key challenges and potential design choices. The proposed solutios look plain.

  • The design choices are not justified. In Sec 3.2.1, the paper decides rr and SrS_r based on an experiment with only 128 questions and the first 512 documents of TriviaQA-10M.

  • The baselines used in evaluation is weak (only one). The models are also small and of limited classes.

  • The presentation could be substantially improved. The fonts in the figures are too small. The paper structure can also be improved (few paper place background in methodology).

Questions for authors

See the questions above.

4 comments

We thank the reviewer for their comments and questions. Below, we address the presented weaknesses:

Measurement of the bottleneck; design space. The revision measures the bottleneck instead of asserting it: on MS MARCO 100M, MSA's routing keys alone occupy 41 GiB on each of 2 GPUs and MSA runs out of memory at 4 questions per GPU (Section 3); its routing pass takes 1.8-2.5 s at 100M tokens against 5-9 ms for MonoRoute (Tables 13-14). Section 4.1 now opens with the design space (number of searching layers, head grouping, rank, construction of the projection, score calibration, KV cap, approximate index) and says which choices we made and why; each choice is a row of the ablation with a paired confidence interval (items 1-3 and 6 in the common response above; Appendix A.9).

ANN mismatch argued, no ANN index used. The deployed system scans the rank-128 keys exactly, and Section 4.2 now says what we found with an index: an earlier version of the system routed through in-memory DiskANN graphs over the routing keys of every searching layer, one CPU graph walk per query token, and gained little beyond a batch of 8 questions, whereas a single searching layer at rank 128 turns the exact scan into one batched matrix product on the GPU whose throughput keeps growing with the batch. At 100M tokens that scan takes 5-9 ms per routing pass, a few percent of a question's time, so the index is kept as the option for far larger corpora (Section 5.5), where the orthogonal projection supplies the geometry such an index needs.

Design choices made on 128 questions and 512 documents. Those proxies motivated the construction; every choice is now validated end to end on 1,000-question benchmarks. Rank 128 is level with the full-width router on both ablation benchmarks when both are calibrated (judge within 0.02, citation recall within 0.003, every interval includes zero; Appendix A.7), ranks 256 and 512 change nothing, and the flattened weighting is compared with the weighted and signed alternatives (Appendix A.9, "Further projections"). Section 5.5 states that these choices were made on evaluation questions rather than held-out ones.

Baselines and models. Items 4 and 5 in the common response above: 2 stronger retrieval-augmented baselines on 6 benchmarks (Table 1). All systems answer with the same 4B reader, and HippoRAG-2's graph is built with a 70B model. MSA-4B is the only released MSA checkpoint; the method changes no weights, and Section 5.5 lists the single checkpoint as a limitation.

Presentation. The background on rectangular attention and MSA is now a section of its own (Section 3), before the Methodology, which opens with the design space. The derivation of the symmetrized projection and its fidelity tables moved to Appendix A.3, and the ablation figures were regenerated with larger fonts and taller panels (item 9 in the common response above).

We hope that this addresses the reviewer's concern, and we are ready to answer additional questions and discuss further if needed.

Dear Reviewer, as the discussion period closes in 2 days, we wanted to check whether our bottleneck measurements, stronger baselines (FlashRAG-RR and HippoRAG-2), and revised structure resolve your soundness concerns. Please let us know if any questions remain. We have also uploaded the complete reproducible code in the supplementary material. Thank you.

Thanks for the rebuttal. My concerns are addressed partially and I raise my score from 1 to 2. That said, it is suggested that the authors pay more efforts on presentation (try not placing a large chunk of text bodies in a solution section: use sub-titles, short cuts, etc.).

Thank you for raising your score and for this suggestion. We have revised the method section accordingly: the paragraphs of Section 4 now open with short bold titles (e.g. Design space, Routing score,...), so the main technical choices are easy to locate. If any part still reads as too dense, we would be glad to revise it further.

Reviewer_JskdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MonoRoute is an improvement over Memory Sparse Attention, which enables memory-augmented generation for large-scale question-answering tasks. To reduce the high compute and memory costs of Memory Sparse Attention, MonoRoute replaces layer-wise retrieval with retrieval performed only at the shallowest routing layer and compresses the retrieval queries and keys into a lower-dimensional space using a symmetric projection. Compared with Memory Sparse Attention, MonoRoute substantially reduces memory and compute costs and outperforms FlashRAG on QA datasets.

Critical strengths
  1. MonoRoute reduces both the number of retrieval operations and the dimensionality of the routing vectors through single-layer retrieval and low-rank projection, directly addressing the compute and memory bottlenecks caused by MSA's layer-wise full-corpus searches. In theory, these two changes reduce the leading-order similarity-scoring cost by 144x and routing-key storage by 8x. Experiments against the original MSA also show substantial reductions in routing latency and GPU memory usage.

  2. MonoRoute has a relatively simple design. It performs document selection only in the shallowest routing layer, and subsequent routing layers reuse that selection. Its symmetrized low-rank projection can be constructed directly from the existing routing projections without additional training, allowing the method to be applied to an existing MSA system with relatively localized changes.

Critical weaknesses
  1. Although the authors compare MonoRoute with FlashRAG in Section 4.2, the more direct baseline for MonoRoute is MSA. In Section 4.3, however, the authors compare them only on the MS MARCO and KILT benchmarks, and the results show that MonoRoute's performance drops significantly on multi-hop QA. If performance degradation is accepted, many approaches could improve MSA's efficiency. The authors do not adequately demonstrate that their method achieves a better trade-off.

  2. MonoRoute proposes projection symmetrization but does not provide sufficient evidence that it is necessary. Sections 4.2 and 4.3 show improved throughput and latency after applying the symmetrized projection, but Section 4.3 states, "The symmetrized implementation also reduces query heads from 32 to eight, shrinking scoring tensors fourfold." This trick is not unique to the symmetrized implementation and therefore does not demonstrate that projection symmetrization itself improves efficiency. The paper also states, "The symmetry was also targeted in order to resolve the query-key distribution mismatch," but does not provide an empirical comparison against an SVD-based method. Moreover, the symmetrized implementation leads to performance degradation on multi-hop QA. Thus, the effectiveness of projection symmetrization is not sufficiently demonstrated.

Questions for authors
  1. Could the authors provide a more comprehensive comparison between MonoRoute and MSA, similar to Table 4, and explain why the performance degradation introduced by MonoRoute represents a better trade-off than other methods?

  2. It would be helpful to compare the efficiency and effectiveness of the symmetrized projection and an SVD-based projection under the same head setting.

2 comments

We thank the reviewer for their comments and questions. Below, we address the 2 questions:

MSA as the direct baseline; the trade-off (Q1). We agree that MSA is the direct baseline. The revision compares with it on MS MARCO 100M and KILT-HotpotQA 100M, the single- and the multi-hop benchmark with the largest corpora, as paired comparisons on the same 1,000 questions: 13 variants in the ablation (Figures 2-3, Tables 13-16 in Appendix A.9) and the deployed configurations under the main experiment's retrieval loop (Appendix A.10). MSA was not run on the 4 smaller benchmarks. The multi-hop degradation the reviewer saw came from the symmetric projection, and the ablation now locates it there (Section 5.4; item 1 in the common response above). With the SVD router that we've introduced the trade-off has changed: under the deployed loop MonoRoute-SVD is above MSA by 0.20 judge points on KILT-HotpotQA and 0.18 on MS MARCO, at 8x MSA's throughput at batch 1 on KILT-HotpotQA (1.0 against 0.13 questions per second), with batches of 64 per GPU where MSA runs out of memory at 8 (4 on MS MARCO), and with 0.23 GiB of resident routing keys per GPU against MSA's 32 GiB.

Symmetrization versus SVD; head reduction (Q2). Agreed on both counts. The head reduction we've introduced in the revised version is an exact rewrite that MSA can use as well, so the speedups are now credited against the grouped baselines (item 6 in our common response). Hence the two projections are compared under the same head setting, rank, key memory (0.23 to 0.29 GiB per GPU) and routing time (5-9 ms per pass): rows LT18-sym-r128-cap29 and LT18-SVD128-cap29 of Figures 2-3 and Tables 13-14, which give routing time, routing keys, peak GPU memory, throughput at batch 1 and 16, citation recall and judge score. On KILT-HotpotQA the calibrated SVD router is 0.38 judge points above the symmetric one with 0.13 higher citation recall, and about 0.23 points ahead without the calibration; on MS MARCO the two are level in judge score and the symmetric router is 7-12 % faster because it cites fewer documents. This comparison is why the SVD is now the default (item 1 in our common response).

Dear Reviewer, with the discussion period closing in 2 days, we wanted to check whether our paired MSA-4B comparisons and matched-head SVD vs. symmetric ablations address your questions. Please let us know if anything further would be helpful. We have also updated the supplementary material with the full reproducible codebase. Thank you.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised paper - MonoRoute-SVD is fully tested, stronger baselines and more benchmarks.

We thank the reviewers for three careful reviews. They converge on one point, that the symmetrized projection was not shown to be necessary or better than a plain low-rank factorization, and we agree. In the revised paper we adjust the paper's main claim accordingly and add the experiments the reviews asked for. Please find the brief summary of our improvements w.r.t. the submitted manuscript:

  1. Projection: the truncated SVD is now the default (Section 4.2; ablations in Sections 5.3-5.4). A rank-128 truncated SVD of the routing form, with separate query and key projections, is compared with the symmetrized projection at the same rank, the same key memory and the same head setting, on MS MARCO and KILT-HotpotQA 100M. The SVD router keeps the quality of the full-width search and the symmetric router does not, so MonoRoute-SVD is the default configuration and the title no longer names symmetry. The symmetric projection stays as a variant: it cites fewer documents and is faster on single-passage questions. Its derivation and fidelity tables moved to Appendix A.3.
  2. Router score calibration (new; Section 4.4, Appendix A.7). For our SVD variant we found that each query token's scores should be divided by their spread over the corpus. This calibration is computed once from the stored keys and folded into the query vector, so the scan complexity is unchanged. This improvement made the SVD variant truly excellent in retrieval quality.
  3. KV cap at layer 29 and the three-stage backbone (Section 4.3, Appendix A.6). The submission studied withholding document KV states starting from layer 18 and until the end. Both deployed configurations (sym and SVD) now use the cap, a paired sweep on 1,000 KILT-HotpotQA questions confirms layer 29 as the smallest cap whose judge difference against no cap includes zero, and the paper describes the backbone as 18 abstracting, 12 retrieval and 6 generation layers, the last 6 never loading document KV states (a third of the store).
  4. Strong Baselines (Section 5.1, Appendix A.11). The submission's FlashRAG baseline (e5-base-v2, 3 documents, no reranker) is replaced by two retrieval-augmented pipelines on the same reader (Qwen3-4B-Instruct-2507): FlashRAG-RR, which retrieves 16 candidates with Qwen3-Embedding-0.6B, reranks them with Qwen3-Reranker-4B, reads 3 and runs IRCoT on multi-hop questions; and HippoRAG-2, the official code with its published settings. A Recall@16 column compares the 16-candidate pools of every system.
  5. Benchmarks and question sets (Section 5.1, Appendix A.8). 6 benchmarks instead of 5: NarrativeQA (the HippoRAG-2 subset) is added. MS MARCO is a new, clean set: the submission used the 128 questions of the MSA release, 104 of which are queries of MSA's continual pretraining, and the revision uses 1,000 MS MARCO test questions that are neither training queries nor linked to training passages. Every benchmark now has 1,000 questions (999 on MuSiQue) except TriviaQA (128) and NarrativeQA (293), the ablations run on 1,000 questions instead of 128, and every system answers the same questions and is scored by the same code.
  6. Grouped query heads (Appendix A.9, "Grouped query heads"). Summing the 4 query heads that share a key head before scoring is an exact rewrite of MSA's search. We added the grouping improvement as one of the baselines, and evaluated it in the ablations. It is now applied to MSA and to the single-layer baseline (MSA-GQ, LT18-GQ).
  7. Measurement (Section 5.1, Appendix A.9-A.10). MuSiQue under the common protocol and client-side clock; 33 repeated timing cells with a throughput spread of 1.0 % median and 3.4 % maximum; an MSA comparison under the deployed retrieval loop; and the b=1 versus b=16 quality drift (at most 0.04 judge points).
  8. Corpus sizes. Strengthening the baselines, as the reviews asked, changed one conclusion: on the 500M and 1B KILT-HotpotQA corpora the reranked pipeline now leads MonoRoute in judged answer quality, so those results no longer support the billion-token quality claim of the submission (although MonoRoute was still much faster). We have relaxed the 1B claim of our paper to 100M, hence the revision presents corpora of up to 100M tokens, and the title no longer names a billion tokens. We thank the reviewers for pressing for this comparison.
  9. Figures. The ablation figures (Figures 2-3, Appendix A.9) have larger fonts (as requested) and the architecture figure (Appendix A.2) now shows the generation stage above the retrieval layers.

We have substantially improved MonoRoute's quality and hope the reviewers find the revised manuscript satisfactory.

AuthorsOpenReview ↗

Rebuttal summary

Dear reviewers and area chair,

Before the discussion closes, we ask you to judge the revised PDF as the paper it now is. Following your reviews we reran every experiment, replaced our single baseline with 2 stronger ones, made the SVD router our default after it won the comparison you asked for, and tested every design choice against MSA on 1,000 questions. Our MonoRoute-SVD now scores above MSA itself on KILT-HotpotQA (Appendix A.10) at 8 times its throughput. Where our new experiments did not support a claim, we removed it. We would be grateful for a last look and an updated assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.