- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MonoRoute starts from the observation that Memory Sparse Attention (MSA) mechanism is inefficient especially in memory consumption and computation. The authors introduce two ways to mitigate it; diminishing size of routing key from 1024 to 128 without retraining, and reuse 18th layer’s search result in latter layers instead of re-compute them repeatedly. For dimension reduction, they use a symmetrized method instead of singular value decomposition (SVD), so that hidden state and routing key share projection to common space. To reuse the search result, they suggest this based on the observation that searched indexes from latter layer are similar to the first (18th layer) one.
Compare to the previous works like FlashRAG, MonoRoute outperforms both in quality and throughput (especially in batch mode). They compare MonoRoute with FlashRAG on five QA datasets, and ablate each change against the original MSA-4B.
Critical strengths
- The main strength of this work is clarity. The authors address the concrete problem of MSA and suggest practical solutions without training. They observe that routing dominates MSA’s cost (88% of generation time on multi-hop QA, with routing keys occupying 41 of 92 GiB of GPU memory), and that the top-ranked documents of later layers are mostly already selected by the first routing layer. Then they propose the straightforward solutions for each problem; dimension reduction for routing key by symmetric projection, and reuse the searched indexes rather than recompute for all layers. This is consequential because routing latency drops by 280x and 1B-token served on a 2 GPU node. It makes MSA more practical.
Critical weaknesses
-
The authors propose symmetrization over SVD, but they haven’t showed sound result that symmetrization outperforms SVD in term of search quality. They state plain SVD already preserves full accuracy, and both symmetrization and SVD have the same cost (both storage and compute). Symmetrization requires shared projection matrix, so it is compatible with ANN indices, but in the experiment, there is no result on it because all experiments use exact flat search.
-
All their observations and experiments only based on single checkpoint (MSA-4B). For instance, similarity of searched indexed among layers clearly happens in 4B model. However, MSA trains routing layer with the the same contrastive loss on the same dataset, so this agreement may be caused by MSA’s training rather than the general property. Besides, there is no evidence that it holds when model size, architecture, and training dataset vary.
Questions for authors
-
There is no comparison between the symmetrized projection and a rank-128 SVD factorization, although the authors address that the latter already preserves accuracy. Could you add an experiment to compare 128 rank SVD and the symmetrized projection both under the same flat search and ANN search? The latter one is because they claim the advantage of symmetrization is compatibility with graph-based search.
-
MonoRoute and FlashRAG have different number of documents. According to Sec. 4.1 and Table 10, FlashRAG approximately refers 10 documents (3 documents per round, and about 3 rounds) while MonoRoute refers about 16 per hop, about 24 ~ 32 candidates per question. Since MonoRoute selects documents from 16 candidates, it is still unclear the actual number it reads. Could the authors report the number of documents that MonoRoute refers and compare against a RAG baseline with similar document budgets such as FlashRAG with 16 retrieved candidates and rerank?
4 comments
Dear Reviewer, as the discussion period closes in 2 days, we wanted to check whether our SVD pivot, 16-candidate FlashRAG-RR budget comparison, and single-checkpoint framing address your concerns. Please let us know if any questions remain. The complete reproducible code is also now available in the supplementary material. Thank you.
Thank you for the substantial revisions and additional experiments. The SVD comparison and stronger RAG baselines address some of my concerns. While I appreciate the clearer acknowledgment of the single-checkpoint limitation, my concern about generalizability remains.
Thank you for the quick reply. We agree with your diagnosis: the layers agree because MSA trains every routing layer with the same contrastive loss on the same positive documents (MSA paper, Sec. 3.3.1). We only draw a different conclusion from it. MonoRoute is not proposed for transformers in general, but only to models trained with MSA's recipe, so a property that this recipe produces should be present wherever our method can be used as is. What may differ is which layer is the first sufficient one, and our released code measures that in a few GPU hours. We cannot show this on a second checkpoint, because none has been released, but our revised paper now states this scope in the abstract and explains the point in Section 4.3.
Since our SVD comparison and our stronger baselines addressed your other concerns, and the remaining one concerns the scope of our claims and not their correctness, we would be grateful if you would consider updating your assessment.
We thank the reviewer for their thorough review. Please find our response to the critical weaknesses and to your 2 questions (Q1, Q2) below:
Symmetrization versus SVD under flat search (Q1). The reviewer is right, and the revision reverses the emphasis (item 1 in our common response above). Under the same flat search, rank and key memory, the truncated SVD keeps the quality of the full-width router on both ablation benchmarks (both calibrated: judge within 0.02, citation recall within 0.003), while the symmetric projection loses 0.38 judge points to it on KILT-HotpotQA. MonoRoute-SVD is therefore the default, the title no longer names symmetry, and the sentence that SVD preserves accuracy is now backed by the ablation. The symmetric projection stays as a variant for single-passage questions, where it cites fewer documents and is faster.
Symmetrization versus SVD under ANN search (Q1). We did not add this comparison, and the revision says why (Section 4.2): an earlier version of the system routed through in-memory DiskANN graphs over the routing keys of every searching layer, one CPU graph walk per query token, and gained little beyond a batch of 8 questions, whereas the single-layer rank-128 scan is one batched matrix product on the GPU and keeps scaling with the batch. At 100M tokens that scan costs 5-9 ms per routing pass, a few percent of a question's time, so an index could not change the reported throughput measurably, and an approximate search cannot raise a router's quality above its exact result. The compatibility of the shared projection with graph or quantized indexes is stated as an expectation for corpora where the scan, linear in the corpus size, comes to dominate (Section 5.5), not as a result.
Single checkpoint; agreement across layers. We agree that the agreement of the routing layers may follow from MSA training every routing layer with the same objective on the same data rather than from a general property, and we cannot test other sizes or architectures: MSA-4B is the only released MSA checkpoint. The revision therefore presents the coverage table as motivation only (Appendix A.5) and rests the single-layer claim on the end-to-end ablation: on 1,000 KILT-HotpotQA questions the single-layer router matches MSA within the confidence interval, and on MS MARCO it loses 0.12 judge points (Figures 2-3). Section 5.5 lists the single checkpoint as a limitation.
Document budgets (Q2). MonoRoute attends to the pooled KV states of 16 candidates per retrieval round and injects the full text only of the documents it cites: 1.7 documents per question on MS MARCO, where it retrieves once, and 1.6 on KILT-HotpotQA over 1.6 rounds (Appendix A.9, "Per-question work", and Figures 7-8); it takes 2 rounds on 2WikiMultiHopQA and MuSiQue. FlashRAG-RR, the baseline the reviewer asked for, retrieves 16 candidates, reranks them and reads 3 per IRCoT round, over 2.6 to 3.0 rounds on the multi-hop benchmarks; HippoRAG-2 reads its 5 best passages (item 4 in our common response above; Appendix A.11). The new Recall@16 column compares the 16-candidate pools of all systems: FlashRAG-RR's pool holds the gold documents more often on KILT-HotpotQA and MuSiQue, and it keeps the higher judge score on KILT-HotpotQA, while MonoRoute-SVD leads in judge score on the other 5 benchmarks.
We hope that we have properly addressed the reviewer concerns, and will be glad to answer additional questions if needed.