acceptodds
Under review as a conference paper at ICLR 2027

Retrieval Layers: Fast End-to-End Retrieval with Sparse Attention

19 views

Abstract

Language model applications commonly require handling massive amounts of external information, for example querying internal documentation, reasoning over codebases, or user personalization based on past interaction history. Retrieval-augmented generation (RAG) is the dominant mechanism for grounding models in such external information. However, standard RAG frameworks are post-hoc scaffolds around the model, using a brittle lookup that cannot be trained end-to-end. Furthermore, RAG incurs significant computational overhead from prefilling large volumes of text, in turn slowing down subsequent generation by increasing the context length. To address these limitations, we introduce Retrieval Layers, end-to-end learnable sparse cross-attention layers that conduct fast lookup of external text in latent space, rather than prefilling retrieved text into the model’s context. Retrieval layers can be integrated into existing language models with minor post-training cost. We insert Retrieval Layers into an off-the-shelf Qwen3-4B and finetune it with custom SFT and RLVR datasets. On challenging multihop QA benchmarks, Retrieval Layers achieve Pareto-dominance on an accuracy versus end-to-end latency tradeoff in comparison to RAG baselines. We also show stronger performance than the Pareto frontier on long conversation reasoning and demonstrate that Retrieval Layers can extend the needle-in-a-haystack ability to 30M tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NzjVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Retrieval Layers, a sparse cross-attention mechanism that allows language models to access external information through a precomputed token-level KV bank instead of prefilling retrieved documents into the context. The authors integrate a Retrieval Layer into Qwen3-4B and propose a training procedure combining supervised fine-tuning, document-access loss, and reinforcement learning. Experiments on multi-hop QA and long-context benchmarks show improved accuracy and latency trade-offs compared with conventional RAG baselines. The method also demonstrates strong performance on long conversation reasoning and maintains high needle retrieval accuracy at context lengths of up to 30M tokens.

Critical strengths
  1. The paper addresses an important efficiency limitation of conventional RAG systems. By accessing precomputed document representations through sparse cross-attention, the proposed method avoids repeatedly prefilling retrieved documents into the LLM context. The architecture can also be integrated into existing pretrained models without requiring full pretraining.

  2. The experimental results are promising. ReLay-LM achieves competitive accuracy with substantially lower inference latency on several multi-hop QA benchmarks. The long-context results are also impressive, particularly the ability to maintain high needle retrieval accuracy at 30M tokens.

Critical weaknesses
  1. The architectural novelty is limited, and several closely related studies are missing from the discussion. Previous work such as LongMem [1], CEPE [2], and DecoupledRAG [3] has already explored accessing external document representations through attention without directly incorporating retrieved text into the LLM context. In particular, CEPE and DecoupledRAG have demonstrated similar architectures that can be integrated into existing pretrained LLMs. The main differences appear to be sparse top-k attention, a single retrieval layer, and the training procedure, but the paper does not sufficiently discuss or compare against these studies.

  2. The baseline comparisons are unfair due to substantial differences in task-specific training. ReLay-LM jointly trains the 4B generator, 0.6B encoder, and additional retrieval parameters using 100K SFT steps on 1.6M QA examples, followed by RL on the training splits of HotpotQA, MuSiQue, and 2WikiMultihopQA, which are also used for evaluation. In contrast, the RAG baselines use the original Qwen3-4B without comparable task-specific fine-tuning or RL. Table 3 shows that ReLay-LM before RL underperforms RAG@10 on all three benchmarks, making it difficult to attribute the final improvements to the proposed architecture rather than the additional training. ReLay-LM also uses 50 retrieved documents compared with at most 10 for RAG. The lack of key ablations on document-access loss and joint encoder training, together with missing latency and memory measurements at 30M tokens, further limits the experimental evidence.

[1] Augmenting Language Models with Long-Term Memory. NeurIPS 2023.

[2] Long-Context Language Modeling with Parallel Context Encoding. ACL 2024.

[3] Decoupling Knowledge and Context: An Efficient and Effective Retrieval Augmented Generation Framework via Cross Attention. WWW 2025.

Questions for authors
  1. How does the proposed method differ from closely related architectures such as CEPE and DecoupledRAG, and what are its specific contributions beyond sparse top-k attention?

  2. Could the authors compare ReLay-LM against RAG baselines trained with the same task-specific data and comparable SFT and RL budgets, as well as under matched retrieval budgets?

  3. How much do document-access loss and joint encoder training contribute to the final performance, and could the authors provide corresponding ablations?

  4. What are the encoding overhead, retrieval latency, and peak memory consumption at 1M, 10M, and 30M tokens, and how is the token-level KV bank stored at these scales?

  5. Since the multi-hop QA experiments still rely on an external retriever to select the initial 50 documents, could the authors clarify the scope of the claimed end-to-end retrieval optimization?

5 comments

We thank the reviewer for recognizing that our method addresses an important efficiency limitation of RAG and efficiently integrates into existing pretrained models, and for finding our long-context results impressive. Below we address the remaining concerns.

"...Could the authors compare ReLay-LM against RAG baselines trained with the same task-specific data and comparable SFT and RL budgets..." We agree this comparison is important, and we have added exactly this baseline (see Figure 1). Qwen3-4B RAG is now trained on the same 1.6M SFT samples for the same 100K steps, then with the same DAPO [4] RL recipe and hyperparameters as ReLay-LM, retrieving the top 20 chunks per rollout (see Appendix B.1 for setup). Averaged over HotpotQA, MuSiQue, and 2WikiMultihopQA (Table 4):

MethodAccuracy (%)TTFT (ms)
RAG@2057.55 ± 0.9157.6
RAG@20 (SFT)54.04 ± 0.9057.6
RAG@20 (SFT + RL)61.46 ± 0.6057.6
ReLay-LM (SFT)44.70 ± 1.5417.2
ReLay-LM (SFT + RL)62.02 ± 0.2717.5

Under the same training budget, ReLay-LM achieves higher average Multi-hop QA accuracy than the best RAG configuration with 3.3x lower TTFT. Furthermore, the improvement from the SFT checkpoint to the RL checkpoint is significantly larger for ReLay-LM (39% relative improvement) than for RAG (14% relative improvement). This suggests that the end-to-end learnability of ReLay-LM allows it to benefit more from post-training.

"...still rely on an external retriever to select the initial 50 documents..." / "...compared with at most 10 for RAG..." To strengthen our end-to-end optimization, we now perform RL with lookups over the full benchmark corpus, rather than using an intermediate external retriever. We find RL still yields comparable improvements in multi-hop QA accuracy. In the revised submission, at inference, ReLay-LM conducts sparse lookup across the entire corpus at each decoding step, and does not use an external retriever at all. See the table above for how ReLay-LM compares to RAG@20.

"...several closely related studies are missing from the discussion. Previous work such as LongMem [1], CEPE [2], and DecoupledRAG [3]..." We thank the reviewer for these references, and our revised submission now includes discussion of all 3 in Section 2.2 and Table 1, as well as ablations that test their main design choices inside ReLay-LM (Appendix G). We comment on the suggested references below:

  • LongMem [1] keeps its LLM frozen and retrieves chunks rather than tokens: It caches the keys and values computed by a frozen LLM, and trains a side network that retrieves chunks of this cache using mean-pooled chunk keys, then attends to every token of the retrieved chunks. ReLay-LM instead trains the encoder, the Retrieval Layer, and the LLM jointly. We ablate the importance of this joint training by freezing each backbone (Section 4.6, Figure 5, Appendix G.3). We find that freezing the LLM reduces multi-hop accuracy relatively by 19%, freezing the encoder reduces it by 33%, and freezing either drops retrieval performance (measured by average gold document attention mass) by at least 4x. We also test LongMem's chunk-level lookup inside ReLay-LM's setup (Appendix G.10). We search over chunks using a single pooled key per chunk and then perform token-level dense attention on the selected chunks. Selecting just one chunk retrieves 4 times as many values as ReLay-LM and still reduces accuracy by 33.6% relative to ReLay-LM. Retrieving more chunks (4-5) does not outperform ReLay-LM.

  • CEPE [2] does not train retrieval end-to-end and uses dense attention: CEPE only finetunes cross-attention layers (rest of the LLM is frozen) to attend to a KV bank produced by a lightweight trained encoder. Although ReLay-LM shares some architectural similarities, the lack of fully end-to-end training, sparse attention, and a document-access loss limits its suitability for retrieval over large corpora. We find that if we use dense attention instead of sparse attention, accuracy is reduced by 16% relatively and retrieval performance drops by 33% (Appendix G.6). Furthermore, we find that the document-access loss is crucial for ReLay-LM's ability to extract relevant information from a large KV bank. Removing it reduces accuracy by 36.2% relatively and collapses retrieval performance, with gold attention mass falling from 0.249 to below 0.001 (Appendix G.4).

AuthorsOpenReview ↗

Rebuttal Continued

  • DecoupledRAG [3] relies on a fixed external retriever: An external retriever selects K documents per query, whose keys and values are precomputed by the frozen LLM. The LLM learns to densely cross-attend to the KV cache of retrieved documents using only a low-rank gate. Once again, not training end-to-end sacrifices accuracy and retrieval ability. Furthermore, DecoupledRAG's experiments only deal with non-reasoning models. Due to a lack of readily available SFT data for corpus-retrieval QA, it is not obvious how to adapt an existing reasoning model to cross-attend to documents. We show that our SFT data generation scheme (Section 3.2) is able to close this gap and that our reasoning models can further benefit from RL.

"...what are its specific contributions beyond sparse top-k attention?..." Beyond the usage of sparse top-kk attention, ReLay-LM contributes end-to-end training of retrieval with the LLM, a recipe for adapting existing reasoning models to retrieval, and a design that keeps token-level lookup fast and memory-efficient at tens of millions of tokens. The revision adds results supporting each:

  • End-to-end training of retrieval and generation. The encoder, the Retrieval Layer, and the LLM are trained jointly. We ablate the importance of this joint training by freezing each backbone (Section 4.6, Figure 5, Appendix G.3). We find that freezing the LLM reduces multi-hop accuracy relatively by 19%, freezing the encoder reduces it by 33%, and freezing either drops retrieval performance (measured by average gold document attention mass) by at least 4x. Furthermore, we find that the document-access loss is crucial for ReLay-LM's ability to extract relevant information from a large KV bank. Removing it reduces accuracy by 36.2% relatively and collapses retrieval performance, with gold attention mass falling from 0.249 to below 0.001 (Appendix G.4).

  • End-to-end adaptation of reasoning models. Prior methods either require pretraining (REALM [5], Atlas [6], RETRO [7], MSA [8]) or fine-tune non-reasoning LLMs (RA-DIT [9], ImpRAG [10], DecoupledRAG [3]). There is no readily available SFT data for reasoning over a retrieved corpus, so we generate it from Qwen3-4B's own reasoning traces (Section 3.2), which preserves the model's reasoning style. We then apply a modern RL stack using DAPO with verifiable rewards. RL improves ReLay-LM's accuracy by 39% relatively compared with 14% for the training-matched RAG@20 baseline (Table 4).

  • Lookup speed and memory efficiency. Token-level retrieval over corpora of millions of tokens is only practical if KV caches are small and decode throughput remains fast. Our ablation shows that a single Retrieval Layer is sufficient, allowing us to substantially reduce GPU memory (Appendix G.9). Furthermore, our inference cost scaling benchmarking (see below) shows that by training with sparse attention, we can utilize approximate top-kk algorithms to achieve high throughput even at 32M token banks (Figure 4).

"...How much do document-access loss and joint encoder training contribute to the final performance, and could the authors provide corresponding ablations?..." We agree, and the revised submission adds both ablations. They are summarized in Section 4.6 (Figure 5), with full results in Appendix G.3 and G.4. To make multi-seed ablations computationally feasible, our ablations shorten SFT from 100K to 10K steps and skip RL. Accuracy is averaged over HotpotQA, 2WikiMultihopQA, and MuSiQue and gold mass is the fraction of retrieval attention assigned to gold documents.

Joint encoder training. We freeze the document encoder, the LLM, or both, while the Retrieval Layer remains trainable (Table 9):

VariantAccuracy (%)Gold mass
Default21.1 ± 1.60.248 ± 0.007
Frozen LLM17.0 ± 0.80.062 ± 0.001
Frozen encoder14.1 ± 0.70.041 ± 0.001
Frozen encoder + LLM14.9 ± 0.80.019 ± 0.001

Freezing the LLM, the encoder, or both reduces accuracy by 19%, 33%, and 29% relatively, and reduces gold mass by 4x, 6x, and 13x.

Document-access loss. Training stages 1–2 use the document-access loss alone with weight 0.1. We vary its weight in stages 3–4, where it is added to the language-modeling loss, and separately remove it from every stage ("Off") (Table 10):

λacc\lambda_{acc}Accuracy (%)Gold mass
Off13.9 ± 0.70.000 ± 0.000
016.3 ± 1.90.033 ± 0.002
0.0116.9 ± 1.00.075 ± 0.004
0.1 (default)21.8 ± 1.40.249 ± 0.010
1.016.9 ± 1.80.315 ± 0.001

The document-access loss is necessary for sparse retrieval. However, placing too much weight on it harms overall QA accuracy. The default weight of 0.1 gives the highest accuracy.

AuthorsOpenReview ↗

Rebuttal Continued

"...What are the encoding overhead, retrieval latency, and peak memory consumption at 1M, 10M, and 30M tokens, and how is the token-level KV bank stored at these scales?..." We agree that encoding overhead, retrieval latency, and peak GPU memory are important for determining practical usage. The revised submission adds an inference-cost scaling study up to 32M tokens on two H100 GPUs (Section 4.5, Figure 4, Appendix F). It compares ReLay-LM with exact top-k search, ReLay-LM with approximate top-k search (CAGRA), and Qwen3-4B/Qwen3.5-4B with the same evidence in their context. Below we highlight results for 1M, 8M, and 32M tokens.

Encoding Overhead (s)

Method1M8M32M
ReLay-LM: bank encoding4.637.1150.2
ReLay-LM: CAGRA index construction (approx. top-k only)11.7122.8551.3
Qwen3.5-4B: prefill64.1OOMOOM

Time per decoded token, including retrieval (ms)

Method1M8M32M
ReLay-LM (exact top-k)3.386.0914.97
ReLay-LM (approx. top-k)3.383.533.54
Qwen3.5-4B8.91OOMOOM

Peak GPU memory, summed over both GPUs (GiB)

Method1M8M32M
ReLay-LM (exact top-k)21.150.3146.3
ReLay-LM (approx. top-k)23.160.5159.3
Qwen3.5-4B52.2OOMOOM

The KV bank is encoded once per corpus, so this overhead appears offline and is negligible for online queries. For comparison, we show Qwen3.5-4B's prefill time, which is required for every query.

At 1M tokens, ReLay-LM with exact top-k has 1,300x lower TTFT than Qwen3.5-4B (14x including bank encoding), decodes 2.6x faster, and uses 2.5x less GPU memory. At 32M tokens, approximate top-k maintains per-token latency at 3.5 ms, vs. 15.0 ms with exact top-k. This highlights that approximate search with sparse attention enables efficient token-level lookup at massive contexts. See Section 4.5 for more details.

In a Retrieval Layer, each corpus token is encoded once and stored as a single key and value. The bank is kept in GPU memory and sharded by rows across the two GPUs. Keys and the search data are stored in INT8 with FP32 per-row key scales and values are stored in FP16 (Appendix A.3).

Additional New Results.

  • Attention concentration analysis: we analyze how the fraction of attention mass on the gold documents of multi-hop questions changes as we add more distractor documents to the corpus, up to about 1M tokens (Section 4.4, Figure 3, Table 7). We find that full-context models (Qwen3-4B, Qwen3.5-4B) degrade quickly while ReLay-LM's gold attention mass tapers off slowly. At about 1M tokens, ReLay-LM keeps on average 59% of its attention on the gold documents, 36 times more than full-context Qwen3.5-4B.
  • Ablations: we ablate retrieval layer position, dense vs. sparse attention, multiple Retrieval Layers, document-level retrieval, doc access loss, frozen encoder/LLM, the training schedule, retrieval layer temperature, key quantization, and approximate top-k (Section 4.6, Appendix G). These ablations support the individual design choices in ReLay-LM.
  • Expanded related work: we discuss and compare RA-DIT [9], ImpRAG [10], memory layers [11, 12], LongMem [1], CEPE [2], and DecoupledRAG [3] (Section 2, Table 1, Appendix G).
Reviewer_FK91OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper seeks to build efficient RAG methods for LLMs, by building an end-to-end approach. They propose retrieval layers and claim that this approach achieves better efficiency without comparing to many existing closely related work. In experiments, they only evaluate on 2 long context benchmarks, and showing improvements solely based on the trends of the line plot.

Critical strengths

None

Critical weaknesses

This work omits A LOT OF existing related work on jointly training retrievers and LLMs, e.g., memory layers, RADIT, and ImpRAG

The motivation points of this work are not new at all. Prior work already discussed and addressed them to some extent, yet none of them are discussed or compared to in the paper. This is where this work needs a complete rework.

The experiment results are very limited. There is only ONE plot serving as the main result. NO numerical results are reported in the paper, and no analysis results demonstrating insights of the method. This is clearly not ready for publication at a top conference like ICLR.

Questions for authors

none

4 comments

We thank the reviewer for their feedback. Before addressing the remaining concerns, we note that several statements in the review do not match our initial submission.

"...There is only ONE plot serving as the main result..." Our initial submission had 5 plots: Figure 1 had showed accuracy against latency on each of the three multi-hop benchmarks, and Figure 3 had showed accuracy against latency on BEAM and accuracy against context length on RULER. The revised submission adds 3 more figures: attention concentration (now Figure 3), inference cost scaling (Figure 4), and ablations (Figure 5).

"...NO numerical results are reported in the paper..." Appendix A.2 of our initial submission reported accuracy and latency for every method on every benchmark. The main text also stated the key numerical results. The other two reviewers quote these numbers in their reviews. The revised submission expands these tables with seed standard deviations, TTFT, decode time, throughput, and generated tokens (Tables 4–6).

"...yet none of them are discussed or compared to in the paper..." Our initial submission already had a dedicated related work subsection on end-to-end learned retrieval (Section 2.2), which discussed REALM, Atlas, RETRO, and Memory Sparse Attention (MSA). We also evaluated MSA-4B, an end-to-end trained sparse-attention memory model, as a baseline on BEAM. The revised submission adds discussion of RA-DIT [3], ImpRAG [4], Memory Layers [1, 2], LongMem [5], CEPE [6], and DecoupledRAG [7] and comparison with their key attributes (shown below).

"...they only evaluate on 2 long context benchmarks..." Our submission evaluated 5 benchmarks: three multi-hop QA benchmarks (HotpotQA, MuSiQue, and 2WikiMultihopQA) and two long-context benchmarks (BEAM and RULER). The revised submission also evaluates BEAM at 256K, 512K, and 1M tokens.

--

We now address the remaining concerns.

"...no analysis results demonstrating insights of the method..." We agree that additional analysis of ReLay-LM's behaviour, components, and scaling would complement accuracy and efficiency results. The revised submission adds:

  • Attention concentration analysis: We analyze how the fraction of attention mass on the gold documents of multi-hop questions changes as we add more distractor documents to the corpus, up to about 1M tokens (Section 4.4, Figure 3, Table 7). We find that full-context models (Qwen3-4B, Qwen3.5-4B) degrade quickly while ReLay-LM's gold attention mass tapers off slowly. At about 1M tokens, ReLay-LM keeps on average 59% of its attention on the gold documents, 36 times more than full-context Qwen3.5-4B.
  • Ablations: We ablate retrieval layer position, dense vs. sparse attention, multiple Retrieval Layers, document-level retrieval, doc access loss, frozen encoder/LLM, the training schedule, retrieval layer temperature, key quantization, and approximate top-k (Section 4.6, Appendix G). These ablations support the individual design choices in ReLay-LM.
  • Inference cost scaling: We measure how TTFT, decode throughput, and peak GPU memory scale as we add more tokens to the Retrieval Layer's bank versus to an LLM's context, up to 32M tokens (Figure 4, Section 4.5). Full-context costs grow quickly, while ReLay-LM's grow far more slowly. At 2M tokens, ReLay-LM with exact top-k has 26x lower TTFT than Qwen3.5-4B, decodes 3.9x faster, and uses 3.3x less GPU memory. Full-context Qwen3-4B and Qwen3.5-4B run out of memory beyond 1M and 4M tokens, respectively, while ReLay-LM handles up to 32M tokens. At 32M tokens, approximate top-k maintains decode throughput at 282 tokens/s, vs. 67 tokens/s with exact top-k. This highlights that approximate search with sparse attention enables efficient token-level lookup at massive contexts.

"...omits A LOT OF existing related work on jointly training retrievers and LLMs, e.g., memory layers, RADIT, and ImpRAG..." We thank the reviewer for these references, and our revised submission now includes discussion of all 3 in Section 2.2 and Table 1, as well as ablations that test their main design choices (Appendix G). We comment on the suggested references below:

  • Memory layers [1, 2] are not a retrieval method: They are a pretraining architecture that replaces feed-forward layers with a sparse top-k lookup into trainable key-value parameters. These parameters are learned during pretraining and are not tied to any external corpus. Memory layers therefore cannot be compared with ReLay-LM on retrieval tasks.
AuthorsOpenReview ↗

Rebuttal Continued

  • RA-DIT [3] trains the retriever and the LLM separately: It first finetunes the LLM to use retrieved documents placed in its context. It then finetunes only the retriever's query encoder toward the documents the LLM prefers, with the LLM and the document encoder frozen. The retriever is never trained by the LLM's generation loss, so RA-DIT is not an end-to-end method, and its retrieved documents are still prefilled as text. ReLay-LM instead trains the encoder, the Retrieval Layer, and the LLM jointly. We evaluate the importance of this joint training by freezing each backbone (Section 4.6, Figure 5, Appendix G.3). Freezing the LLM reduces multi-hop accuracy relatively by 19%, and freezing the encoder reduces it by 33%. Freezing either backbone also collapses retrieval.

  • ImpRAG [4] is end to end, but uses only one round of retrieval and a single embedding per passage: Its lower layers embed the query and each passage, its top-k search selects whole documents once per prompt, and its middle layers then attend densely to every token of those documents. ReLay-LM instead searches token-level keys and selects individual tokens at every decoding step, enabling it to do multi-hop retrieval during its Chain-of-Thought. We evaluate the importance of this token-level lookup by applying ImpRAG's retrieve-then-read procedure inside ReLay-LM's training setup (Appendix G.10). We search over chunks using a single pooled key per chunk and then perform token-level dense attention on the selected chunks. Selecting just one chunk retrieves 4 times as many values as ReLay-LM and still reduces accuracy by 33.6% relative to ReLay-LM. Retrieving more chunks (4-5) does not outperform ReLay-LM.

"...The motivation points of this work are not new at all..." We believe many of our motivations are novel, even though prior work has also pursued end-to-end training of retrieval with the LLM. Below we outline motivations that, to the best of our knowledge, have not been jointly pursued before. We also present evidence of how ReLay-LM justifies these motivations.

  • Token-level sparse retrieval. Prior end-to-end methods retrieve whole documents, use one key embedding per document, and attend densely to every encoded token. Retrieval Layers instead select the top-64 individual tokens from the entire corpus, at every decoding step. We find that sparse retrieval is particularly beneficial. A ReLay-LM that uses dense attention over the full bank instead, reduces multi-hop accuracy by 16% relatively and has a 33% lower gold attention mass (Appendix G.6). We have also shown that densely retrieving full chunks reduces accuracy (Appendix G.10).
  • Lookup speed and memory efficiency. Token-level retrieval over corpora of millions of tokens is only practical if KV caches are small and decode throughput remains fast. Our ablation shows that a single Retrieval Layer is sufficient, allowing us to substantially reduce GPU memory (Appendix G.9). Furthermore, our inference cost scaling benchmarking shows that by training with sparse attention, we can utilize approximate top-kk algorithms to achieve high throughput even at 32M token banks (Figure 4).
  • End-to-end adaptation of reasoning models. Prior retrieval-based methods either require pretraining or use training procedures that do not account for reasoning models. REALM, Atlas, and RETRO pretrain the LLM together with retrieval, and MSA requires continued pretraining before SFT. RA-DIT and ImpRAG fine-tune non-reasoning LLMs with supervised objectives. We instead take an existing reasoning model, Qwen3-4B, and teach it to use a Retrieval Layer with SFT on its own generations (Section 3.2). This preserves the model's own reasoning style. We then apply a modern RL stack, DAPO with verifiable rewards, to improve its reasoning further. The end-to-end signal is essential because freezing either backbone reduces accuracy by 19–33% relatively (Appendix G.3).
AuthorsOpenReview ↗

Rebuttal Continued

Thank you again for your valuable feedback. Prompted by your points, we have added extensive analysis and paper edits, which we think have improved the quality of our work. We would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other feedback we can address?

[1] Lample, Guillaume, et al. "Large Memory Layers with Product Keys." Advances in Neural Information Processing Systems 32 (2019).

[2] Berges, Vincent-Pierre, et al. "Memory Layers at Scale." arXiv preprint arXiv:2412.09764 (2024).

[3] Lin, Xi Victoria, et al. "RA-DIT: Retrieval-Augmented Dual Instruction Tuning." ICLR 2024.

[4] Zhang, Wenzheng, et al. "ImpRAG: Retrieval-Augmented Generation with Implicit Queries." Findings of EMNLP 2025.

[5] Wang, Weizhi, et al. "Augmenting Language Models with Long-Term Memory." NeurIPS 2023.

[6] Yen, Howard, Tianyu Gao, and Danqi Chen. "Long-Context Language Modeling with Parallel Context Encoding." ACL 2024.

[7] Dong, Qian, et al. "Decoupling Knowledge and Context: An Efficient and Effective Retrieval Augmented Generation Framework via Cross Attention." The Web Conference 2025.

Thank you again for your feedback. Before the author-reviewer discussion period closes, are there are any other questions we can answer? We made a strong effort to address all your points, including new experiments and paper edits, and we would greatly appreciate it if you would consider increasing your score accordingly.

Reviewer_7wRSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Retrieval Layers, which use sparse cross-attention to read token-level document key-value representations produced by an independent encoder, reducing the overhead of prefilling retrieved text into the language model’s context. Its main contributions are this external-memory architecture and a joint training recipe combining a document-access loss, supervised fine-tuning, and reinforcement learning with verifiable rewards. The authors implement ReLay-LM on Qwen3-4B and evaluate it on three multi-hop QA benchmarks, BEAM conversation information extraction, and RULER single-needle retrieval. The reported results show approximately 1.89× average online query speedup over the evaluated RAG@10 baseline with comparable or higher accuracy, an improved accuracy–latency trade-off on BEAM, and approximately 88.5% average single-needle retrieval accuracy at 30 million tokens.

Critical strengths

1、A practically useful architecture for reducing RAG prefilling overhead. Retrieval Layers use a single sparse cross-attention layer to access independently encoded, token-level document KV representations, avoiding the need to insert retrieved text into the generator’s context while supporting joint training of the document encoder and language model. This provides a concrete approach to combining efficient external-information access with task-specific adaptation, constituting the paper’s main technical strength. 2、Consistent accuracy–latency improvements across multiple QA benchmarks. On three multi-hop QA benchmarks, the final model achieves comparable or higher accuracy than the evaluated RAG@10 baseline with approximately 1.89× average online query speedup. On BEAM, it improves accuracy from 46.2% to 54.7% while reducing latency from 110.6 to 29.1 seconds relative to Qwen3-4B. These results provide empirical support for the practical efficiency benefits of the design and weigh positively in my assessment.

Critical weaknesses

1、Unmatched training budgets obscure the source of the gains. ReLay-LM receives additional SFT and RL that the RAG baseline does not.Table 3 shows that RL contributes substantially to the final accuracy.A training-matched baseline is needed to isolate the architectural contribution.

2、The comparison with MSA is limited to a restricted configuration. The authors disable Memory Interleave to isolate latent-space retrieval (§4.1). While this comparison is informative, it does not establish the accuracy–latency trade-off against the full MSA system. Reporting results with Memory Interleave enabled would strengthen the system-level comparison.

Questions for authors

1、How would the RAG baseline perform with the same SFT data and a similar RL training budget? Would ReLay-LM still be faster while maintaining comparable or better accuracy?

2、How would MSA perform with its original MemoryInterleave enabled? Could you include its accuracy and latency so we can better understand the difference between the two complete systems?

3、Does RL reduce the total number of generated tokens, including reasoning traces and final answers? Please report average generation lengths before and after RL and for the RAG baseline, together with a latency breakdown, to clarify how much of the observed speedup is associated with shorter generations versus reduced document-prefilling overhead.

3 comments

We thank the reviewer for finding Retrieval Layers a practically useful architecture for reducing RAG prefilling overhead, and for recognizing its accuracy–latency improvements across benchmarks. Below we address the remaining concerns.

"...A training-matched baseline is needed to isolate the architectural contribution..." We agree, and we have added exactly this baseline (see Figure 1). Qwen3-4B RAG is now trained on the same 1.6M SFT samples for the same 100K steps, then with the same DAPO RL recipe and hyperparameters as ReLay-LM, retrieving the top 20 chunks per rollout (see Appendix B.1 for setup). Averaged over HotpotQA, MuSiQue, and 2WikiMultihopQA (Table 4):

MethodAccuracy (%)TTFT (ms)
RAG@2057.55 ± 0.9157.6
RAG@20 (SFT)54.04 ± 0.9057.6
RAG@20 (SFT + RL)61.46 ± 0.6057.6
ReLay-LM (SFT)44.70 ± 1.5417.2
ReLay-LM (SFT + RL)62.02 ± 0.2717.5

Under the same training budget, ReLay-LM achieves higher average Multi-hop QA accuracy than the best RAG configuration with 3.3x lower TTFT. Furthermore, the improvement from the SFT checkpoint to the RL checkpoint is significantly larger for ReLay-LM (39% relative improvement) than for RAG (14% relative improvement). This suggests that the end-to-end learnability of ReLay-LM allows it to benefit more from post-training.

"...How would MSA perform with its original MemoryInterleave enabled?..." We now run MSA-4B [1] with Memory Interleave, which retrieves document chunks into the context, and without, which directly answers using only its sparse attention layers. We show accuracy and latency up to 1M tokens on BEAM (see Figure 1, Table 5) below:

BEAM lengthReLay-LMMSA with Memory InterleaveMSA w/o interleaveQwen3-4BQwen3.5-4B
128K40.2% / 1.78 s41.0% / 4.82 s7.7% / 3.54 s35.9% / 19.1 s62.2% / 20.7 s
256K35.2% / 1.45 s30.0% / 4.27 s1.0% / 3.90 s17.1% / 65.5 s52.5% / 34.4 s
512K38.6% / 1.38 s28.6% / 4.56 s1.4% / 6.85 s0.0% / 94.1 s40.1% / 67.9 s
1M33.8% / 1.43 s31.4% / 4.54 s4.8% / 5.79 s0.0% / 184.8 s26.0% / 185.0 s

We see that ReLay-LM outperforms MSA with interleave on BEAM lengths greater than 128K while having significantly lower latency at all lengths. ReLay-LM also outperforms Qwen3.5-4B on BEAM 1M, demonstrating superior scaling with conversation length.

"...how much of the observed speedup is associated with shorter generations versus reduced document-prefilling overhead..." We agree that a breakdown of generation length, decode time, and time-to-first-token (TTFT) is important for evaluating latency. For multi-hop QA, Table 4 now reports generated tokens, TTFT, decode time, end-to-end latency, and decode throughput for every configuration. To summarize, we show averages across benchmarks below:

MethodGenerated tokensTTFTDecodeLatency
RAG@587233 ms4.35 s4.39 s
RAG@1095639 ms4.81 s4.85 s
RAG@2097758 ms4.99 s5.05 s
RAG@20 (SFT)43458 ms2.22 s2.27 s
RAG@20 (SFT + RL)25958 ms1.32 s1.38 s
ReLay-LM (SFT)43717 ms2.42 s2.44 s
ReLay-LM (SFT + RL)32018 ms1.79 s1.81 s

This breakdown confirms that both our SFT and RL procedures reduce the average generation length, which largely contributes to end-to-end latency. However, it is important to note that our initial motivation for sparse latent retrieval was to avoid prefill cost - a metric that cannot be improved with post-training. Indeed, we see that the TTFT gap between ReLay-LM and RAG consistently grows as RAG retrieves more documents, regardless of training budget.

To demonstrate this further, in Figure 4, we measure inference costs when scaling up to 32M tokens on two H100 GPUs. We show how TTFT (including bank construction), decode throughput, and peak GPU memory scale as we add more tokens in the Retrieval Layer versus in the context. We show some of these measurements below:

TTFT (s)

Method128K2M32M
ReLay-LM (exact top-k)0.629.20150.6
ReLay-LM (approx. top-k)4.4730.8701.6
Qwen3.5-4B2.23239.2OOM

Decode throughput (tokens/s)

Method128K2M32M
ReLay-LM (exact top-k)31125767
ReLay-LM (approx. top-k)281284282
Qwen3.5-4B26666OOM

Peak GPU memory, summed over both GPUs (GiB)

Method128K2M32M
ReLay-LM (exact top-k)16.426.3146.3
ReLay-LM (approx. top-k)20.126.5159.3
Qwen3.5-4B23.385.4OOM
AuthorsOpenReview ↗

Rebuttal Continued

At 2M tokens, ReLay-LM with exact top-k has 26x lower TTFT than Qwen3.5-4B, decodes 3.9x faster, and uses 3.3x less GPU memory. At 32M tokens, approximate top-k maintains throughput at 282 tokens/s, vs. 67 tokens/s with exact top-k. This highlights that approximate search with sparse attention enables efficient token-level lookup at massive contexts. See Section 4.5 for more details.

Additional New Results.

  • Attention concentration analysis: we analyze how the fraction of attention mass on the gold documents of multi-hop questions changes as we add more distractor documents to the corpus, up to about 1M tokens (Section 4.4, Figure 3, Table 7). We find that full-context models (Qwen3-4B, Qwen3.5-4B) degrade quickly while ReLay-LM's gold attention mass tapers off slowly. At about 1M tokens, ReLay-LM keeps on average 59% of its attention on the gold documents, 36 times more than full-context Qwen3.5-4B.
  • Ablations: we ablate retrieval layer position, dense vs. sparse attention, multiple Retrieval Layers, document-level retrieval, doc access loss, frozen encoder/LLM, the training schedule, retrieval layer temperature, key quantization, and approximate top-k (Section 4.6, Appendix G). These ablations support the individual design choices in ReLay-LM.
  • Expanded related work: we discuss and compare RA-DIT [2], ImpRAG [3], memory layers [4, 5], LongMem [6], CEPE [7], and DecoupledRAG [8] (Section 2, Table 1, Appendix G).

--

Thank you again for your valuable feedback. Prompted by your points, we have added extensive analysis and paper edits, which we think have improved the quality of our work. We would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other feedback we can address?

[1] Chen, Yu, et al. "MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens." arXiv preprint arXiv:2603.23516 (2026).

[2] Lin, Xi Victoria, et al. "RA-DIT: Retrieval-Augmented Dual Instruction Tuning." ICLR 2024.

[3] Zhang, Wenzheng, et al. "ImpRAG: Retrieval-Augmented Generation with Implicit Queries." Findings of EMNLP 2025.

[4] Lample, Guillaume, et al. "Large Memory Layers with Product Keys." Advances in Neural Information Processing Systems 32 (2019).

[5] Berges, Vincent-Pierre, et al. "Memory Layers at Scale." arXiv preprint arXiv:2412.09764 (2024).

[6] Wang, Weizhi, et al. "Augmenting Language Models with Long-Term Memory." NeurIPS 2023.

[7] Yen, Howard, Tianyu Gao, and Danqi Chen. "Long-Context Language Modeling with Parallel Context Encoding." ACL 2024.

[8] Dong, Qian, et al. "Decoupling Knowledge and Context: An Efficient and Effective Retrieval Augmented Generation Framework via Cross Attention." The Web Conference 2025.

Thank you again for your feedback. Before the author-reviewer discussion period closes, are there are any other questions we can answer? We made a strong effort to address all your points, including new experiments and paper edits, and we would greatly appreciate it if you would consider increasing your score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.