- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Retrieval Layers, a sparse cross-attention mechanism that allows language models to access external information through a precomputed token-level KV bank instead of prefilling retrieved documents into the context. The authors integrate a Retrieval Layer into Qwen3-4B and propose a training procedure combining supervised fine-tuning, document-access loss, and reinforcement learning. Experiments on multi-hop QA and long-context benchmarks show improved accuracy and latency trade-offs compared with conventional RAG baselines. The method also demonstrates strong performance on long conversation reasoning and maintains high needle retrieval accuracy at context lengths of up to 30M tokens.
Critical strengths
-
The paper addresses an important efficiency limitation of conventional RAG systems. By accessing precomputed document representations through sparse cross-attention, the proposed method avoids repeatedly prefilling retrieved documents into the LLM context. The architecture can also be integrated into existing pretrained models without requiring full pretraining.
-
The experimental results are promising. ReLay-LM achieves competitive accuracy with substantially lower inference latency on several multi-hop QA benchmarks. The long-context results are also impressive, particularly the ability to maintain high needle retrieval accuracy at 30M tokens.
Critical weaknesses
-
The architectural novelty is limited, and several closely related studies are missing from the discussion. Previous work such as LongMem [1], CEPE [2], and DecoupledRAG [3] has already explored accessing external document representations through attention without directly incorporating retrieved text into the LLM context. In particular, CEPE and DecoupledRAG have demonstrated similar architectures that can be integrated into existing pretrained LLMs. The main differences appear to be sparse top-k attention, a single retrieval layer, and the training procedure, but the paper does not sufficiently discuss or compare against these studies.
-
The baseline comparisons are unfair due to substantial differences in task-specific training. ReLay-LM jointly trains the 4B generator, 0.6B encoder, and additional retrieval parameters using 100K SFT steps on 1.6M QA examples, followed by RL on the training splits of HotpotQA, MuSiQue, and 2WikiMultihopQA, which are also used for evaluation. In contrast, the RAG baselines use the original Qwen3-4B without comparable task-specific fine-tuning or RL. Table 3 shows that ReLay-LM before RL underperforms RAG@10 on all three benchmarks, making it difficult to attribute the final improvements to the proposed architecture rather than the additional training. ReLay-LM also uses 50 retrieved documents compared with at most 10 for RAG. The lack of key ablations on document-access loss and joint encoder training, together with missing latency and memory measurements at 30M tokens, further limits the experimental evidence.
[1] Augmenting Language Models with Long-Term Memory. NeurIPS 2023.
[2] Long-Context Language Modeling with Parallel Context Encoding. ACL 2024.
[3] Decoupling Knowledge and Context: An Efficient and Effective Retrieval Augmented Generation Framework via Cross Attention. WWW 2025.
Questions for authors
-
How does the proposed method differ from closely related architectures such as CEPE and DecoupledRAG, and what are its specific contributions beyond sparse top-k attention?
-
Could the authors compare ReLay-LM against RAG baselines trained with the same task-specific data and comparable SFT and RL budgets, as well as under matched retrieval budgets?
-
How much do document-access loss and joint encoder training contribute to the final performance, and could the authors provide corresponding ablations?
-
What are the encoding overhead, retrieval latency, and peak memory consumption at 1M, 10M, and 30M tokens, and how is the token-level KV bank stored at these scales?
-
Since the multi-hop QA experiments still rely on an external retriever to select the initial 50 documents, could the authors clarify the scope of the claimed end-to-end retrieval optimization?
5 comments
Rebuttal Continued
- DecoupledRAG [3] relies on a fixed external retriever: An external retriever selects K documents per query, whose keys and values are precomputed by the frozen LLM. The LLM learns to densely cross-attend to the KV cache of retrieved documents using only a low-rank gate. Once again, not training end-to-end sacrifices accuracy and retrieval ability. Furthermore, DecoupledRAG's experiments only deal with non-reasoning models. Due to a lack of readily available SFT data for corpus-retrieval QA, it is not obvious how to adapt an existing reasoning model to cross-attend to documents. We show that our SFT data generation scheme (Section 3.2) is able to close this gap and that our reasoning models can further benefit from RL.
"...what are its specific contributions beyond sparse top-k attention?..." Beyond the usage of sparse top- attention, ReLay-LM contributes end-to-end training of retrieval with the LLM, a recipe for adapting existing reasoning models to retrieval, and a design that keeps token-level lookup fast and memory-efficient at tens of millions of tokens. The revision adds results supporting each:
-
End-to-end training of retrieval and generation. The encoder, the Retrieval Layer, and the LLM are trained jointly. We ablate the importance of this joint training by freezing each backbone (Section 4.6, Figure 5, Appendix G.3). We find that freezing the LLM reduces multi-hop accuracy relatively by 19%, freezing the encoder reduces it by 33%, and freezing either drops retrieval performance (measured by average gold document attention mass) by at least 4x. Furthermore, we find that the document-access loss is crucial for ReLay-LM's ability to extract relevant information from a large KV bank. Removing it reduces accuracy by 36.2% relatively and collapses retrieval performance, with gold attention mass falling from 0.249 to below 0.001 (Appendix G.4).
-
End-to-end adaptation of reasoning models. Prior methods either require pretraining (REALM [5], Atlas [6], RETRO [7], MSA [8]) or fine-tune non-reasoning LLMs (RA-DIT [9], ImpRAG [10], DecoupledRAG [3]). There is no readily available SFT data for reasoning over a retrieved corpus, so we generate it from Qwen3-4B's own reasoning traces (Section 3.2), which preserves the model's reasoning style. We then apply a modern RL stack using DAPO with verifiable rewards. RL improves ReLay-LM's accuracy by 39% relatively compared with 14% for the training-matched RAG@20 baseline (Table 4).
-
Lookup speed and memory efficiency. Token-level retrieval over corpora of millions of tokens is only practical if KV caches are small and decode throughput remains fast. Our ablation shows that a single Retrieval Layer is sufficient, allowing us to substantially reduce GPU memory (Appendix G.9). Furthermore, our inference cost scaling benchmarking (see below) shows that by training with sparse attention, we can utilize approximate top- algorithms to achieve high throughput even at 32M token banks (Figure 4).
"...How much do document-access loss and joint encoder training contribute to the final performance, and could the authors provide corresponding ablations?..." We agree, and the revised submission adds both ablations. They are summarized in Section 4.6 (Figure 5), with full results in Appendix G.3 and G.4. To make multi-seed ablations computationally feasible, our ablations shorten SFT from 100K to 10K steps and skip RL. Accuracy is averaged over HotpotQA, 2WikiMultihopQA, and MuSiQue and gold mass is the fraction of retrieval attention assigned to gold documents.
Joint encoder training. We freeze the document encoder, the LLM, or both, while the Retrieval Layer remains trainable (Table 9):
| Variant | Accuracy (%) | Gold mass |
|---|---|---|
| Default | 21.1 ± 1.6 | 0.248 ± 0.007 |
| Frozen LLM | 17.0 ± 0.8 | 0.062 ± 0.001 |
| Frozen encoder | 14.1 ± 0.7 | 0.041 ± 0.001 |
| Frozen encoder + LLM | 14.9 ± 0.8 | 0.019 ± 0.001 |
Freezing the LLM, the encoder, or both reduces accuracy by 19%, 33%, and 29% relatively, and reduces gold mass by 4x, 6x, and 13x.
Document-access loss. Training stages 1–2 use the document-access loss alone with weight 0.1. We vary its weight in stages 3–4, where it is added to the language-modeling loss, and separately remove it from every stage ("Off") (Table 10):
| Accuracy (%) | Gold mass | |
|---|---|---|
| Off | 13.9 ± 0.7 | 0.000 ± 0.000 |
| 0 | 16.3 ± 1.9 | 0.033 ± 0.002 |
| 0.01 | 16.9 ± 1.0 | 0.075 ± 0.004 |
| 0.1 (default) | 21.8 ± 1.4 | 0.249 ± 0.010 |
| 1.0 | 16.9 ± 1.8 | 0.315 ± 0.001 |
The document-access loss is necessary for sparse retrieval. However, placing too much weight on it harms overall QA accuracy. The default weight of 0.1 gives the highest accuracy.
Rebuttal Continued
"...What are the encoding overhead, retrieval latency, and peak memory consumption at 1M, 10M, and 30M tokens, and how is the token-level KV bank stored at these scales?..." We agree that encoding overhead, retrieval latency, and peak GPU memory are important for determining practical usage. The revised submission adds an inference-cost scaling study up to 32M tokens on two H100 GPUs (Section 4.5, Figure 4, Appendix F). It compares ReLay-LM with exact top-k search, ReLay-LM with approximate top-k search (CAGRA), and Qwen3-4B/Qwen3.5-4B with the same evidence in their context. Below we highlight results for 1M, 8M, and 32M tokens.
Encoding Overhead (s)
| Method | 1M | 8M | 32M |
|---|---|---|---|
| ReLay-LM: bank encoding | 4.6 | 37.1 | 150.2 |
| ReLay-LM: CAGRA index construction (approx. top-k only) | 11.7 | 122.8 | 551.3 |
| Qwen3.5-4B: prefill | 64.1 | OOM | OOM |
Time per decoded token, including retrieval (ms)
| Method | 1M | 8M | 32M |
|---|---|---|---|
| ReLay-LM (exact top-k) | 3.38 | 6.09 | 14.97 |
| ReLay-LM (approx. top-k) | 3.38 | 3.53 | 3.54 |
| Qwen3.5-4B | 8.91 | OOM | OOM |
Peak GPU memory, summed over both GPUs (GiB)
| Method | 1M | 8M | 32M |
|---|---|---|---|
| ReLay-LM (exact top-k) | 21.1 | 50.3 | 146.3 |
| ReLay-LM (approx. top-k) | 23.1 | 60.5 | 159.3 |
| Qwen3.5-4B | 52.2 | OOM | OOM |
The KV bank is encoded once per corpus, so this overhead appears offline and is negligible for online queries. For comparison, we show Qwen3.5-4B's prefill time, which is required for every query.
At 1M tokens, ReLay-LM with exact top-k has 1,300x lower TTFT than Qwen3.5-4B (14x including bank encoding), decodes 2.6x faster, and uses 2.5x less GPU memory. At 32M tokens, approximate top-k maintains per-token latency at 3.5 ms, vs. 15.0 ms with exact top-k. This highlights that approximate search with sparse attention enables efficient token-level lookup at massive contexts. See Section 4.5 for more details.
In a Retrieval Layer, each corpus token is encoded once and stored as a single key and value. The bank is kept in GPU memory and sharded by rows across the two GPUs. Keys and the search data are stored in INT8 with FP32 per-row key scales and values are stored in FP16 (Appendix A.3).
Additional New Results.
- Attention concentration analysis: we analyze how the fraction of attention mass on the gold documents of multi-hop questions changes as we add more distractor documents to the corpus, up to about 1M tokens (Section 4.4, Figure 3, Table 7). We find that full-context models (Qwen3-4B, Qwen3.5-4B) degrade quickly while ReLay-LM's gold attention mass tapers off slowly. At about 1M tokens, ReLay-LM keeps on average 59% of its attention on the gold documents, 36 times more than full-context Qwen3.5-4B.
- Ablations: we ablate retrieval layer position, dense vs. sparse attention, multiple Retrieval Layers, document-level retrieval, doc access loss, frozen encoder/LLM, the training schedule, retrieval layer temperature, key quantization, and approximate top-k (Section 4.6, Appendix G). These ablations support the individual design choices in ReLay-LM.
- Expanded related work: we discuss and compare RA-DIT [9], ImpRAG [10], memory layers [11, 12], LongMem [1], CEPE [2], and DecoupledRAG [3] (Section 2, Table 1, Appendix G).
Rebuttal
We thank the reviewer for recognizing that our method addresses an important efficiency limitation of RAG and efficiently integrates into existing pretrained models, and for finding our long-context results impressive. Below we address the remaining concerns.
"...Could the authors compare ReLay-LM against RAG baselines trained with the same task-specific data and comparable SFT and RL budgets..." We agree this comparison is important, and we have added exactly this baseline (see Figure 1). Qwen3-4B RAG is now trained on the same 1.6M SFT samples for the same 100K steps, then with the same DAPO [4] RL recipe and hyperparameters as ReLay-LM, retrieving the top 20 chunks per rollout (see Appendix B.1 for setup). Averaged over HotpotQA, MuSiQue, and 2WikiMultihopQA (Table 4):
Under the same training budget, ReLay-LM achieves higher average Multi-hop QA accuracy than the best RAG configuration with 3.3x lower TTFT. Furthermore, the improvement from the SFT checkpoint to the RL checkpoint is significantly larger for ReLay-LM (39% relative improvement) than for RAG (14% relative improvement). This suggests that the end-to-end learnability of ReLay-LM allows it to benefit more from post-training.
"...still rely on an external retriever to select the initial 50 documents..." / "...compared with at most 10 for RAG..." To strengthen our end-to-end optimization, we now perform RL with lookups over the full benchmark corpus, rather than using an intermediate external retriever. We find RL still yields comparable improvements in multi-hop QA accuracy. In the revised submission, at inference, ReLay-LM conducts sparse lookup across the entire corpus at each decoding step, and does not use an external retriever at all. See the table above for how ReLay-LM compares to RAG@20.
"...several closely related studies are missing from the discussion. Previous work such as LongMem [1], CEPE [2], and DecoupledRAG [3]..." We thank the reviewer for these references, and our revised submission now includes discussion of all 3 in Section 2.2 and Table 1, as well as ablations that test their main design choices inside ReLay-LM (Appendix G). We comment on the suggested references below:
LongMem [1] keeps its LLM frozen and retrieves chunks rather than tokens: It caches the keys and values computed by a frozen LLM, and trains a side network that retrieves chunks of this cache using mean-pooled chunk keys, then attends to every token of the retrieved chunks. ReLay-LM instead trains the encoder, the Retrieval Layer, and the LLM jointly. We ablate the importance of this joint training by freezing each backbone (Section 4.6, Figure 5, Appendix G.3). We find that freezing the LLM reduces multi-hop accuracy relatively by 19%, freezing the encoder reduces it by 33%, and freezing either drops retrieval performance (measured by average gold document attention mass) by at least 4x. We also test LongMem's chunk-level lookup inside ReLay-LM's setup (Appendix G.10). We search over chunks using a single pooled key per chunk and then perform token-level dense attention on the selected chunks. Selecting just one chunk retrieves 4 times as many values as ReLay-LM and still reduces accuracy by 33.6% relative to ReLay-LM. Retrieving more chunks (4-5) does not outperform ReLay-LM.
CEPE [2] does not train retrieval end-to-end and uses dense attention: CEPE only finetunes cross-attention layers (rest of the LLM is frozen) to attend to a KV bank produced by a lightweight trained encoder. Although ReLay-LM shares some architectural similarities, the lack of fully end-to-end training, sparse attention, and a document-access loss limits its suitability for retrieval over large corpora. We find that if we use dense attention instead of sparse attention, accuracy is reduced by 16% relatively and retrieval performance drops by 33% (Appendix G.6). Furthermore, we find that the document-access loss is crucial for ReLay-LM's ability to extract relevant information from a large KV bank. Removing it reduces accuracy by 36.2% relatively and collapses retrieval performance, with gold attention mass falling from 0.249 to below 0.001 (Appendix G.4).