acceptodds
Under review as a conference paper at ICLR 2027

Learning What Retrieval Misses: Federated Retrieval-Residual Memory for Language Models

Abstract

Retrieval-augmented language models rely on a retriever to supply what their parameters lack, but a retriever returns only its top-ranked passages, and part of the evidence in the very corpus it searches is never among them. Because a deployed retriever is fixed, the same evidence is missed every time and nothing in the system learns from the miss. Yet these persistent misses mark the knowledge a model should carry in its parameters. Memorizing documents indiscriminately spends capacity instead on knowledge that retrieval delivers reliably, and it can make the model answer from memory even where the retrieved context is already correct. We introduce Federated Retrieval-Residual Memory (FedR²M), which uses the persistent misses of the fixed retriever as the criterion for what to memorize. Each data owner becomes a client and discovers its retrieval residuals, the questions whose supporting evidence its own corpus holds but its fixed retriever misses. The client then consolidates the missing knowledge into a low-rank adapter in three stages that write it, force the model to recall it from parameters, and protect the questions that retrieval already serves correctly. A server averages the client adapters into one global adapter, so memory built from one client's retrieval residuals serves all clients while raw data stays local, and inference pairs that global adapter with the unchanged retriever for every question, without per-question routing. Across four question-answering benchmarks, FedR²M improves macro F1 by a relative 36% or more over every baseline in our comparison, gains 30% on retrieval hits and 59% on retrieval misses over retrieval-only prompting, and, without pooling raw data, matches centralized training of the same memory. The entire shared memory is one 13 MiB adapter, about a thousandth of the 13 GiB of per-passage adapters that document parameterization builds for the same questions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, parametric memory, retrieval failure, federated learning, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JTBgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Federated Retrieval-Residual Memory (FedR2M), which uses the questions whose supporting evidence a fixed retriever fails to return (retrieval residuals) as the criterion for what a model should memorize. Each client trains a LoRA adapter in three stages: write with the missing evidence, recall without it, and protect retrieval hits. A server averages the adapters with FedAvg into one global adapter, which is used with the unchanged retriever at inference. The training questions come from the HotpotQA and 2Wiki training splits and are hashed into six clients. The main comparison uses 1,200 questions from HotpotQA, 2Wiki, PopQA and CWQ with a shared 1,500-document corpus. With Llama-3.2-1B, FedR2M reaches 0.4074 macro F1, versus 0.2984 for retrieval-only prompting and 0.2749 for FedMosaic, and matches centralized training of the same adapter (0.4054). In a comparison with Parametric RAG it stores one 13 MiB adapter instead of 13.23 GiB of per-passage adapters. A blind test with an 8B centralized adapter compares the residual boundary against permuted and no-boundary controls.

Critical strengths

S1: The statistical reporting is careful.

The main comparisons come with paired bootstrap 95% confidence intervals, and FedR2M and centralized training are rerun with three seeds. The blind test hashes all predictions before the answers are revealed, and the appendix tables make it possible to cross-check most reported numbers. This makes the results easy to audit, but it does not address the design problems below.

S2: The paper reports some useful side observations.

Two document-adapter systems score below retrieval-only prompting on retrieval hits: FedMosaic at 0.4109 and DistilledPRAG at 0.3282, versus 0.4600 (Table 1). The recall and protection stages also trade off against each other on the holdout (Table 12). The cost accounting (Tables 5 and 15) is also more complete than usual. These observations are informative, but they are secondary to the paper's main claim.

Critical weaknesses

W1: The motivation does not hold up: if the retriever is the problem, the retriever should be improved.

The paper's case against retriever-side fixes is brief: "cost and stability keep a production retriever frozen, and any replacement is still a fixed retriever with misses of its own" (l. 46–47; see also l. 103–104). I do not find this convincing, for three reasons.

  1. The supervision is already there. The method requires gold supporting-fact annotations for every training question (Eq. 3, Appendix A). These are exactly the labels needed to train a reranker over the frozen BM25 top-N, or a learned query expansion. Either would leave the index unchanged, and neither is obviously more costly than fine-tuning the generator.
  2. Generation-side memory has narrower reach. It mainly helps questions whose missing knowledge was actually memorized. A retriever-side improvement applies to any future question that needs the same evidence.
  3. Retriever-side alternatives are evaluated only partially. The paper swaps BM25 for an off-the-shelf dense retriever (0.2852 vs. 0.2984 macro F1 at 1B, Table 1; 0.3270 vs. 0.3172 at 8B, Table 9) and compares three silo-routing systems. BM25 top-5 nevertheless misses about half of the main-comparison questions (593 of 1,200). No reranker, larger K, or retriever or reranker trained on the residual labels is tested.

W2: The novelty is limited.

The main components already exist:

  • Parametric knowledge for RAG. Writing knowledge into LoRA parameters for retrieval-augmented generation is the idea of Parametric RAG, and FedMosaic extends it to the federated setting. The paper cites both.
  • Training on simulated retrieval success and failure. Training on examples with and without the supporting documents in context, so that the model uses the context when it is relevant and its parameters otherwise, has been studied by RAFT (Zhang et al., 2024; arXiv:2403.10131) and PA-RAG (Bhushan et al., Findings of NAACL 2025; arXiv:2502.08356). The paper cites neither.
  • Aggregation. The paper itself presents its aggregation step as "standard federated averaging" (l. 242).

The new elements are two. First, the training questions are chosen by a fixed retriever's misses rather than by random or constructed buckets. Second, the stages are ordered as write, recall, protect. By my calculation from the paper's own budget-matched control (see W3), the residual filter accounts for only about 10–17% of the gain over retrieval-only prompting.

W3: The experiments do not convincingly show that the passages the retriever misses are an effective learning signal.

The headline experiment cannot test this.

  • The residuals are discovered on HotpotQA and 2Wiki training questions, each ranked only within its own 10 candidate paragraphs. The main comparison instead runs under "a retriever and corpus other than the ones the residuals were discovered on" (Appendix A, l. 691–692).
  • None of the evaluation questions is drawn from the training folds.
  • PopQA and CWQ have no training data at all, yet show the largest gains over retrieval-only prompting (+0.116 and +0.186 F1, Table 1).
  • No evaluation asks the memorized residual questions again, nor their paraphrases, nor new questions that need the same evidence.

Where the boundary is tested, its effect is small. These tests use only the 8B centralized adapter.

  • Budget-matched control. It keeps the same pipeline and budget and only removes the residual filter.
    • On the four-benchmark set it is 0.0168 F1 below the full method, 95% CI [−0.0019, 0.0353], and it was evaluated after the answers were revealed (Appendix B).
    • On the blind test set it is 0.0261 F1 (4%) below, 95% CI [0.0079, 0.0450], and 0.0492 below on the residuals. The difference is significant but small.
    • The original no-boundary control is 13% below, but it is not budget-matched.
  • Share of the gain. By my calculation from Tables 3 and 17 and Appendix B, the budget-matched control recovers about 90% (four-benchmark set) and 83% (blind test) of the gain over retrieval-only prompting.
  • Permuted control. In the write and recall stages, it pairs each residual question with another question's evidence and answer; the protection stage is unchanged. It still recovers 57% and 43% of the gain.

These results suggest that most of the gain comes from general answer-format and reading skills learned through supervised fine-tuning, with only a small additional effect from the residual filter.

W4: The comparison with the baselines is not like-for-like: the baselines do not receive the same supervision.

FedR2M is the only method in Table 1, apart from its own centralized reference, that is trained on gold, human-annotated question–answer pairs from the benchmarks' training splits. These are 4,400 examples built from 2,800 HotpotQA and 2Wiki training questions, plus gold evidence in the write examples. The baselines receive weaker supervision or none:

  • Retrieval-only prompting, dense retrieval, the local model and the three routing systems leave the generator untrained.
  • FedMosaic trains its document adapters on LLM-generated rewrites and question–answer pairs about the documents.
  • DistilledPRAG's adapter generator is trained on synthetic question–answer pairs.
  • Neither FedMosaic nor DistilledPRAG uses gold question–answer pairs.

The SFT-style controls trained on the same question pool (no-boundary, budget-matched, permuted) exist only for the 8B centralized adapter (Tables 3 and 17). The main comparison has no equal-budget LoRA RAG-SFT or RAFT baseline.

Answer format clearly matters here. The 8B model's retrieval-only EM (0.1833, Table 9) is below the 1B model's (0.2158, Table 1). At 8B, FedR2M raises EM 2.28-fold but F1 only 1.53-fold.

The same-question comparison with Parametric RAG has related issues:

  • The reported PRAG score (0.2566) is close to pure P-RAG with LLaMA-1B in Su et al. (2025a, Table 1): the unweighted mean of its four dataset-level F1 scores is 0.2613 by my calculation. The corresponding mean for the stronger Combine Both variant is 0.3247. The paper does not state which variant it ran, although FedR2M itself reads the top-3 passages in context.
  • The storage claim ("about a thousandth") compares adapters trained on the test passages (PRAG) with an adapter that was not trained on any of them.

W5: The federated setting looks forced: it costs accuracy, and privacy is addressed only nominally.

The setting is artificial. The six clients are hash partitions of two public datasets (l. 685–687).

Federation costs accuracy and time.

  • On the in-distribution holdout, the federated adapter scores 0.6381 versus 0.7684 for centralized training (Table 12). This holds even though federated training makes six times as many example visits: 19,584 vs. 3,200, with 102 local updates per client vs. 100 centralized updates (l. 728–732).
  • Federated training takes 5.8 times as long in the paper's single-GPU simulation (l. 433).
  • The paper says FedR2M "matches centralized training". This holds only on the main comparison, whose questions and corpus differ from those on which the residuals were discovered.

Privacy is defined only as data locality (l. 474), with stronger guarantees left as orthogonal (l. 474–477). Even data locality is not respected in the experiments:

  • At inference, the shared BM25 ranks all 1,500 documents across the six silos, and the top-5 passages are sent in plaintext (Appendix A, l. 692–693; Table 15). This departs from Eq. 8, which retrieves from the serving client's own corpus. The paper acknowledges the plaintext transfer (l. 476–477) but does not address it.
  • The global adapter is designed as "the only channel through which a retrieval residual observed at one client can change how another client's questions are answered" (l. 243–245). The recall stage trains it to output the private answers without evidence. This is exactly the channel that an extraction or membership-inference attack would target.
  • No such attack is evaluated. FedMosaic, which the paper compares against, does report targeted and prefix data-extraction attacks against its adapters.

These issues severely limit the method's practical value.

Questions for authors

Q1: How does FedR2M compare with spending the same supervision on the retriever?

This is a requested experiment that could change my assessment. Please train a reranker, or fine-tune the retriever, on the same residual labels and gold supporting facts. Please also report retrieval-only prompting with a larger K and with a stronger retriever. If these close most of the gap, the motivation for parametric memory does not hold.

Q2: Is the missing knowledge actually stored and used?

This is a requested experiment. Please evaluate on three sets: (a) the training residual questions themselves, (b) their paraphrases, and (c) new questions whose supporting evidence is a training residual. Please compare against an equal-budget LoRA RAG-SFT adapter and a RAFT-style adapter trained on the same questions without the residual filter.

Q3: Does the residual boundary matter in the federated 1B setting?

This is a requested experiment. Please run the budget-matched no-boundary control and the permuted control with the 1B federated adapter on the main comparison, evaluated blind.

Q4: Why does the paper claim that FedR2M "matches centralized training"?

This is a clarification request. Federated training is 0.13 F1 below centralized training on the holdout (Table 12). What is the gap when both use the same number of updates?

Q5: What exactly is private in the proposed system?

This is a clarification request with a requested experiment. Please explain how reading plaintext top-5 passages from all silos at inference fits the premise that corpora cannot be pooled. Please also report extraction or membership-inference attacks against the shared adapter, as FedMosaic does.

Reviewer_twFiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedR2M, a federated framework that selectively stores knowledge missed by a fixed retriever in a shared LoRA adapter. Retrieval residuals are identified using local evidence availability and top-KK retrieval coverage. A three-stage write--recall--protect procedure consolidates missing knowledge while attempting to preserve retrieval-supported predictions. Client adapters are aggregated through FedAvg and deployed alongside the unchanged retriever. Experiments on four QA benchmarks report improved F1 over retrieval-only, routing, and document-adapter baselines, with ablations examining residual selection, training stages, and federated aggregation.

Critical strengths

S1. Retrieval-aware memory selection. The formulation zq=vq(1−hq)z_q=v_q(1-h_q) provides an intuitive criterion for allocating parametric capacity to evidence missed by retrieval, rather than indiscriminately memorizing documents. The budget-matched no-boundary ablation offers useful evidence for this design.

S2. Explicit treatment of memory--retrieval interference. The write--recall--protect pipeline directly addresses the tension between memorized knowledge and retrieved evidence. Stage-wise experiments demonstrate that improving residual recall can hurt retrieval hits, motivating the final protection stage.

Critical weaknesses

W1. The residual boundary relies on imperfect evidence proxies. Retrieval coverage is determined largely through answer-alias matching and annotated supporting passages (Appendix A). Answer presence does not guarantee sufficient evidence, while semantically sufficient passages may lack exact aliases. Consequently, the claimed boundary between retrieval failure and knowledge deficiency may be noisy, particularly for multi-hop questions.

W2. Limited realism of the federated setting. Training uses question-associated candidate paragraphs, six clients constructed through hashing, and residuals primarily from HotpotQA and 2Wiki. This differs substantially from real federated deployments with independent corpora, heterogeneous distributions, and potentially overlapping knowledge. The observed benefits may not generalize to realistic client heterogeneity.

W3. Parametric knowledge acquisition is insufficiently distinguished from supervised QA adaptation. Both write and recall stages directly supervise question--answer pairs. Improvements could therefore reflect learning benchmark-specific answer mappings or task adaptation rather than storing reusable facts from missing documents. The current controls do not conclusively isolate this distinction.

W4. Some central conclusions exceed the experimental evidence. The federated procedure uses 612 updates versus 100 centralized updates, complicating claims of matching centralized training. Moreover, overlapping confidence intervals do not establish equivalence. The claimed importance of stage ordering is also less convincing on the main benchmark, where the alternative-order difference has a confidence interval containing zero.

W5. Privacy and deployment costs remain insufficiently validated. Keeping raw data local does not guarantee that shared LoRA updates preserve sensitive information. No leakage or membership-inference evaluation is provided. Moreover, the reported approximately 0.42-second inference overhead raises questions about the practical advantages of the shared adapter.

Questions for authors

Q1. How accurately do the alias-based retrieval labels reflect semantic evidence sufficiency? Can the authors report a manually verified subset or evaluate sensitivity to alternative residual definitions?

Q2. How does FedR2M perform when residual discovery and inference use the same realistic, full-scale local corpus rather than question-associated candidate paragraphs? Does the advantage persist under larger KK or stronger retrievers?

Q3. Can the authors test whether the adapter learns transferable facts rather than question-specific mappings, for example by evaluating paraphrased, compositionally different, or previously unseen questions requiring the same memorized evidence?

Q4. How robust is federated aggregation under strongly non-IID client distributions, unequal corpus sizes, conflicting facts, or clients with very few retrieval residuals?

Q5. Can the authors provide an optimization- and data-exposure-matched comparison with centralized training and demonstrate statistical equivalence, rather than relying on a nonsignificant difference?

Q6. Why does reversing the stage order yield a large degradation on the training-side holdout but no statistically significant overall degradation on the main comparison? How sensitive are the results to stage duration and protection-set composition?

Q7. Can the authors evaluate privacy leakage from uploaded adapters, particularly extraction of client-specific memorized facts or membership inference?

Reviewer_FsoROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a means of learning from retrieval failures for RAG systems. The process requires gold retrievals, which are collected and learned parametrically with LORA. Additionally the system is federated, so different corpus consumers can update their retrievers differently.

Critical strengths

Learning from retrieval failures is an underexplored area AFAIK, and it's nice to see more work explicitly on this.

Critical weaknesses

The definition of h_q (eq 2) seems to require gold relevance information, which is not generally available except in rather artificial settings. This is at odds with the focus on production-ready approaches.

The learned adapters are parametrically storing information in text, which seems less likely to generalize well to new questions than changes to the retriever. The authors argue that changing a dense retriever is awkward technically in production but the federation of adapters also seems like a heavy and complex solution to the problem.

Questions for authors

Can you give some concrete evidence about the tradeoff in quality between this and simpler approaches (eg, adding residuals with associated questions to the corpus, updated the dense retriever?)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.