Learning What Retrieval Misses: Federated Retrieval-Residual Memory for Language Models

Retrieval-augmented language models rely on a retriever to supply what their parameters lack, but a retriever returns only its top-ranked passages, and part of the evidence in the very corpus it searches is never among them. Because a deployed retriever is fixed, the same evidence is missed every time and nothing in the system learns from the miss. Yet these persistent misses mark the knowledge a model should carry in its parameters. Memorizing documents indiscriminately spends capacity instead on knowledge that retrieval delivers reliably, and it can make the model answer from memory even where the retrieved context is already correct. We introduce Federated Retrieval-Residual Memory (FedR²M), which uses the persistent misses of the fixed retriever as the criterion for what to memorize. Each data owner becomes a client and discovers its retrieval residuals, the questions whose supporting evidence its own corpus holds but its fixed retriever misses. The client then consolidates the missing knowledge into a low-rank adapter in three stages that write it, force the model to recall it from parameters, and protect the questions that retrieval already serves correctly. A server averages the client adapters into one global adapter, so memory built from one client's retrieval residuals serves all clients while raw data stays local, and inference pairs that global adapter with the unchanged retriever for every question, without per-question routing. Across four question-answering benchmarks, FedR²M improves macro F1 by a relative 36% or more over every baseline in our comparison, gains 30% on retrieval hits and 59% on retrieval misses over retrieval-only prompting, and, without pooling raw data, matches centralized training of the same memory. The entire shared memory is one 13 MiB adapter, about a thousandth of the 13 GiB of per-passage adapters that document parameterization builds for the same questions.