acceptodds
Under review as a conference paper at ICLR 2027

From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory

Abstract

Retrieval-augmented generation (RAG) helps large language models (LLMs) answer factual questions beyond their training data, but it adds storage and inference costs. Distilling previously retrieved information into parametric memories enables reuse without online retrieval. However, training the memory to mimic a retriever rewards accurate independent predictions, which do not necessarily improve the final output quality when combined with the LLM's existing predictions. To address this, we introduce ResMem, in which the LLM reads retrieved text offline to generate teacher targets. This design avoids storing massive token representations, unlike existing parametric memory approaches such as MLP Memory, which relies on k-nearest neighbor (kNN) datastores. Furthermore, by using the LLM as an offline teacher that reads privileged retrieved evidence alongside the input, we train a parametric memory to learn an additive logit residual. This residual aligns the predictions of a student sharing the same backbone with the teacher distribution, while self-distillation updates remain confined to the parametric memory. On a Wikipedia subset, ResMem reduces the storage retained for teacher construction by 94.7% compared with the evaluated kNN pipeline. Across five question answering datasets, ResMem raises mean token F1 from 32.83% for the frozen LLM to 34.48%. It also breaks 21% fewer of the LLM's correct answers than a kNN-trained memory. These findings validate our motivation of treating retrieval as a learning resource, with parametric memory shaped by how retrieved knowledge is used to inform the model's future predictions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parametric Memory, Residual Learning, Knowledge Distillation, Knowledge Augmentation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_4hRWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ResMem, which trains an MLP memory to adjust a frozen LLM’s logits. The same LLM reads retrieved passages to provide offline teacher targets, and the memory is trained with distillation, while inference uses only the base model and memory. The main changes from MLP Memory are the teacher and the residual training objective. Across five QA datasets, average F1 increases from 32.83 to 34.48. The paper also measures construction storage, inference costs, and which answers are helped or hurt.

Critical strengths
  • The full method improves EM and F1 over the base on all five QA datasets. It also improves on MLP Memory overall at essentially the same inference cost. Avoiding the large kNN datastore is another useful practical benefit.

  • The correction/harm analysis is informative. Separating newly corrected answers from previously correct answers that become wrong helps explain the results.

Critical weaknesses
  • I think the benefit of the residual formulation over simpler distillation is still unclear. With the same teacher, its F1 advantage over standalone training is only 0.117 points (Table 2). A natural alternative is to distill the same teacher directly into LoRA. The paper includes LoRA, but I could not tell whether it uses the same teacher, loss, and training exposure. This seems an important comparison for justifying the 1.54B-parameter memory.

  • The overall cost trade-off is not fully evaluated. The reported 94.7% saving concerns retained teacher-construction files. Both pipelines subsequently train from cached targets. Without target-generation and memory-training costs, it is difficult to tell whether the new pipeline is cheaper overall or mainly reduces disk storage. This matters because the teacher change accounts for much of the practical contribution.

Questions for authors
  1. Does LoRA use the same teacher, loss, and training budget? What is the motivation for fitting the residual instead of simple distillation?
  2. How expensive is it to construct the teacher target?
  3. What are the QA results for the paired kNN-trained models in Figure 4b?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4hRW (1/2)

We thank the reviewer for the concrete and constructive questions. Each of them led to a new experiment, and we believe the answers make the paper stronger.

How to read this response. W1, W2, … and Q1, Q2, … number the weaknesses and questions in the order of your review; where a weakness and a question raise the same point, we answer them together. G1–G5 refer to our General Response, posted as a separate comment to all reviewers, which summarizes the points that several reviewers raised. Each answer below is self-contained: where it relies on a result, the table is repeated here, so there is no need to look it up elsewhere. In the revised PDF, passages changed in response to your comments are set in brown, with a margin tag naming the point they answer (e.g., 4hRW Q3). A passage that answers several reviewers carries one reviewer’s colour, and its tag lists every point it answers, so all changes for you can be found by the tag “4hRW”.

W1, Q1. The benefit of the residual formulation over simpler distillation is unclear. Does LoRA use the same teacher, loss and training budget, and what motivates fitting a residual?

Thank you for this important question. The LoRA in Table 1 is a baseline trained without our teacher, so for a direct comparison we retrained it with the same teacher, loss and exposure as ResMem, with rank 588 so that it has the same 1.54B parameters as ResMem’s memory (G2). The dev-selected adapter scores 25.77 EM / 32.93 F1 (base 25.91 / 32.83), and ResMem reaches 26.03 / 33.24, +0.31 F1 [+0.10, +0.51] above LoRA. We swept the learning rate from 5e−6 to 1e−4 and, as for γ\gamma, scaled the adapter update by a coefficient chosen on development data; the selected adapter uses learning rate 5e−6 and scale 0.2, and at every larger learning rate development selection picks scale 0.

EM (%)F1 (%)Substring (%)Repairs (#)Breaks (#)Net (#)
Base25.9132.8327.73———
Base + LoRA25.7732.9327.86486600−114
Base + ResMem (Ours)26.0333.2428.49370305+65

The last three columns count the base errors each model repairs and the base-correct answers it breaks on the test set, and they give our motivation for fitting a residual instead of distilling into the model. LoRA repairs 486 base errors but breaks 600 base-correct answers, while ResMem repairs 370 and breaks 305. Writing the teacher’s corrections into the backbone disturbs answers the base already gets right, whereas a residual trained through the combined prediction only changes the positions where the base and the teacher disagree. The 0.117 F1 over a standalone memory comes from a teacher that stays close to the base; with a sharper kNN teacher held fixed, the residual objective adds +1.30 F1 (Q3). The advantage also does not depend on the 1.54B size: at 0.49B a residual memory still beats a standalone memory of the same size (G1).

AuthorsOpenReview ↗

Response to Reviewer 4hRW (2/2)

W2, Q2. The 94.7% saving concerns retained teacher-construction files. How expensive is it to construct the teacher target, and is the pipeline cheaper overall or mainly in disk?

Thank you for the question. The pipeline is cheaper in compute as well as in disk (G4). On full English Wikipedia with one 8×A100 node, the BM25 index takes 0.5 hours on CPU (charged as 4 GPU-hours) and the teacher pass over all training spans 677 GPU-hours. With 371 GPU-hours of training, ResMem takes 1,052 GPU-hours end to end. MLP Memory with stock FAISS takes 2,442 GPU-hours, or 9,604 with the official index configuration (nlist 4,096); its training is identical, so the whole gap is in target generation. On disk, counting the cached targets that both pipelines keep, ResMem retains about 2.2 TB against 41.8 TB for MLP Memory (18.8×), and its BM25 index alone is 16.7 GB (revised PDF, Appendix G and Table 10, p. 20).

Pipeline (GPU-hours)DatastoreIndex buildTargetsTrainingTotalvs. ResMem
ResMem—4 (BM25)6773711,0521.0×
MLP Memory, nlist 524,288 (FAISS guideline)1133181,6403712,4422.3×
MLP Memory, nlist 4,096 (official code)113409,0803719,6049.1×

All numbers are A100-80GB GPU-hours. Both pipelines kept all eight GPUs of the node busy, at 99% average utilization by nvidia-smi, so each wall-clock hour counts as eight GPU-hours; the 0.5-hour BM25 indexing runs on CPU and is charged at the same rate. The two MLP Memory rows differ only in the number of IVF clusters: 4,096 is the setting in MLP Memory’s official code and the one behind its Table 1 results, and 524,288 follows FAISS’s guideline for a datastore of this size. Both use stock FAISS IVF-PQ with the official index settings (64 PQ sub-quantizers of 8 bits, nprobe 32, k=1,024k = 1{,}024). A query scans only the 32 lists closest to it, so with 4,096 lists it reads about 38M of the 4.83B entries, which is why target generation alone takes 9,080 GPU-hours. FAISS recommends between 4N4\sqrt{N} and 16N16\sqrt{N} lists, and 219=524,2882^{19} = 524{,}288 sits in the middle of that range, so a query reads about 295K entries.

Q3. What are the QA results for the paired kNN-trained models in Figure 4b?

Thank you for the question. The table below gives the means over two seeds, with 95% question-level bootstrap intervals for the differences and the base measured on the same GPUs, and we have added it to Section 5.4 (revised PDF, Table 3, p. 10).

TeacherMemoryEM (%)F1 (%)Substring (%)
—Base25.8832.8127.73
kNNStandalone25.8933.1428.12
Residual26.4934.4329.58
Difference+0.60 [+0.44, +0.77]+1.30 [+1.13, +1.47]+1.47 [+1.26, +1.67]

We retrained the pair with the Figure 4b recipe, keeping the kNN targets, initialization and data order the same so that only the objective and readout differ. Both memories were selected on the same coefficient grid, which we extended below 0.1 because a residual fitted to near-one-hot kNN targets has a larger scale. Averaged over two seeds, the residual memory is better by +0.60 EM, +1.30 F1 and +1.47 substring accuracy, and each seed alone is positive on all three. In the first seed the two memories repair a similar number of base errors (615 residual, 670 standalone), but the standalone memory breaks 717 base-correct answers against 344, which puts its net EM below the base.

The residual objective thus helps under both teachers. For the teacher, both quality and cost support BM25-RAG. Because it is the base model itself reading passages, its targets improve with the base: on medicine the gap between ResMem and MLP Memory grows from 1.7 and 3.4 points at Gemma-3 4B to 3.8 and 4.3 points at 27B, and full-recipe ResMem repairs 704 base errors against 633 for MLP Memory (Section 5.3). It also builds its targets without a datastore, at about a third of the kNN teacher’s cost (681 against 2,071 GPU-hours on full Wikipedia, W2).

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer 4hRW,

Thank you again for your time and for the careful review of our paper. As the discussion period closes on November 18, we wanted to kindly check whether our response has addressed your concerns. In particular, we responded to the comparison with LoRA under the same teacher, loss and training budget (W1, Q1), the cost of building teacher targets (681 against 2,071 GPU-hours; W2, Q2), and the QA results of the paired kNN-trained models, now in Table 3 of the revised PDF (Q3). In the revised PDF, the passages related to your comments are marked in brown with margin tags.

If any point is still unclear, or if further analysis would help, we would be glad to provide it while the discussion is open. If our response has resolved your concerns, we would be grateful if you would consider updating your assessment.

Best regards, The authors

Reviewer_8GyROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed an improvement over the previous MLP memory method, as explained in their explanation: MLP Memory fits a memory alone to kNN targets; the base enters only at inference (b) ResMem fits a logit residual so that base plus residual matches an evidence-reading teacher; inference drops the teacher

Critical strengths

The explanation and comparison with previous method is very clear. And the performance benefits are clearly outlined.

Critical weaknesses

The compute cost for this method isn't mentioned, I'm interested in comparison with the older methods. The improvement of the storage is also very huge, but in practice, it may not be so significant as many GPU servers are built with huge amount of storage. For ablations study, the mentioned "residual configuration leads by 0.117 points" is mild, which weaken the significance of the residual design.

Questions for authors

Can you add more results on the computing resources needed, comparing with the older methods? Can you add deeper study for the residual part? It isn't clear to me how important this is

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8GyR (1/2)

We thank the reviewer for the review and for the practical perspective on cost and storage.

How to read this response. W1, W2, … and Q1, Q2, … number the weaknesses and questions in the order of your review; where a weakness and a question raise the same point, we answer them together. G1–G5 refer to our General Response, posted as a separate comment to all reviewers, which summarizes the points that several reviewers raised. Each answer below is self-contained: where it relies on a result, the table is repeated here, so there is no need to look it up elsewhere. In the revised PDF, passages changed in response to your comments are set in blue, with a margin tag naming the point they answer (e.g., 8GyR W1). A passage that answers several reviewers carries one reviewer’s colour, and its tag lists every point it answers, so all changes for you can be found by the tag “8GyR”.

W1, Q1. The compute cost of the method is not reported. Can you add results on the computing resources needed, compared with older methods?

Thank you for the suggestion. We now give an end-to-end breakdown on full English Wikipedia, with both pipelines run at full scale with Mistral-7B on one 8×A100-80GB node.

Pipeline (GPU-hours)DatastoreIndex buildTargetsTrainingTotalvs. ResMem
ResMem—4 (BM25)6773711,0521.0×
MLP Memory, nlist 524,288 (FAISS guideline)1133181,6403712,4422.3×
MLP Memory, nlist 4,096 (official code)113409,0803719,6049.1×

All numbers are A100-80GB GPU-hours. Both pipelines kept all eight GPUs of the node busy, at 99% average utilization by nvidia-smi, so each wall-clock hour counts as eight GPU-hours; the 0.5-hour BM25 indexing runs on CPU and is charged at the same rate. The two MLP Memory rows differ only in the number of IVF clusters: 4,096 is the setting in MLP Memory’s official code and the one behind its Table 1 results, and 524,288 follows FAISS’s guideline for a datastore of this size. Both use stock FAISS IVF-PQ with the official index settings (64 PQ sub-quantizers of 8 bits, nprobe 32, k=1,024k = 1{,}024). A query scans only the 32 lists closest to it, so with 4,096 lists it reads about 38M of the 4.83B entries, which is why target generation alone takes 9,080 GPU-hours. FAISS recommends between 4N4\sqrt{N} and 16N16\sqrt{N} lists, and 219=524,2882^{19} = 524{,}288 sits in the middle of that range, so a query reads about 295K entries.

ResMem takes 1,052 A100 GPU-hours, against 2,442 (2.3×) for MLP Memory with stock FAISS and 9,604 (9.1×) with the official index configuration (nlist 4,096). Training costs the same for both. The difference is in building the training targets, where ResMem needs one BM25 lookup and one forward pass per span instead of a nearest-neighbour search for each of 4.8B positions. At inference ResMem adds 8.6% latency, against 63.5% for Memory Decoder.

W2. The storage improvement may matter less in practice, since many GPU servers have large storage.

Thank you for the practical perspective. We see the saving less as a question of disk capacity than of what the pipeline has to build and move. The kNN route has to write a 39.6 TB datastore, more than twice the 15.6 TB local disk of our 8×A100 server, and then search it once for every training position; at full Wikipedia scale we had to attach a separate 40 TB volume to run the baseline, and building and searching the datastore accounts for most of its extra cost (W1). Counting the cached targets that both routes keep, ResMem retains about 2.2 TB in total against 41.8 TB (18.8×); its BM25 index is only 16.7 GB and fits in host memory, and the ratio does not shrink as the corpus grows (G4).

AuthorsOpenReview ↗

Response to Reviewer 8GyR (2/2)

W3, Q2. The 0.117-point lead of the residual configuration in the ablation is mild. Can you add a deeper study of the residual part?

Thank you for asking for a deeper study, which we now provide. The 0.117 F1 is the smallest effect we measured: it comes from a teacher that stays close to the base, which leaves a standalone memory little room to do damage. When we hold a sharper kNN teacher fixed and change only the objective and readout, the residual memory gains +0.60 EM and +1.30 F1 over two seeds. At the token level, the standalone memory degrades 14.2–22.5% of the gold tokens the base already predicts confidently, and the residual memory degrades none. Across medicine and law with four backbones, the residual objective wins all 58 teacher-fixed comparisons (G3).

TeacherMemoryEM (%)F1 (%)Substring (%)
BM25-RAGBase25.9132.8327.73
Standalone25.8733.1228.13
Residual26.0333.2428.49
Difference+0.15 [+0.01, +0.29]+0.12 [−0.02, +0.26]+0.37 [+0.21, +0.52]
kNNBase25.8832.8127.73
Standalone25.8933.1428.12
Residual26.4934.4329.58
Difference+0.60 [+0.44, +0.77]+1.30 [+1.13, +1.47]+1.47 [+1.26, +1.67]

The BM25-RAG rows are the eight paired seeds of Table 2, the kNN rows average two retrained seeds of the Figure 4b pair, and the brackets give 95% question-level bootstrap intervals. The two teacher blocks were trained with different budgets and scored on different GPUs, so each has its own base row and is read on its own; within each, the residual objective comes out ahead. For the teacher, both quality and cost support BM25-RAG. Because it is the base model itself reading passages, its targets improve with the base: on medicine the gap between ResMem and MLP Memory grows from 1.7 and 3.4 points at Gemma-3 4B to 3.8 and 4.3 points at 27B (G3), and full-recipe ResMem repairs 704 base errors against 633 for MLP Memory (Section 5.3). It also builds its targets without a datastore, at about a third of the kNN teacher’s cost (681 against 2,071 GPU-hours on full Wikipedia, G4).

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer 8GyR,

Thank you again for your time and for the helpful review of our paper. As the discussion period closes on November 18, we wanted to kindly check whether our response has addressed your concerns. In particular, we reported the full compute cost in A100 GPU-hours (1,052 for ResMem against 2,442 for MLP Memory; W1, Q1), compared total storage (2.2 TB against 41.8 TB; W2), and added a deeper study of the residual component (W3, Q2). In the revised PDF, the passages related to your comments are marked in blue with margin tags.

If any point is still unclear, or if further analysis would help, we would be glad to provide it while the discussion is open. If our response has resolved your concerns, we would be grateful if you would consider updating your assessment.

Best regards, The authors

Reviewer_F7WsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ResMem, a simple yet interesting approach to incorporating retrieved knowledge into a model’s parametric memory, reducing the need for online retrieval at inference time. I find the idea of learning residual logits on top of a frozen language model appealing, as it allows the memory module to complement the model’s existing knowledge rather than replace its predictions. Experiments on five QA benchmarks show consistent improvements, alongside analyses of hallucination detection, inference cost, and storage efficiency. The work explores a practical approach to reducing reliance on online retrieval while retaining the base model’s existing knowledge.

Critical strengths
  1. A well-motivated residual formulation. Rather than training a memory module to independently predict the next-token distribution, ResMem learns an additive logit residual on top of a frozen language model. This formulation provides a reasonable way to incorporate retrieved knowledge while reducing the risk of overwriting correct predictions from the base model.
  2. Comprehensive evaluation beyond QA accuracy. The authors evaluate ResMem on five open-domain QA benchmarks and compare it with several memory and adaptation baselines. They also investigate answer corrections and harms, hallucination detection, inference overhead, and teacher-construction storage costs. These analyses help characterize both the effectiveness and practical trade-offs of the proposed approach.
Critical weaknesses

The evaluation is limited in model and knowledge settings. The main experiments use Mistral-7B-v0.3 with an English Wikipedia-based knowledge source. It remains unclear whether the observed benefits transfer to other base models and domains, or how the method handles changing or conflicting knowledge. These limitations make it difficult to assess the generality of the approach beyond the evaluated setting.

Questions for authors
  1. Contribution of the residual objective. In the controlled ablation, ResMem achieves only a small F1 improvement over independent memory learning, with a 95% confidence interval crossing zero. Could the authors clarify whether the main benefit is improved overall accuracy or better preservation of correct base-model predictions?
  2. Sensitivity to the residual scaling coefficient. The inference-time coefficient gamma is selected on the validation set. How sensitive are the results to this choice, and does the optimal value transfer across datasets? This would help clarify whether the method provides robust gains or requires careful task-specific tuning.
  3. Teacher construction and storage efficiency. The paper reports substantial storage savings when constructing teacher targets from BM25-retrieved passages rather than a kNN datastore. How do these savings compare when accounting for the additional parameters and inference memory of ResMem, and how sensitive is teacher quality to the retrieval configuration?
  4. Generalization to other models and knowledge sources. The main experiments use Mistral-7B-v0.3 and English Wikipedia. Do the authors have evidence that ResMem generalizes to other base models or knowledge domains? In particular, would the residual formulation remain effective when retrieved evidence conflicts with the base model's prior knowledge?
6 comments
AuthorsOpenReview ↗

Response to Reviewer F7Ws (1/3)

We thank the reviewer for the positive assessment and for the clear and constructive questions.

How to read this response. W1, W2, … and Q1, Q2, … number the weaknesses and questions in the order of your review; where a weakness and a question raise the same point, we answer them together. G1–G5 refer to our General Response, posted as a separate comment to all reviewers, which summarizes the points that several reviewers raised. Each answer below is self-contained: where it relies on a result, the table is repeated here, so there is no need to look it up elsewhere. In the revised PDF, passages changed in response to your comments are set in teal, with a margin tag naming the point they answer (e.g., F7Ws Q3). A passage that answers several reviewers carries one reviewer’s colour, and its tag lists every point it answers, so all changes for you can be found by the tag “F7Ws”.

W1, Q4. Does ResMem generalize to other base models and knowledge domains, and how does it handle changing or conflicting knowledge?

Thank you for the question. ResMem carries over to every backbone and domain we tested (G3). The table below gives the change in reference-answer perplexity relative to each base model, averaged over three seeds for Gemma-3 4B and Mistral-7B and two seeds at 12B and 27B (one for 27B law). On medicine (memories trained on MedRAG textbooks) and law (the case-law passages of Bar Exam QA) with Gemma-3 4B, 12B and 27B and Mistral-7B, ResMem lowers answer perplexity by 1.3–4.3% on every backbone while MLP Memory stays near the base. With either teacher held fixed, the residual objective beats the standalone one in all 58 comparisons across backbones, datasets and seeds, and the gain carries over to freely generated answers. On MedQA with Mistral-7B, where each model answers without seeing the options and is scored against the gold option text, ResMem reaches 6.52 EM / 14.77 F1 over three seeds, against 5.94 / 13.55 for MLP Memory and 6.21 / 14.00 for the base (+1.23 F1 [+0.60, +1.89] over MLP Memory); on Bar Exam the F1 difference is +2.87 [+0.98, +4.78].

BackboneResMem PPL change vs. Base (%): MedQA / MMLU-Med / Bar ExamMLP Memory PPL change vs. Base (%): MedQA / MMLU-Med / Bar Exam
Gemma-3 4B−1.83 / −3.48 / −1.33−0.18 / −0.13 / +0.31
Gemma-3 12B−3.03 / −3.91 / −1.62+0.29 / +0.03 / +0.88
Gemma-3 27B−3.61 / −4.33 / −1.26+0.19 / −0.06 / +1.18
Mistral-7B−2.31 / −2.87 / −2.16−0.59 / −0.35 / −0.11

For conflicting evidence, we looked at the questions where the retrieved passages mislead the base. On 4,339 test questions the base answers correctly on its own but incorrectly after reading the top-5 BM25 passages. On these, the residual memory breaks 4.6% of the answers and the standalone memory trained on the same teacher breaks 6.5% (−1.91 points, [−2.39, −1.43]). On the 4,006 questions where the passages fix a base error, the two memories repair a similar share (3.3% and 3.7%). The residual memory is therefore more conservative where the passages pull the base away from a correct answer.

Changing knowledge is also cheap to absorb: a new corpus needs a new BM25 index (0.5 h on full Wikipedia), a new teacher pass and memory retraining, but no new datastore (G4).

AuthorsOpenReview ↗

Response to Reviewer F7Ws (2/3)

Q1. Is the main benefit improved overall accuracy or better preservation of correct base-model predictions?

Thank you for this question, which helped us state the contribution more precisely. The direct effect is preservation, and it becomes an accuracy gain whenever the standalone memory would otherwise overwrite correct answers (G1). On gold tokens the base predicts with probability at least 0.9, the residual memory never raises NLL by more than 0.1 nats, and it keeps 96.11% of base-correct answers against 94.05% for the standalone memory (Table 2). The size of the accuracy gain depends on the teacher. With the kNN teacher the residual objective gains +0.60 EM and +1.30 F1 over two seeds, while the evidence-reading teacher stays close to the base and leaves the standalone memory less to overwrite.

TeacherMemoryEM (%)F1 (%)Substring (%)
BM25-RAGBase25.9132.8327.73
Standalone25.8733.1228.13
Residual26.0333.2428.49
Difference+0.15 [+0.01, +0.29]+0.12 [−0.02, +0.26]+0.37 [+0.21, +0.52]
kNNBase25.8832.8127.73
Standalone25.8933.1428.12
Residual26.4934.4329.58
Difference+0.60 [+0.44, +0.77]+1.30 [+1.13, +1.47]+1.47 [+1.26, +1.67]

The BM25-RAG rows are the eight paired seeds of Table 2, the kNN rows average two retrained seeds of the Figure 4b pair, and the brackets give 95% question-level bootstrap intervals. The two teacher blocks were trained with different budgets and scored on different GPUs, so each has its own base row and is read on its own; within each, the residual objective comes out ahead. For the teacher, both quality and cost support BM25-RAG. Because it is the base model itself reading passages, its targets improve with the base: on medicine the gap between ResMem and MLP Memory grows from 1.7 and 3.4 points at Gemma-3 4B to 3.8 and 4.3 points at 27B (G3), and full-recipe ResMem repairs 704 base errors against 633 for MLP Memory (Section 5.3). It also builds its targets without a datastore, at about a third of the kNN teacher’s cost (681 against 2,071 GPU-hours on full Wikipedia, G4).

Q2. How sensitive are the results to the choice of γ\gamma, and does the optimal value transfer across datasets?

Thank you for the question. One value transfers across all five datasets (G5). With γ=0.3\gamma = 0.3 everywhere, ResMem’s net EM gain over the base is positive on every dataset, from +0.05 on WebQ to +1.24 on HotpotQA. Every value from 0.2 to 0.4 beats MLP Memory at its best λ\lambda. In the appendix’s deployment run, which fixes γ=0.4\gamma = 0.4 and λ=0.25\lambda = 0.25 on all datasets, ResMem reaches 34.77 F1 against 33.68 for MLP Memory (base 32.79 in that run), and we have added the per-dataset EM–γ\gamma curves to Appendix H (revised PDF, p. 21). The useful range depends mostly on the teacher: a residual fitted to near-one-hot kNN targets has a larger scale and works best around γ=0.05\gamma = 0.05.

AuthorsOpenReview ↗

Response to Reviewer F7Ws (3/3)

Q3. How do the storage savings compare when accounting for the additional parameters and inference memory of ResMem, and how sensitive is teacher quality to the retrieval configuration?

We appreciate the request to count the memory itself. With the memory counted, ResMem still saves on both construction and inference (G4). Its memory has the same 1.54B parameters as MLP Memory’s; at inference it adds 2.875 GiB of GPU memory and 8.6% latency and needs no index or datastore. For construction on full Wikipedia, counting the cached targets that both routes keep, ResMem retains about 2.2 TB against 41.8 TB for MLP Memory (18.8×), of which its BM25 index is only 16.7 GB. Its pipeline also takes 1,052 GPU-hours end to end against 2,442 on the same node.

Pipeline (GPU-hours)DatastoreIndex buildTargetsTrainingTotalvs. ResMem
ResMem—4 (BM25)6773711,0521.0×
MLP Memory, nlist 524,288 (FAISS guideline)1133181,6403712,4422.3×
MLP Memory, nlist 4,096 (official code)113409,0803719,6049.1×

All numbers are A100-80GB GPU-hours. Both pipelines kept all eight GPUs of the node busy, at 99% average utilization by nvidia-smi, so each wall-clock hour counts as eight GPU-hours; the 0.5-hour BM25 indexing runs on CPU and is charged at the same rate. The two MLP Memory rows differ only in the number of IVF clusters: 4,096 is the setting in MLP Memory’s official code and the one behind its Table 1 results, and 524,288 follows FAISS’s guideline for a datastore of this size. Both use stock FAISS IVF-PQ with the official index settings (64 PQ sub-quantizers of 8 bits, nprobe 32, k=1,024k = 1{,}024). A query scans only the 32 lists closest to it, so with 4,096 lists it reads about 38M of the 4.83B entries, which is why target generation alone takes 9,080 GPU-hours. FAISS recommends between 4N4\sqrt{N} and 16N16\sqrt{N} lists, and 219=524,2882^{19} = 524{,}288 sits in the middle of that range, so a query reads about 295K entries.

For the retrieval configuration, we varied the number of retrieved passages. The teacher’s gold-token NLL is 1.865 with the top-1 passage, 1.832 with the top-3 and 1.830 with the top-5 (base 1.897), so teacher quality levels off after about three passages. The top-5 teacher is the teacher of Table 2; over its eight paired seeds the residual memory leads by +0.15 EM [+0.01, +0.29] and +0.37 substring accuracy [+0.21, +0.52], with lower reference-answer NLL in every seed. We also trained both memories on the weaker top-1 teacher with two seeds. There, no tested strength of the standalone memory improves development accuracy, so development selection turns it off in both seeds, while the residual memory still lowers answer NLL relative to the base (−0.022 and −0.020). We have added these results to Appendix I (revised PDF, p. 21).

Reviewer_jNquOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ResMem, a method for training a parametric memory module that adds a logit residual to a frozen base language model's predictions. The key innovations are: (1) training the memory through the joint base-plus-memory prediction rather than as a standalone component, and (2) replacing the kNN datastore used in prior work (MLP Memory) with BM25 retrieval and an evidence-reading teacher, cutting construction storage by 94.7% (18.8×). The method is evaluated on five knowledge-QA datasets, HaluEval hallucination detection, and a shared-teacher ablation isolating objective and readout effects

Critical strengths
  1. The distinction between fitting a memory alone (MLP Memory) versus fitting it through the joint prediction (ResMem) is well articulated. The gradient analysis in Section 3.2 makes precise why positions where the base already matches the target produce no update, and Eq. 2 characterizes the optimum as a log-ratio correction.

  2. Eq. 7 and the measurements in Section 5.2 provide a clean accounting of retained storage for both routes. The 94.7% reduction is substantial, and the decomposition showing that the kNN datastore alone takes ~21× the bytes of cached targets is illuminating. The observation that the ratio is independent of corpus size (depends only on d, K, P, and b) strengthens the claim.

  3. Figure 3 and the accompanying analysis (704 corrections vs. 381 harms for ResMem, against 633/482 for MLP Memory) go beyond aggregate EM/F1 and reveal where the gains come from. The finding that most of ResMem's advantage comes from avoided harms (101 fewer broken answers) rather than extra repairs is insightful and consistent with the theoretical analysis of the additive vs. mixture readouts in Section 3.3.

  4. Holding the teacher, architecture, data, and optimizer fixed while varying objective and readout (four configurations) is the right experiment to isolate the method's contribution. The finding that the residual configuration preserves 96.11% of base-correct answers (vs. 94.05% for standalone+mixture) supports the central claim, even if the F1 difference (0.117 points) is modest and the confidence interval includes zero.

Critical weaknesses
  1. Table 1 shows ResMem at 26.74 EM / 34.48 F1 vs. MLP Memory at 26.11 / 33.85 — a gain of ~0.6 EM and ~0.6 F1 points. However, this comparison changes three things at once: the teacher (BM25-RAG vs. kNN), the objective (residual vs. standalone), and the readout (additive vs. mixture). The shared-teacher ablation (Section 5.4) isolates objective and readout and finds only a 0.117 F1 advantage for the residual configuration — with a 95% interval of [−0.020, 0.258] that includes zero. The paper acknowledges this but still attributes the full-recipe gain primarily to the method. The teacher change (evidence-reading vs. kNN) appears to be doing most of the work, as the conclusion itself notes ("the extra repairs come with the evidence-reading teacher"). This is a valuable finding, but it means the paper's title and framing (residual self-distillation) may overstate the objective's contribution relative to the teacher/data change.

  2. The method is essentially: (a) context distillation with a frozen teacher and student sharing a backbone, plus (b) an additive logit correction trained through the joint prediction. Both components have precedent — context distillation (Snell et al., 2022), Born-Again Networks (Furlanello et al., 2018), and the additive composition in DExperts/contrastive decoding are all cited. The paper positions ResMem against MLP Memory and Memory Decoder, but does not deeply compare to TransMem (Lei et al., 2026a), which it acknowledges is "the closest work" but dismisses in two sentences. A more thorough comparison — even a conceptual table of differences (student input, output type, readout, training signal) — would help readers assess the novelty.

  3. All experiments use Mistral-7B-v0.3 and an 8-layer, 1.54B-parameter memory. The shared-teacher ablation also compares against Memory Decoder (0.47B), finding the hidden-state memory stronger "under this training budget and at these sizes." But the paper does not test whether the residual objective's advantage persists at smaller or larger memory sizes, or with different base architectures. Given that the method's appeal is partly its simplicity, a scaling study (even a small one) would strengthen the claims.

  4. Appendix E describes a 22-feature logistic gate that decides when to apply the residual. It changes mean EM by +0.07 points (interval includes zero) and mean F1 by −0.02. This is a reasonable negative result, but it is relegated to an appendix and the main text simply says "ResMem is reported without it." Since the gate is a natural extension (and the paper includes it as a table), a brief discussion in the main text of why it fails — e.g., the features are insufficient, or the residual's benefits and harms are not separable at the token level — would be informative. The finding that the gate accepts 10,614 changes and blocks 8,640 (of 19,254 total) suggests it is doing something, but not enough to improve aggregate metrics.

  5. Table 1 shows ResMem has lower PPL than MLP Memory on every dataset, which is consistent with the F1 gains. However, Appendix A reports that on held-out Wikipedia text, MLP Memory achieves lower perplexity (7.441) than ResMem (8.660). The paper notes this ("A closer fit to the corpus thus does not by itself give better final answers") but does not reconcile it with the QA PPL results. Is the QA PPL measured on reference answers (which would favor the method that better matches those answers) rather than on generated text? If so, the PPL comparison in Table 1 may be less informative than it appears. Clarifying what PPL is measured on (reference answers vs. model-generated continuations) would help.

  6. Section 5.3 uses one global coefficient (γ*=0.3 for ResMem) chosen on development data, while Section 5.1/Table 1 selects per-dataset coefficients to maximize EM. The correction/harm numbers (704/381) are therefore not directly comparable to the per-dataset EM gains in Table 1. This is a minor inconsistency but could confuse readers trying to map the analysis to the main results.

Questions for authors

On the shared-teacher ablation: The residual configuration's F1 advantage is 0.117 points with a 95% interval of [−0.020, 0.258] (question-cluster bootstrap) or [−0.180, 0.477] (with seed resampling). Given that both intervals include zero, how confident are you that the residual objective itself (independent of the teacher) provides a reliable gain? Would a larger-scale run (more seeds, more exposure) be needed to establish this, or do you view the preservation/repair trade-off (96.11% vs. 94.05% preservation) as the more robust finding?

On the teacher's role: The full-recipe ResMem uses a BM25-RAG teacher, while MLP Memory uses a kNN teacher. The paper attributes most of the repair gains to the teacher's shape (Figure 4a). If you trained MLP Memory with a BM25-RAG teacher (i.e., the shared-teacher standalone configuration) and compared it to full-recipe ResMem, how much of the full-recipe gain would remain attributable to the residual objective? The shared-teacher ablation suggests the objective contributes little F1, but the full-recipe comparison confounds teacher and objective. A direct "same teacher, different objective, full data" comparison would be valuable.

On Table 6 and the gate: The caption says "ResMem matches Table 1" but the numbers differ slightly (e.g., NQ EM 18.98 vs. 18.98, F1 25.69 vs. 25.69 — actually identical; but mean EM 26.74 vs. 26.74). Why does the gate table report "ResMem matches Table 1" if the gate changes 190 answers? Is the gate applied at inference only for the "ResMem + gate" rows, and the "ResMem" rows are identical to Table 1? Clarifying the experimental setup would help.

On the PPL discrepancy: Table 1 shows ResMem has lower PPL than MLP Memory on QA datasets, but Appendix A reports MLP Memory has lower PPL on held-out Wikipedia text (7.441 vs. 8.660). What exactly is the QA PPL measuring — the likelihood of reference answers under the model, or the likelihood of the model's own generated answers? If it's reference-answer likelihood, does the lower PPL for ResMem simply reflect that its training target (the evidence-reading teacher) is closer to reference answers than the kNN teacher's target?

On storage accounting: Eq. 7 gives S_RAG(n) = 6Kn + bn, where b is "the text and BM25 index per position." The measured b is 72.9 bytes on the subset. Does this include the BM25 index overhead (which typically scales with vocabulary size and document count, not just positions)? If the BM25 index is built over the full corpus (16.7 GB for all of Wikipedia), how does that scale with corpus size relative to the kNN datastore? The paper says the ratio approaches 22.5 as b→0, but b is not zero in practice; a plot of the ratio vs. corpus size (or vs. b) would clarify when the advantage shrinks.

On the additive vs. mixture readout: Section 3.3 shows that the additive readout switches to a target token iff γm_T > (1−γ)m_B, while the mixture switches iff λ[Δp_T] > (1−λ)[Δp_B]. The analysis assumes the memory is at its standalone optimum (q_M = p̃_T) for the mixture, but in practice MLP Memory's memory is trained standalone and may not reach that optimum. How sensitive is the harm analysis to this assumption? Would a memory trained with the standalone objective but evaluated with the additive readout (the "crossed" configuration in Section 5.4) show the same harm pattern?

On architecture: The paper uses an 8-layer, 1.54B-parameter memory. Memory Decoder uses a 0.47B-parameter decoder and performs worse in the shared-teacher ablation. Is the residual objective's advantage (if any) dependent on the memory being large enough to represent the log-ratio correction? A smaller memory (e.g., 4 layers) trained with the residual objective might show a larger or smaller gap vs. standalone — this would inform whether the method's benefits come from the objective or from the memory's capacity.

On generalization: All QA experiments use Mistral-7B. Would the residual objective work as well with a base model that has different calibration properties (e.g., Llama-3, Qwen)? The method's appeal is partly its simplicity and model-agnosticism, but the paper does not test this. Even a single additional base model (e.g., Llama-3-8B) on a subset of the QA tasks would strengthen the generality claim.

On the "residual self-distillation" term: The method trains a residual on a frozen base using a teacher that shares the base's backbone but sees retrieved evidence. This is close to context distillation (Snell et al., 2022) with an additive output parameterization. What distinguishes "residual self-distillation" from "context distillation with a logit-residual memory"? Is the novelty in the objective (joint prediction), the readout (additive), or the combination? A clearer positioning against context distillation and TransMem would help readers assess the contribution.

On the gate's failure: The gate accepts 10,614 of 19,254 residual-induced top-token changes and blocks 8,640. Of the accepted changes, 7,493 occur at the first answer token. Does this mean the gate is essentially learning to apply the residual at the first token and suppress it later? If so, would a simpler rule (e.g., apply residual only at the first token) achieve similar results? Understanding why the gate fails to improve aggregate metrics would be informative for future work on when to trust a parametric memory.

7 comments
AuthorsOpenReview ↗

Response to Reviewer jNqu (1/4)

We sincerely thank the reviewer for the detailed and careful review. The questions are precise, and several of them led directly to new experiments.

How to read this response. W1, W2, … and Q1, Q2, … number the weaknesses and questions in the order of your review; where a weakness and a question raise the same point, we answer them together. G1–G5 refer to our General Response, posted as a separate comment to all reviewers, which summarizes the points that several reviewers raised. Each answer below is self-contained: where it relies on a result, the table is repeated here, so there is no need to look it up elsewhere. In the revised PDF, passages changed in response to your comments are set in brick red, with a margin tag naming the point they answer (e.g., jNqu Q4). A passage that answers several reviewers carries one reviewer’s colour, and its tag lists every point it answers, so all changes for you can be found by the tag “jNqu”.

W1, Q1, Q2. The full-recipe comparison changes the teacher, objective and readout at once, and the shared-teacher F1 gain (0.117) has an interval that includes zero. How much of the gain belongs to the residual objective, and how reliable is it?

Thank you for this careful reading of Table 2. We see the teacher and the objective as the two halves of the method rather than as a confound: “self-distillation” names the teacher, the base model itself reading retrieved evidence, and “residual” names the objective. The teacher determines which corrections are available, and the objective determines whether they can be added without breaking what the base already answers correctly. The question is then whether the objective matters once the teacher is fixed, and the new teacher-fixed experiments show that it does (G1).

With the kNN teacher held fixed, changing only the objective and readout adds +0.60 EM and +1.30 F1 over two seeds, and the interval excludes zero in each seed separately. The BM25-RAG standalone row is the configuration you describe, MLP Memory trained on our teacher. With this teacher the F1 gain is smaller because the teacher stays close to the base and leaves a standalone memory little to overwrite; even so, the residual memory has lower reference-answer NLL in 8 of 8 seeds and +0.37 substring accuracy [+0.21, +0.52]. Its answers are slightly shorter than the standalone memory’s (1.46 vs. 1.51 normalized words), so the substring gain does not come from longer answers. The mechanism is the one in Section 3.2: of the gold tokens the base already predicts with probability at least 0.9, the standalone memory raises the NLL of 14.2–22.5% and the residual memory of none. We have stated this division of roles explicitly in Section 5.4 (revised PDF, p. 9).

TeacherMemoryEM (%)F1 (%)Substring (%)
BM25-RAGBase25.9132.8327.73
Standalone25.8733.1228.13
Residual26.0333.2428.49
Difference+0.15 [+0.01, +0.29]+0.12 [−0.02, +0.26]+0.37 [+0.21, +0.52]
kNNBase25.8832.8127.73
Standalone25.8933.1428.12
Residual26.4934.4329.58
Difference+0.60 [+0.44, +0.77]+1.30 [+1.13, +1.47]+1.47 [+1.26, +1.67]

The BM25-RAG rows are the eight paired seeds of Table 2, the kNN rows average two retrained seeds of the Figure 4b pair, and the brackets give 95% question-level bootstrap intervals. The two teacher blocks were trained with different budgets and scored on different GPUs, so each has its own base row and is read on its own; within each, the residual objective comes out ahead. For the teacher, both quality and cost support BM25-RAG. Because it is the base model itself reading passages, its targets improve with the base: on medicine the gap between ResMem and MLP Memory grows from 1.7 and 3.4 points at Gemma-3 4B to 3.8 and 4.3 points at 27B (G3), and full-recipe ResMem repairs 704 base errors against 633 for MLP Memory (Section 5.3). It also builds its targets without a datastore, at about a third of the kNN teacher’s cost (681 against 2,071 GPU-hours on full Wikipedia, G4).

AuthorsOpenReview ↗

Response to Reviewer jNqu (2/4)

W2, Q9. The method combines context distillation with an additive logit correction, both of which have precedent. What distinguishes it from context distillation and from TransMem?

Thank you for the question. The novelty is in the combination: a separate memory is trained through the same additive readout it is used with, so that the base and the residual together match the teacher. Each precedent has one half of this. Context distillation and Born-Again Networks train the student’s own weights, so the teacher’s knowledge ends up in the backbone. DExperts and contrastive decoding combine logits additively, but their component models are trained on their own and combined only at inference. The combination is what protects the base: distilling the same teacher into a LoRA of the same size (rank 588) repairs 486 base errors but breaks 600 base-correct answers, against 370 and 305 for ResMem, and a standalone memory read through the additive readout is switched off by development selection (G2).

TransMem follows a similar idea and trains a parametric module through the frozen model to match an evidence-conditioned teacher. However, its parameters learn how to use the external text rather than the knowledge in it, so the text must still be supplied at inference, whereas ResMem’s memory holds the distilled knowledge and needs no external knowledge store or retriever. The comparison table you suggested is below, and we have revised the positioning in Section 2 accordingly (revised PDF, p. 3).

Teacher readsStudent readsTraining dataWhat is learnedEdits backboneExternal knowledge at inference
Context distillationinstructions or examples + inputinputtask promptsprompted behaviour, in the backboneYesNo
LoRAevidence + inputinputunlabeled corpuscorpus knowledge, in the backboneYesNo
TransMemevidence + inputlong context with the evidence + inputQA with annotated evidencea skill for finding evidence in the contextNoYes (long context)
ResMem (Ours)evidence + inputinputunlabeled corpuscorpus knowledge, in a separate logit residualNoNo
AuthorsOpenReview ↗

Response to Reviewer jNqu (3/4)

W3, Q7, Q8. Does the residual objective’s advantage persist at other memory sizes and with other base models?

Thank you for the suggestion; we tested both. A two-layer, 0.49B memory (4 seeds) keeps the advantage over a standalone memory of the same size, and the gap is larger than at eight layers: +0.60 substring accuracy against +0.37, and +0.20 F1 against +0.12 (G1). A residual only needs to encode what the base is missing, which takes less capacity than reproducing the whole next-token distribution.

Memory sizeMemoryEM (%)F1 (%)Substring (%)
—Base25.9132.8327.73
Eight layers, 1.54B (8 seeds)Standalone25.8733.1228.13
Residual26.0333.2428.49
Difference+0.15 [+0.01, +0.29]+0.12 [−0.02, +0.26]+0.37 [+0.21, +0.52]
Two layers, 0.49B (4 seeds)Standalone26.0133.2428.20
Residual26.1033.4328.80
Difference+0.09 [−0.05, +0.24]+0.20 [+0.06, +0.34]+0.60 [+0.46, +0.75]

Both two-layer memories score slightly higher than their eight-layer counterparts. All runs share the Table 2 setup, with the same BM25-RAG teacher and 8.4M position exposures, and at this small training budget the smaller memory is easier to fit, which helps the residual most. The 1.54B size of the main experiments matches MLP Memory’s memory for a like-for-like comparison, and with the full training budget it reaches 34.48 F1 (Table 1). We have added this table and discussion to Appendix F (revised PDF, Table 9, pp. 18–19).

For other base models, we chose Gemma-3, a current open family with its own tokenizer, at three sizes. We repeated the four-way comparison on medicine and law with Gemma-3 4B, 12B and 27B and with Mistral-7B (G3); the table below gives the change in reference-answer perplexity relative to each base model, averaged over three seeds for Gemma-3 4B and Mistral-7B and two seeds at 12B and 27B (one for 27B law). ResMem lowers answer perplexity by 1.3–4.3% on every backbone while MLP Memory stays near the base, and with either teacher held fixed, the residual objective beats the standalone one in all 58 comparisons across backbones, datasets and seeds. The largest domain gains come with Gemma-3 27B and its 4.2B memory, and on medicine the gap to MLP Memory grows with model size.

BackboneResMem PPL change vs. Base (%): MedQA / MMLU-Med / Bar ExamMLP Memory PPL change vs. Base (%): MedQA / MMLU-Med / Bar Exam
Gemma-3 4B−1.83 / −3.48 / −1.33−0.18 / −0.13 / +0.31
Gemma-3 12B−3.03 / −3.91 / −1.62+0.29 / +0.03 / +0.88
Gemma-3 27B−3.61 / −4.33 / −1.26+0.19 / −0.06 / +1.18
Mistral-7B−2.31 / −2.87 / −2.16−0.59 / −0.35 / −0.11

W4, Q10. Why does the gate not improve the aggregate metrics, and is it essentially learning to apply the residual at the first token?

Thank you for this careful reading of the gate counts. We read the gate result as a property of the residual: blocking 8,640 of its 19,254 top-token changes leaves EM and F1 essentially unchanged (+0.07 EM, −0.02 F1), so the residual’s changes contain no harmful subset that these 22 features can separate. This is consistent with Section 3.2, where training through the combined prediction already leaves the positions the base gets right untouched.

On the first token, largely yes: of the 10,614 changes the gate accepts on the test set, 7,493 are at the first answer token. Applying the residual only at that token gives +0.24 EM [−0.01, +0.49] and +0.25 F1 [+0.03, +0.47] over applying it at every step, more than the learned gate’s +0.07 EM. The same rule lowers substring accuracy by 0.77 points, so the residual at later tokens still helps the model complete the answer. All results we report therefore apply the residual at every step, and the first-token rule is a cheaper decoding option, since the memory then runs once per answer. We have added this discussion and the comparison to Appendix E next to the gate ablation (revised PDF, p. 18).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the four reviewers for their careful reading and constructive feedback. Their questions led us to run a number of new experiments, and we believe the paper is clearer and stronger for them. This overview summarizes how we addressed the points raised in more than one review (G1–G5). Each reviewer’s reply gives the full results and tables for that reviewer’s questions, and every change is in the revised PDF. Intervals are 95% question-level bootstrap CIs over each benchmark’s test questions (38,627 for the five QA datasets).

G1. Does the residual objective matter once the teacher is fixed? (jNqu, F7Ws, 8GyR, 4hRW)

Yes. With the kNN teacher held fixed, changing only the objective and readout gains +0.60 EM and +1.30 F1, positive in each seed. The 0.117 F1 of Table 2 is the smallest effect we measured, because that teacher stays close to the base; even there the residual memory has lower reference-answer NLL in 8 of 8 seeds. The gain comes from preservation, as Section 3.2 predicts: on gold tokens the base already predicts with probability at least 0.9, standalone memories degrade 14.2–22.5% and residual memories none. The advantage also holds for a two-layer, 0.49B memory and in all 58 teacher-fixed comparisons on medicine and law.

G2. Why fit a residual instead of distilling directly, e.g. into LoRA? (4hRW, jNqu)

We retrained LoRA with the same teacher, loss and exposure as ResMem and matched its size (rank 588, 1.54B parameters). It repairs more base errors than ResMem (486 against 370) but breaks about twice as many base-correct answers (600 against 305), so its net EM change is negative, and ResMem leads it by +0.31 F1. What sets ResMem apart from context distillation, output-composition methods and TransMem is a separate memory trained through the same additive readout it is used with, which distills the knowledge into parameters while leaving what the base already answers correctly untouched.

G3. Other backbones, domains and memory sizes (jNqu, F7Ws)

On medicine and law with Gemma-3 4B, 12B and 27B and with Mistral-7B, ResMem lowers reference-answer perplexity by 1.3–4.3% on every backbone while MLP Memory stays near the base. The gap grows with model size and carries over to freely generated answers (+1.23 F1 over MLP Memory on MedQA). A two-layer, 0.49B memory keeps the residual advantage with one third of the parameters.

G4. Compute, storage and inference cost (8GyR, 4hRW, F7Ws, jNqu)

End to end on full English Wikipedia with Mistral-7B on one 8×A100 node, ResMem takes 1,052 A100 GPU-hours, against 2,442 (2.3×) for MLP Memory with stock FAISS and 9,604 (9.1×) with its official index configuration, which is the one behind its Table 1 results. Training is identical, so the whole gap is in target generation. Counting the cached targets that both routes keep, ResMem retains about 2.2 TB on full Wikipedia against 41.8 TB for MLP Memory (18.8×), of which its BM25 index is only 16.7 GB. At inference it adds 8.6% latency, against 63.5% for Memory Decoder.

G5. Sensitivity to γ\gamma (F7Ws, jNqu)

A single γ=0.3\gamma = 0.3 gives a positive net EM gain over the base on all five datasets, from +0.05 on WebQ to +1.24 on HotpotQA, and every value from 0.2 to 0.4 beats MLP Memory at its best λ\lambda.

Changes to the paper (all made in the revised PDF).

  • Section 5.4: the kNN-teacher pair table and the token-level preservation result (G1; pp. 9–10).
  • Section 5.4: the same-teacher LoRA comparison (G2; p. 9, details in Appendix A, p. 15), and the reference-answer PPL definition in the caption of Table 1 (p. 7).
  • Section 2: a sharper positioning against context distillation and TransMem (G2; p. 3).
  • Appendix: the backbone and domain results with generated answers and the memory-size comparison (G3, G1; Appendix F, pp. 18–19), the cost breakdown and storage plot (G4; Appendix G, p. 20), the γ\gamma curves (G5; Appendix H, p. 21), and the gate discussion with the first-token rule (jNqu; Appendix E, pp. 17–18).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.