- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ResMem, which trains an MLP memory to adjust a frozen LLM’s logits. The same LLM reads retrieved passages to provide offline teacher targets, and the memory is trained with distillation, while inference uses only the base model and memory. The main changes from MLP Memory are the teacher and the residual training objective. Across five QA datasets, average F1 increases from 32.83 to 34.48. The paper also measures construction storage, inference costs, and which answers are helped or hurt.
Critical strengths
-
The full method improves EM and F1 over the base on all five QA datasets. It also improves on MLP Memory overall at essentially the same inference cost. Avoiding the large kNN datastore is another useful practical benefit.
-
The correction/harm analysis is informative. Separating newly corrected answers from previously correct answers that become wrong helps explain the results.
Critical weaknesses
-
I think the benefit of the residual formulation over simpler distillation is still unclear. With the same teacher, its F1 advantage over standalone training is only 0.117 points (Table 2). A natural alternative is to distill the same teacher directly into LoRA. The paper includes LoRA, but I could not tell whether it uses the same teacher, loss, and training exposure. This seems an important comparison for justifying the 1.54B-parameter memory.
-
The overall cost trade-off is not fully evaluated. The reported 94.7% saving concerns retained teacher-construction files. Both pipelines subsequently train from cached targets. Without target-generation and memory-training costs, it is difficult to tell whether the new pipeline is cheaper overall or mainly reduces disk storage. This matters because the teacher change accounts for much of the practical contribution.
Questions for authors
- Does LoRA use the same teacher, loss, and training budget? What is the motivation for fitting the residual instead of simple distillation?
- How expensive is it to construct the teacher target?
- What are the QA results for the paired kNN-trained models in Figure 4b?
5 comments
Response to Reviewer 4hRW (2/2)
W2, Q2. The 94.7% saving concerns retained teacher-construction files. How expensive is it to construct the teacher target, and is the pipeline cheaper overall or mainly in disk?
Thank you for the question. The pipeline is cheaper in compute as well as in disk (G4). On full English Wikipedia with one 8×A100 node, the BM25 index takes 0.5 hours on CPU (charged as 4 GPU-hours) and the teacher pass over all training spans 677 GPU-hours. With 371 GPU-hours of training, ResMem takes 1,052 GPU-hours end to end. MLP Memory with stock FAISS takes 2,442 GPU-hours, or 9,604 with the official index configuration (nlist 4,096); its training is identical, so the whole gap is in target generation. On disk, counting the cached targets that both pipelines keep, ResMem retains about 2.2 TB against 41.8 TB for MLP Memory (18.8×), and its BM25 index alone is 16.7 GB (revised PDF, Appendix G and Table 10, p. 20).
| Pipeline (GPU-hours) | Datastore | Index build | Targets | Training | Total | vs. ResMem |
|---|---|---|---|---|---|---|
| ResMem | — | 4 (BM25) | 677 | 371 | 1,052 | 1.0× |
| MLP Memory, nlist 524,288 (FAISS guideline) | 113 | 318 | 1,640 | 371 | 2,442 | 2.3× |
| MLP Memory, nlist 4,096 (official code) | 113 | 40 | 9,080 | 371 | 9,604 | 9.1× |
All numbers are A100-80GB GPU-hours. Both pipelines kept all eight GPUs of the node busy, at 99% average utilization by nvidia-smi, so each wall-clock hour counts as eight GPU-hours; the 0.5-hour BM25 indexing runs on CPU and is charged at the same rate. The two MLP Memory rows differ only in the number of IVF clusters: 4,096 is the setting in MLP Memory’s official code and the one behind its Table 1 results, and 524,288 follows FAISS’s guideline for a datastore of this size. Both use stock FAISS IVF-PQ with the official index settings (64 PQ sub-quantizers of 8 bits, nprobe 32, ). A query scans only the 32 lists closest to it, so with 4,096 lists it reads about 38M of the 4.83B entries, which is why target generation alone takes 9,080 GPU-hours. FAISS recommends between and lists, and sits in the middle of that range, so a query reads about 295K entries.
Q3. What are the QA results for the paired kNN-trained models in Figure 4b?
Thank you for the question. The table below gives the means over two seeds, with 95% question-level bootstrap intervals for the differences and the base measured on the same GPUs, and we have added it to Section 5.4 (revised PDF, Table 3, p. 10).
| Teacher | Memory | EM (%) | F1 (%) | Substring (%) |
|---|---|---|---|---|
| — | Base | 25.88 | 32.81 | 27.73 |
| kNN | Standalone | 25.89 | 33.14 | 28.12 |
| Residual | 26.49 | 34.43 | 29.58 | |
| Difference | +0.60 [+0.44, +0.77] | +1.30 [+1.13, +1.47] | +1.47 [+1.26, +1.67] |
We retrained the pair with the Figure 4b recipe, keeping the kNN targets, initialization and data order the same so that only the objective and readout differ. Both memories were selected on the same coefficient grid, which we extended below 0.1 because a residual fitted to near-one-hot kNN targets has a larger scale. Averaged over two seeds, the residual memory is better by +0.60 EM, +1.30 F1 and +1.47 substring accuracy, and each seed alone is positive on all three. In the first seed the two memories repair a similar number of base errors (615 residual, 670 standalone), but the standalone memory breaks 717 base-correct answers against 344, which puts its net EM below the base.
The residual objective thus helps under both teachers. For the teacher, both quality and cost support BM25-RAG. Because it is the base model itself reading passages, its targets improve with the base: on medicine the gap between ResMem and MLP Memory grows from 1.7 and 3.4 points at Gemma-3 4B to 3.8 and 4.3 points at 27B, and full-recipe ResMem repairs 704 base errors against 633 for MLP Memory (Section 5.3). It also builds its targets without a datastore, at about a third of the kNN teacher’s cost (681 against 2,071 GPU-hours on full Wikipedia, W2).
Gentle follow-up on our response
Dear Reviewer 4hRW,
Thank you again for your time and for the careful review of our paper. As the discussion period closes on November 18, we wanted to kindly check whether our response has addressed your concerns. In particular, we responded to the comparison with LoRA under the same teacher, loss and training budget (W1, Q1), the cost of building teacher targets (681 against 2,071 GPU-hours; W2, Q2), and the QA results of the paired kNN-trained models, now in Table 3 of the revised PDF (Q3). In the revised PDF, the passages related to your comments are marked in brown with margin tags.
If any point is still unclear, or if further analysis would help, we would be glad to provide it while the discussion is open. If our response has resolved your concerns, we would be grateful if you would consider updating your assessment.
Best regards, The authors
Response to Reviewer 4hRW (1/2)
We thank the reviewer for the concrete and constructive questions. Each of them led to a new experiment, and we believe the answers make the paper stronger.
How to read this response. W1, W2, … and Q1, Q2, … number the weaknesses and questions in the order of your review; where a weakness and a question raise the same point, we answer them together. G1–G5 refer to our General Response, posted as a separate comment to all reviewers, which summarizes the points that several reviewers raised. Each answer below is self-contained: where it relies on a result, the table is repeated here, so there is no need to look it up elsewhere. In the revised PDF, passages changed in response to your comments are set in brown, with a margin tag naming the point they answer (e.g., 4hRW Q3). A passage that answers several reviewers carries one reviewer’s colour, and its tag lists every point it answers, so all changes for you can be found by the tag “4hRW”.
W1, Q1. The benefit of the residual formulation over simpler distillation is unclear. Does LoRA use the same teacher, loss and training budget, and what motivates fitting a residual?
Thank you for this important question. The LoRA in Table 1 is a baseline trained without our teacher, so for a direct comparison we retrained it with the same teacher, loss and exposure as ResMem, with rank 588 so that it has the same 1.54B parameters as ResMem’s memory (G2). The dev-selected adapter scores 25.77 EM / 32.93 F1 (base 25.91 / 32.83), and ResMem reaches 26.03 / 33.24, +0.31 F1 [+0.10, +0.51] above LoRA. We swept the learning rate from 5e−6 to 1e−4 and, as for γ, scaled the adapter update by a coefficient chosen on development data; the selected adapter uses learning rate 5e−6 and scale 0.2, and at every larger learning rate development selection picks scale 0.
The last three columns count the base errors each model repairs and the base-correct answers it breaks on the test set, and they give our motivation for fitting a residual instead of distilling into the model. LoRA repairs 486 base errors but breaks 600 base-correct answers, while ResMem repairs 370 and breaks 305. Writing the teacher’s corrections into the backbone disturbs answers the base already gets right, whereas a residual trained through the combined prediction only changes the positions where the base and the teacher disagree. The 0.117 F1 over a standalone memory comes from a teacher that stays close to the base; with a sharper kNN teacher held fixed, the residual objective adds +1.30 F1 (Q3). The advantage also does not depend on the 1.54B size: at 0.49B a residual memory still beats a standalone memory of the same size (G1).