- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MeMo, a framework that trains a smaller language model to store knowledge from a document corpus and allows a larger model to access that knowledge through natural-language questions. The framework separates a Memory model, which stores the corpus knowledge, from an Executive model, which handles reasoning and produces the final answer. The Executive remains frozen.
The Memory is trained on synthetic QA pairs called reflections. The data generation pipeline includes fact extraction, consolidation of related facts, verification and rewriting for self-containment, entity-surfacing QA generation, and cross-document synthesis. At inference time, the Executive interacts with the Memory through three stages: grounding, entity identification, and answer seeking.
The paper evaluates MeMo on BrowseComp-Plus, NarrativeQA, and MuSiQue against retrieval and parametric memory baselines. Results vary across Executive models and datasets, with improvements on NarrativeQA using Gemini and on some multi-hop questions. The paper also shows that MeMo experiences smaller performance drops than retrieval baselines as distractor documents are added to the corpus.
Critical strengths
a. The paper provides a concrete implementation and empirical study of using a document-trained expert model as external memory.
Having a larger model consult another model with specialized knowledge is a natural idea, and I do not find the idea itself particularly novel. Still, there is value in developing this approach for document-based knowledge integration, specifying how to train and query the Memory, and examining how well it works in practice.
In particular, the same trained Memory can be used with different Executives, including a closed-source model. This demonstrates a useful consequence of separating the model that stores knowledge from the model that uses it.
b. The distractor robustness results and improvements in some evaluation settings are meaningful.
In Table 4, MeMo degrades less than the retrieval baselines as the number of corpus distractors increases. For example, on BrowseComp-Plus, HippoRAG2 drops from 62.33% to 50.78%, while MeMo remains relatively stable, moving from 53.67% to 52.33%. I consider this one of the clearest empirical takeaways of the paper.
The 20.82 percentage-point improvement over the best RAG baseline on NarrativeQA with Gemini, along with the improvements on 4-hop MuSiQue questions, is also noteworthy. These results suggest that a learned Memory can be useful for particular questions and Executive models.
Critical weaknesses
1. I am not sufficiently convinced by the connection between the difficulties of existing approaches and the proposed design.
My main question while reading was what makes document knowledge integration difficult, where existing approaches fail, and how MeMo addresses those difficulties. The motivation (about external memory systems) is familiar and reasonable. However, I did not find a sufficiently clear research argument leading from that motivation to a separate small Memory model, the five-step data generation pipeline, and the three-stage inference procedure, which are introduced as the main contribution of the paper.
For example, catastrophic forgetting motivates freezing the Executive and separating knowledge learning from it. However, adapters and knowledge modules (like LoRA) already provide ways to separate newly learned knowledge from the original model. Section 5.3 shows that knowledge training can damage a model’s existing capabilities, but I would like to understand what additional difficulty a separate Memory addresses relative to these approaches.
I have a similar concern about generalization. MeMo’s Memory is also fine-tuned on synthetic QA on specific domain (like other external knowledge methods). The central question is therefore which knowledge relationships should be learned, and how they should be accessed, to answer questions that existing training approaches struggle with. Combining related facts, generating questions that identify entities from their attributes, and connecting information across documents are understandable choices. What I found less clear is which unresolved difficulty each choice addresses.
Several directly relevant works should also be discussed and (maybe) cited. Existing work on synthetic data for knowledge acquisition and generalization includes Synthetic Continued Pretraining / EntiGraph, Lampinen et al., New News, and Learning Facts at Scale with Active Reading. These are directly relevant to the motivation and design of MeMo’s reflections. PRAG, Knowledge Modules, and Back et al. are also relevant to learning and modularizing document knowledge. Discussing these connections would help establish the starting point of MeMo and what it adds.
The paper provides ablations supporting parts of the design. Cross-document synthesis contributes substantially in Table 9, and structured inference helps on BrowseComp-Plus and MuSiQue in Table 15. However, some data generation steps and the inference structure behave differently on NarrativeQA. I would have liked these differences to play a more central role in the analysis. I can follow what the procedures do, but I am left with a less clear understanding of what they teach us about effective knowledge learning and access.
2. The overall results leave the reason to choose MeMo, and the broader empirical takeaway, unclear.
Since I found the conceptual contribution limited, I looked to the experiments for a useful empirical distinction. However, in many settings in Table 2, MeMo performs similarly to or below strong RAG baselines. With Qwen as the Executive, it performs below the best RAG baseline on BrowseComp-Plus and NarrativeQA and improves modestly on MuSiQue. With Gemini, the NarrativeQA improvement is substantial, but the differences on the other two datasets are small.
The dependence on the Executive is particularly important. On NarrativeQA, the same Memory produces a large benefit with Gemini but performs below RAG with Qwen. Understanding whether this difference comes from question generation, the use of Memory responses, or final reasoning would make the contribution much clearer. At present, I find it difficult to infer which Executives and question types should benefit from this approach.
I recognize distractor robustness as a strength. However, a smaller performance drop and higher final QA accuracy are different properties. On BrowseComp-Plus, MeMo starts below HippoRAG2 without distractors, maintains its performance, and overtakes it as distractors increase. This is a useful robustness result, but it leaves open the question of MeMo’s value across the broader evaluation.
The repeated Executive–Memory interactions also require substantial inference resources. In Table 16, the Executive token counts on NarrativeQA and MuSiQue are approximately 13 and 10 times those of HippoRAG2, respectively. This makes it more important to understand the conditions under which the performance benefits justify using the system.
Overall, I think MeMo is a natural and worthwhile idea. However, I find the current methodological novelty limited. I would be more interested in seeing this approach developed in settings where the separation between knowledge storage and the Executive offers a clearer advantage. For example, a more extensive study of separately trained Memories for different domains, switching between them as the knowledge collection grows, could help establish the value of this design for knowledge retention and access at a larger scale. Or there could be much more extended forms of experiments and domains like this, where MeMo can show its advantages. In its current form, however, the paper does not provide a sufficiently strong methodological or empirical contribution to support acceptance.
Questions for authors
What is the central unresolved difficulty relative to existing approaches?
Taking prior work on synthetic-data-based knowledge learning and modular approaches such as adapters and knowledge modules as the starting point, what specific difficulty/novelty does MeMo address? Please explain how the separate Memory model and the proposed training and querying procedures improve on that difficulty.
What should we learn from combining the data generation and inference procedures?
Table 9 shows a large contribution from cross-document synthesis, while Table 15 shows that the effect of structured inference varies by dataset. Could you connect these results to concrete success and failure cases? In particular, what errors are reduced by entity-centric training and the entity identification stage, and why do the effects differ on NarrativeQA? This would help me assess the contribution of the proposed procedures.
Why does the NarrativeQA improvement depend so strongly on the Executive?
How do you interpret the large improvement with Gemini and its absence with Qwen when using the same Memory? An analysis of existing runs could help identify whether the difference arises from question generation, interpretation of Memory responses, or final answer construction. As additional validation, evaluating the same Memory with a stronger Executive or one from another model family would help establish whether this pattern extends beyond the two tested Executives.
When does distractor robustness translate into an accuracy advantage?
The stability of MeMo in Table 4 is interesting. Have you analyzed the questions that RAG answers incorrectly as noise increases but MeMo continues to answer correctly? Comparing these with questions that RAG answers correctly and MeMo misses in the no-distractor setting would help explain the trade-off and the conditions for choosing MeMo.
9 comments
Response [2/7]
While our Leave-One-Out (LOO) ablations evaluate the contributions of individual stages and show the importance of the cross-document synthesis step, we acknowledge that it does not directly compare MeMo against existing synthesis approaches. We provide additional ablations on New News only, as EntiGraph and Active Reading are methods for augmenting the data in a Continued Pre-training (CPT) setting rather than Supervised fine-tuning (SFT). To avoid potential confounding factors that may arise from converting CPT-oriented data into SFT-compatible formats, we conduct additional experiments only on New News. We compare the results across the best epoch for New News and MeMo on NQA and MSQ 1N.
| Experiment Settings | New News NQA (epoch 3) | MeMo Data synth NQA (epoch 3) | New News MSQ 1N (epoch 3) | MeMo Data Synth MSQ 1N (epoch 2) |
|---|---|---|---|---|
| (A) Eval with MeMo's inference pipeline | 8.73 ± 0.20 | 23.92 ± 0.20 | 17.97 ± 0.32 | 42.90 ± 1.39 |
| (B) Direct querying of MEMORY | 7.03 ± 0.86 | 17.35 ± 0.68 | 6.53 ± 0.42 | 28.30 ± 0.35 |
| Gain from pipeline (A − B) | +1.70 | +6.58 | +11.43 | +14.60 |
We compare New News and MeMo under two inference settings: (i) direct querying of the MEMORY with the original test-time question, and (ii) MeMo's EXECUTIVE-guided inference pipeline. The results show that MeMo-trained MEMORY outperforms New News-trained MEMORY, suggesting that the choice of data synthesis method affects downstream question-answering performance even without MeMo's evaluation pipeline. Moreover, using the pipeline improves performance for both MEMORY models, with significantly larger gains for MeMo-trained MEMORY on NQA (p=0.022) and MSQ (p=0.036).
Our qualitative analysis of QA pairs confirms that the QA pairs generated by New News contain the relevant facts, but embed them in source-referencing reasoning templates (e.g., for an MSQ QA pair, the answer to "In which London neighborhood is the Mycenae House located?" begins "Step 1: Identify the key information from the provided news. The news states that Mycenae House is located in … Westcombe Park …"). During inference, the answers use the template "Step 1: etc..." and mix up the factual information about "Mycenae House" with that about "Notting Hill". This suggests that the document-dependent reasoning templates used during data synthesis may encourage the model to reproduce answer structures rather than reliably learn the underlying factual associations. Together, these findings suggest that effective question answering depends on both the training data synthesis method and the inference-time procedure, providing evidence consistent with the importance of aligning the two.
Given New News' relatively poor performance under MeMo's inference pipeline, one might question the motivation behind this particular inference pipeline. Among many possible designs, we modeled this interaction as how a person might answer a given question. An individual might begin by considering a wide range of possibilities, before narrowing down their options and diving deeper to finalize an answer. Crucially, we do not claim that this inference pipeline is optimal among all possible inference pipelines; this remains an exciting direction for future work.
Overall, compared with the above-listed synthetic-data-based knowledge-learning works, MeMo contributes by integrating multiple data-synthesis steps into a single pipeline and demonstrates improved performance over New News. Additionally, MeMo contributes by providing empirical observations on how access to knowledge during inference is influenced by both the training and inference pipelines. Our results support the design principle articulated in our introduction: "How we intend the MEMORY model to be queried (by the EXECUTIVE model) should inform how its training data is synthesized."
Modular approaches - Existing modular approaches such as adapters (e.g., LoRA), while helpful in mitigating the effects of catastrophic forgetting in the EXECUTIVE, are not able to effectively answer the queries, as the knowledge access mechanism of direct querying is insufficient (refer to the table below for results on LoRA training relative to MeMo). Other knowledge modules (e.g., ICAE, MemoryDecoder) rely on a specific model or encoder to encode knowledge from text into parametric or latent space, which limits integration across LLMs, as the EXECUTIVE needs to use the same encoder to utilize the knowledge module. The closest knowledge-module approach we found is Cartridges. However, that, like LoRA, relies on white-box access to the model to use the trained KV cache, making it infeasible to integrate with black-box API access models.
↓ ↓ ↓ Continued below ↓ ↓ ↓
Response [3/7]
In line with our response to reviewer FL5d (Q2), we report the performance of the EXECUTIVE model under full fine-tuning and LoRA fine-tuning on the same reflection dataset. For inference, similar to checking for pre-existing knowledge (Sec. J in Appendix, Tab. 12), a direct query method was used, and no additional question breakdown or stage planning was performed, as there is no separate EXECUTIVE present in this pipeline.
| Dataset | Direct Query Best 14B-Full SFT | Direct Query Best 32B Full SFT | Direct Query Best 32B LoRA | MeMo (32B EM + 14B MEM) |
|---|---|---|---|---|
| BCP | 31.1 ± 0.4 | 35.1 ± 1.3 | 33.9 ± 0.4 | 52.78 +- 0.77 |
| NQA | 20.1 ± 0.6 | 20.1 ± 0.0 | 19.5 ± 0.2 | 26.85 +- 0.39 |
| MSQ | 40.4 ± 0.7 | 40.6 ± 0.1 | 35.3 ± 0.5 | 48.30 +- 1.25 |
| MSQ Hops (No. of questions) | Direct Query Best 14B-Full SFT | Direct Query Best 32B Full SFT | Direct Query Best 32B LoRA | MeMo (32B EM + 14B MEM) |
|---|---|---|---|---|
| 2-hop (518) | 44.5 ± 0.9 | 44.2 ± 0.0 | 39.7 ± 1.0 | 53.60 +- 1.83 |
| 3-hop (316) | 40.9 ± 2.1 | 44.5 ± 0.7 | 36.2 ± 0.5 | 46.41 +- 3.16 |
| 4-hop (166) | 26.5 ± 2.8 | 21.9 ± 1.5 | 19.7 ± 0.3 | 35.34 +- 1.14 |
Since the full fine-tuning performance is better than LoRA's, we analyze using full fine-tuning as the reference. We observe that full fine-tuning yields some general improvements when increasing from a 14B to a 32B base model, improving performance on BCP while maintaining performance on NQA and MSQ. We compare the results against the full memo pipeline and demonstrate that it significantly outperforms full fine-tuning of the EXECUTIVE itself. We attribute this performance gain to better handling of queries outside training.
Training methods such as fine-tuning a model on a corpus help it memorize facts but make it difficult to access them, as the knowledge stored during fine-tuning is often only accessible when the query resembles the form in which it was learned [1]. Direct querying of unseen test-time questions, especially multi-hop or multi-constraint ones, often fails to access the knowledge that a model has.
Our analysis of the gap between direct querying of the small model and MeMo demonstrates that the knowledge is stored, but the original question fails to retrieve it. For instance, some failure modes of direct querying involve (i) identifying the right entity but giving the wrong answer, (ii) identifying the wrong entity, (iii) stopping at an intermediate hop (e.g., the question asks for a character that an actor has played; direct query answers about the actor instead of the character), (iv) not following instructions based on what the question is asking (e.g., the question asks for when; direct query responds with who). MeMo addresses each of these failure modes at inference time. For failure modes (i) and (ii), MeMo Inference Stage 2 (entity identification) iteratively asks follow-up questions to narrow down the set of possible candidate entities that have been surfaced. For failure modes (iii) and (iv), MeMo breaks down the question at Stage 1 (Grounding) and iteratively queries the Memory model for additional knowledge of targeted follow-up questions to ensure that the EXECUTIVE has enough to answer the question in its required format.
As such, within the scope of training a parametric store via fine-tuning, the empirical results demonstrate that both the capabilities of the EXECUTIVE (i.e., instruction following) and targeted knowledge follow-ups are necessary to access sufficient knowledge from the parametric store to answer test-time queries. As highlighted in our Future Works section (App C), other post-training methods, such as Reinforcement Learning, are potentially worth exploring but are out of scope for this paper and left to future work.
Other Limitations One constraint with knowledge modules like LoRA, which could mitigate the catastrophic forgetting of learned capabilities, is that it's not always possible given that some models are only available via black-box access. Other knowledge module baselines we mentioned in the paper, such as ICAE, MemoryDecoder, and Cartridges, also require white-box access. Even then, we observe that their performance is not competitive with the RAG baselines of HippoRAG2 and NV-Embed-V2. We provide a discussion of their performance in App. R.
↓ ↓ ↓ Continued below ↓ ↓ ↓
Response [1/7]
We thank the reviewer for a careful and detailed review. We appreciate the recognition that the paper provides a concrete implementation and empirical study of using a document-trained expert model as external memory, and that there is value in developing this approach for document-based knowledge integration, specifying how to train and query the Memory model, and examining how well it works in practice. In particular, we appreciate the recognition that the same trained Memory can be used with different Executives, including a closed-source model, as a useful consequence of separating the model that stores knowledge from the model that uses it. We also appreciate the recognition of the distractor robustness results and the improvements on 4-hop MuSiQue questions, which suggest that a learned Memory can be useful for particular questions and Executive models. We have grouped and addressed related weaknesses and questions as follows:
The central unresolved challenge is how to enable a parametric knowledge store to support multi-hop reasoning and narrative understanding over long contexts, while allowing plug-and-play integration with LLMs across model families, including black-box models.
We are interested in the parametric knowledge store, as the non-parametric store exhibits a decrease in performance as the amount of noise (distractor documents) is present in the document corpus. In contrast, our results for MeMo demonstrate that parametric knowledge stores can increase robustness to noise. We elaborate more in our response to Q7/Q8.
Supporting multi-hop reasoning and narrative understanding across long contexts is challenging for parametric stores, as access to the required knowledge depends on both the training data and the inference pipeline. Existing methods for data synthesis, such as EntiGraph and Active Reading, are only applicable to the Continued Pre-training setting. Methods like New News apply data synthesis only to individual documents, rather than synthesizing information jointly across multiple documents. In our paper and the responses below, we identify failure modes that arise in multi-hop reasoning and narrative understanding with long context when naive methods such as direct querying or unstructured multi-turn queries are used. We elaborate more in our response to Q2/3/4.
Existing parametric (e.g., LoRA, MemoryDecoder, Cartridges) and latent memory methods (e.g., ICAE) are incompatible with LLMs that are only accessible via API calls. Some have additional constraints, requiring that the same encoder be used or that the LLM belong to the same family for these methods to integrate with the EXECUTIVE model.
Synthetic-data-based knowledge learning - From the related works mentioned, EntiGraph, New News, and Active Reading, we acknowledge that their motivations are similar in terms of synthetic knowledge acquisition. For instance, EntiGraph is most closely related to Step 4 (entity surfacing), New News to Step 1b (indirect fact extraction), and Active Reading is broadly similar to Step 2 (consolidation) but not a methodological equivalent. Additionally, these related works have synthesis procedures that operate on individual source documents, missing out on comparison to Step 5 (cross-document synthesis). As such, relative to these 3 related works, our contribution to the data synthesis pipeline lies in combining all the steps and exploring ways to synthesize knowledge across multiple documents.
↓ ↓ ↓ Continued below ↓ ↓ ↓