acceptodds
Under review as a conference paper at ICLR 2027

MeMo: Memory as a Model

Abstract

Large language models (LLMs) achieve strong performance across a wide range of tasks but remain frozen after pretraining until subsequent updates. Many real-world applications require domain-specific information which might be absent in the LLM's retained knowledge, motivating the need for efficient mechanisms to incorporate new knowledge. We introduce MeMo (Memory as a Model), a modular framework that synthesizes and internalizes into a dedicated MEMORY model, which an EXECUTIVE model queries at inference time for relevant knowledge and reasons over the retrieved information. MeMo offers several benefits: (a) it is more robust to the presence of distractors in the target corpus, (b) it prevents catastrophic forgetting in the EXECUTIVE model, and (c) it enables plug-and-play integration with differents LLMs, including closed-source LLMs. Across three benchmarks, MeMo outperforms evaluated parametric and latent memory baselines while remaining competitive with strong non-parametric retrieval baselines, statsisignificantly outperforming them on MuSiQue.

Primary area:
generative models
Keywords:
Large Language Models, Memory Models, Knowledge Augmentation, Catastrophic Forgetting, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XL6COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MeMo, a framework that trains a smaller language model to store knowledge from a document corpus and allows a larger model to access that knowledge through natural-language questions. The framework separates a Memory model, which stores the corpus knowledge, from an Executive model, which handles reasoning and produces the final answer. The Executive remains frozen.

The Memory is trained on synthetic QA pairs called reflections. The data generation pipeline includes fact extraction, consolidation of related facts, verification and rewriting for self-containment, entity-surfacing QA generation, and cross-document synthesis. At inference time, the Executive interacts with the Memory through three stages: grounding, entity identification, and answer seeking.

The paper evaluates MeMo on BrowseComp-Plus, NarrativeQA, and MuSiQue against retrieval and parametric memory baselines. Results vary across Executive models and datasets, with improvements on NarrativeQA using Gemini and on some multi-hop questions. The paper also shows that MeMo experiences smaller performance drops than retrieval baselines as distractor documents are added to the corpus.

Critical strengths

a. The paper provides a concrete implementation and empirical study of using a document-trained expert model as external memory.

Having a larger model consult another model with specialized knowledge is a natural idea, and I do not find the idea itself particularly novel. Still, there is value in developing this approach for document-based knowledge integration, specifying how to train and query the Memory, and examining how well it works in practice.

In particular, the same trained Memory can be used with different Executives, including a closed-source model. This demonstrates a useful consequence of separating the model that stores knowledge from the model that uses it.

b. The distractor robustness results and improvements in some evaluation settings are meaningful.

In Table 4, MeMo degrades less than the retrieval baselines as the number of corpus distractors increases. For example, on BrowseComp-Plus, HippoRAG2 drops from 62.33% to 50.78%, while MeMo remains relatively stable, moving from 53.67% to 52.33%. I consider this one of the clearest empirical takeaways of the paper.

The 20.82 percentage-point improvement over the best RAG baseline on NarrativeQA with Gemini, along with the improvements on 4-hop MuSiQue questions, is also noteworthy. These results suggest that a learned Memory can be useful for particular questions and Executive models.

Critical weaknesses

1. I am not sufficiently convinced by the connection between the difficulties of existing approaches and the proposed design.

My main question while reading was what makes document knowledge integration difficult, where existing approaches fail, and how MeMo addresses those difficulties. The motivation (about external memory systems) is familiar and reasonable. However, I did not find a sufficiently clear research argument leading from that motivation to a separate small Memory model, the five-step data generation pipeline, and the three-stage inference procedure, which are introduced as the main contribution of the paper.

For example, catastrophic forgetting motivates freezing the Executive and separating knowledge learning from it. However, adapters and knowledge modules (like LoRA) already provide ways to separate newly learned knowledge from the original model. Section 5.3 shows that knowledge training can damage a model’s existing capabilities, but I would like to understand what additional difficulty a separate Memory addresses relative to these approaches.

I have a similar concern about generalization. MeMo’s Memory is also fine-tuned on synthetic QA on specific domain (like other external knowledge methods). The central question is therefore which knowledge relationships should be learned, and how they should be accessed, to answer questions that existing training approaches struggle with. Combining related facts, generating questions that identify entities from their attributes, and connecting information across documents are understandable choices. What I found less clear is which unresolved difficulty each choice addresses.

Several directly relevant works should also be discussed and (maybe) cited. Existing work on synthetic data for knowledge acquisition and generalization includes Synthetic Continued Pretraining / EntiGraph, Lampinen et al., New News, and Learning Facts at Scale with Active Reading. These are directly relevant to the motivation and design of MeMo’s reflections. PRAG, Knowledge Modules, and Back et al. are also relevant to learning and modularizing document knowledge. Discussing these connections would help establish the starting point of MeMo and what it adds.

The paper provides ablations supporting parts of the design. Cross-document synthesis contributes substantially in Table 9, and structured inference helps on BrowseComp-Plus and MuSiQue in Table 15. However, some data generation steps and the inference structure behave differently on NarrativeQA. I would have liked these differences to play a more central role in the analysis. I can follow what the procedures do, but I am left with a less clear understanding of what they teach us about effective knowledge learning and access.

2. The overall results leave the reason to choose MeMo, and the broader empirical takeaway, unclear.

Since I found the conceptual contribution limited, I looked to the experiments for a useful empirical distinction. However, in many settings in Table 2, MeMo performs similarly to or below strong RAG baselines. With Qwen as the Executive, it performs below the best RAG baseline on BrowseComp-Plus and NarrativeQA and improves modestly on MuSiQue. With Gemini, the NarrativeQA improvement is substantial, but the differences on the other two datasets are small.

The dependence on the Executive is particularly important. On NarrativeQA, the same Memory produces a large benefit with Gemini but performs below RAG with Qwen. Understanding whether this difference comes from question generation, the use of Memory responses, or final reasoning would make the contribution much clearer. At present, I find it difficult to infer which Executives and question types should benefit from this approach.

I recognize distractor robustness as a strength. However, a smaller performance drop and higher final QA accuracy are different properties. On BrowseComp-Plus, MeMo starts below HippoRAG2 without distractors, maintains its performance, and overtakes it as distractors increase. This is a useful robustness result, but it leaves open the question of MeMo’s value across the broader evaluation.

The repeated Executive–Memory interactions also require substantial inference resources. In Table 16, the Executive token counts on NarrativeQA and MuSiQue are approximately 13 and 10 times those of HippoRAG2, respectively. This makes it more important to understand the conditions under which the performance benefits justify using the system.

Overall, I think MeMo is a natural and worthwhile idea. However, I find the current methodological novelty limited. I would be more interested in seeing this approach developed in settings where the separation between knowledge storage and the Executive offers a clearer advantage. For example, a more extensive study of separately trained Memories for different domains, switching between them as the knowledge collection grows, could help establish the value of this design for knowledge retention and access at a larger scale. Or there could be much more extended forms of experiments and domains like this, where MeMo can show its advantages. In its current form, however, the paper does not provide a sufficiently strong methodological or empirical contribution to support acceptance.

Questions for authors

What is the central unresolved difficulty relative to existing approaches?

Taking prior work on synthetic-data-based knowledge learning and modular approaches such as adapters and knowledge modules as the starting point, what specific difficulty/novelty does MeMo address? Please explain how the separate Memory model and the proposed training and querying procedures improve on that difficulty.

What should we learn from combining the data generation and inference procedures?

Table 9 shows a large contribution from cross-document synthesis, while Table 15 shows that the effect of structured inference varies by dataset. Could you connect these results to concrete success and failure cases? In particular, what errors are reduced by entity-centric training and the entity identification stage, and why do the effects differ on NarrativeQA? This would help me assess the contribution of the proposed procedures.

Why does the NarrativeQA improvement depend so strongly on the Executive?

How do you interpret the large improvement with Gemini and its absence with Qwen when using the same Memory? An analysis of existing runs could help identify whether the difference arises from question generation, interpretation of Memory responses, or final answer construction. As additional validation, evaluating the same Memory with a stronger Executive or one from another model family would help establish whether this pattern extends beyond the two tested Executives.

When does distractor robustness translate into an accuracy advantage?

The stability of MeMo in Table 4 is interesting. Have you analyzed the questions that RAG answers incorrectly as noise increases but MeMo continues to answer correctly? Comparing these with questions that RAG answers correctly and MeMo misses in the no-distractor setting would help explain the trade-off and the conditions for choosing MeMo.

9 comments
AuthorsOpenReview ↗

Response [1/7]

We thank the reviewer for a careful and detailed review. We appreciate the recognition that the paper provides a concrete implementation and empirical study of using a document-trained expert model as external memory, and that there is value in developing this approach for document-based knowledge integration, specifying how to train and query the Memory model, and examining how well it works in practice. In particular, we appreciate the recognition that the same trained Memory can be used with different Executives, including a closed-source model, as a useful consequence of separating the model that stores knowledge from the model that uses it. We also appreciate the recognition of the distractor robustness results and the improvements on 4-hop MuSiQue questions, which suggest that a learned Memory can be useful for particular questions and Executive models. We have grouped and addressed related weaknesses and questions as follows:

Q1 What is the central unresolved difficulty relative to existing approaches?

The central unresolved challenge is how to enable a parametric knowledge store to support multi-hop reasoning and narrative understanding over long contexts, while allowing plug-and-play integration with LLMs across model families, including black-box models.

We are interested in the parametric knowledge store, as the non-parametric store exhibits a decrease in performance as the amount of noise (distractor documents) is present in the document corpus. In contrast, our results for MeMo demonstrate that parametric knowledge stores can increase robustness to noise. We elaborate more in our response to Q7/Q8.

Supporting multi-hop reasoning and narrative understanding across long contexts is challenging for parametric stores, as access to the required knowledge depends on both the training data and the inference pipeline. Existing methods for data synthesis, such as EntiGraph and Active Reading, are only applicable to the Continued Pre-training setting. Methods like New News apply data synthesis only to individual documents, rather than synthesizing information jointly across multiple documents. In our paper and the responses below, we identify failure modes that arise in multi-hop reasoning and narrative understanding with long context when naive methods such as direct querying or unstructured multi-turn queries are used. We elaborate more in our response to Q2/3/4.

Existing parametric (e.g., LoRA, MemoryDecoder, Cartridges) and latent memory methods (e.g., ICAE) are incompatible with LLMs that are only accessible via API calls. Some have additional constraints, requiring that the same encoder be used or that the LLM belong to the same family for these methods to integrate with the EXECUTIVE model.

Q2 Taking prior work on synthetic-data-based knowledge learning and modular approaches such as adapters and knowledge modules as the starting point, what specific difficulty/novelty does MeMo address? Please explain how the separate Memory model and the proposed training and querying procedures improve on that difficulty.

Q3 What should we learn from combining the data generation and inference procedures?

Synthetic-data-based knowledge learning - From the related works mentioned, EntiGraph, New News, and Active Reading, we acknowledge that their motivations are similar in terms of synthetic knowledge acquisition. For instance, EntiGraph is most closely related to Step 4 (entity surfacing), New News to Step 1b (indirect fact extraction), and Active Reading is broadly similar to Step 2 (consolidation) but not a methodological equivalent. Additionally, these related works have synthesis procedures that operate on individual source documents, missing out on comparison to Step 5 (cross-document synthesis). As such, relative to these 3 related works, our contribution to the data synthesis pipeline lies in combining all the steps and exploring ways to synthesize knowledge across multiple documents.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [2/7]

While our Leave-One-Out (LOO) ablations evaluate the contributions of individual stages and show the importance of the cross-document synthesis step, we acknowledge that it does not directly compare MeMo against existing synthesis approaches. We provide additional ablations on New News only, as EntiGraph and Active Reading are methods for augmenting the data in a Continued Pre-training (CPT) setting rather than Supervised fine-tuning (SFT). To avoid potential confounding factors that may arise from converting CPT-oriented data into SFT-compatible formats, we conduct additional experiments only on New News. We compare the results across the best epoch for New News and MeMo on NQA and MSQ 1N.

Experiment SettingsNew News NQA (epoch 3)MeMo Data synth NQA (epoch 3)New News MSQ 1N (epoch 3)MeMo Data Synth MSQ 1N (epoch 2)
(A) Eval with MeMo's inference pipeline8.73 ± 0.2023.92 ± 0.2017.97 ± 0.3242.90 ± 1.39
(B) Direct querying of MEMORY7.03 ± 0.8617.35 ± 0.686.53 ± 0.4228.30 ± 0.35
Gain from pipeline (A − B)+1.70+6.58+11.43+14.60

We compare New News and MeMo under two inference settings: (i) direct querying of the MEMORY with the original test-time question, and (ii) MeMo's EXECUTIVE-guided inference pipeline. The results show that MeMo-trained MEMORY outperforms New News-trained MEMORY, suggesting that the choice of data synthesis method affects downstream question-answering performance even without MeMo's evaluation pipeline. Moreover, using the pipeline improves performance for both MEMORY models, with significantly larger gains for MeMo-trained MEMORY on NQA (p=0.022) and MSQ (p=0.036).

Our qualitative analysis of QA pairs confirms that the QA pairs generated by New News contain the relevant facts, but embed them in source-referencing reasoning templates (e.g., for an MSQ QA pair, the answer to "In which London neighborhood is the Mycenae House located?" begins "Step 1: Identify the key information from the provided news. The news states that Mycenae House is located in … Westcombe Park …"). During inference, the answers use the template "Step 1: etc..." and mix up the factual information about "Mycenae House" with that about "Notting Hill". This suggests that the document-dependent reasoning templates used during data synthesis may encourage the model to reproduce answer structures rather than reliably learn the underlying factual associations. Together, these findings suggest that effective question answering depends on both the training data synthesis method and the inference-time procedure, providing evidence consistent with the importance of aligning the two.

Given New News' relatively poor performance under MeMo's inference pipeline, one might question the motivation behind this particular inference pipeline. Among many possible designs, we modeled this interaction as how a person might answer a given question. An individual might begin by considering a wide range of possibilities, before narrowing down their options and diving deeper to finalize an answer. Crucially, we do not claim that this inference pipeline is optimal among all possible inference pipelines; this remains an exciting direction for future work.

Overall, compared with the above-listed synthetic-data-based knowledge-learning works, MeMo contributes by integrating multiple data-synthesis steps into a single pipeline and demonstrates improved performance over New News. Additionally, MeMo contributes by providing empirical observations on how access to knowledge during inference is influenced by both the training and inference pipelines. Our results support the design principle articulated in our introduction: "How we intend the MEMORY model to be queried (by the EXECUTIVE model) should inform how its training data is synthesized."

Modular approaches - Existing modular approaches such as adapters (e.g., LoRA), while helpful in mitigating the effects of catastrophic forgetting in the EXECUTIVE, are not able to effectively answer the queries, as the knowledge access mechanism of direct querying is insufficient (refer to the table below for results on LoRA training relative to MeMo). Other knowledge modules (e.g., ICAE, MemoryDecoder) rely on a specific model or encoder to encode knowledge from text into parametric or latent space, which limits integration across LLMs, as the EXECUTIVE needs to use the same encoder to utilize the knowledge module. The closest knowledge-module approach we found is Cartridges. However, that, like LoRA, relies on white-box access to the model to use the trained KV cache, making it infeasible to integrate with black-box API access models.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [3/7]

In line with our response to reviewer FL5d (Q2), we report the performance of the EXECUTIVE model under full fine-tuning and LoRA fine-tuning on the same reflection dataset. For inference, similar to checking for pre-existing knowledge (Sec. J in Appendix, Tab. 12), a direct query method was used, and no additional question breakdown or stage planning was performed, as there is no separate EXECUTIVE present in this pipeline.

DatasetDirect Query Best 14B-Full SFTDirect Query Best 32B Full SFTDirect Query Best 32B LoRAMeMo (32B EM + 14B MEM)
BCP31.1 ± 0.435.1 ± 1.333.9 ± 0.452.78 +- 0.77
NQA20.1 ± 0.620.1 ± 0.019.5 ± 0.226.85 +- 0.39
MSQ40.4 ± 0.740.6 ± 0.135.3 ± 0.548.30 +- 1.25
MSQ Hops (No. of questions)Direct Query Best 14B-Full SFTDirect Query Best 32B Full SFTDirect Query Best 32B LoRAMeMo (32B EM + 14B MEM)
2-hop (518)44.5 ± 0.944.2 ± 0.039.7 ± 1.053.60 +- 1.83
3-hop (316)40.9 ± 2.144.5 ± 0.736.2 ± 0.546.41 +- 3.16
4-hop (166)26.5 ± 2.821.9 ± 1.519.7 ± 0.335.34 +- 1.14

Since the full fine-tuning performance is better than LoRA's, we analyze using full fine-tuning as the reference. We observe that full fine-tuning yields some general improvements when increasing from a 14B to a 32B base model, improving performance on BCP while maintaining performance on NQA and MSQ. We compare the results against the full memo pipeline and demonstrate that it significantly outperforms full fine-tuning of the EXECUTIVE itself. We attribute this performance gain to better handling of queries outside training.

Training methods such as fine-tuning a model on a corpus help it memorize facts but make it difficult to access them, as the knowledge stored during fine-tuning is often only accessible when the query resembles the form in which it was learned [1]. Direct querying of unseen test-time questions, especially multi-hop or multi-constraint ones, often fails to access the knowledge that a model has.

Our analysis of the gap between direct querying of the small model and MeMo demonstrates that the knowledge is stored, but the original question fails to retrieve it. For instance, some failure modes of direct querying involve (i) identifying the right entity but giving the wrong answer, (ii) identifying the wrong entity, (iii) stopping at an intermediate hop (e.g., the question asks for a character that an actor has played; direct query answers about the actor instead of the character), (iv) not following instructions based on what the question is asking (e.g., the question asks for when; direct query responds with who). MeMo addresses each of these failure modes at inference time. For failure modes (i) and (ii), MeMo Inference Stage 2 (entity identification) iteratively asks follow-up questions to narrow down the set of possible candidate entities that have been surfaced. For failure modes (iii) and (iv), MeMo breaks down the question at Stage 1 (Grounding) and iteratively queries the Memory model for additional knowledge of targeted follow-up questions to ensure that the EXECUTIVE has enough to answer the question in its required format.

As such, within the scope of training a parametric store via fine-tuning, the empirical results demonstrate that both the capabilities of the EXECUTIVE (i.e., instruction following) and targeted knowledge follow-ups are necessary to access sufficient knowledge from the parametric store to answer test-time queries. As highlighted in our Future Works section (App C), other post-training methods, such as Reinforcement Learning, are potentially worth exploring but are out of scope for this paper and left to future work.

Other Limitations One constraint with knowledge modules like LoRA, which could mitigate the catastrophic forgetting of learned capabilities, is that it's not always possible given that some models are only available via black-box access. Other knowledge module baselines we mentioned in the paper, such as ICAE, MemoryDecoder, and Cartridges, also require white-box access. Even then, we observe that their performance is not competitive with the RAG baselines of HippoRAG2 and NV-Embed-V2. We provide a discussion of their performance in App. R.

↓ ↓ ↓ Continued below ↓ ↓ ↓

Reviewer_e7M1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose training a "memory" model on a given corpus, which can then be queried by any LLM via natural language. The authors claim this avoids the typical issues of non-parametric methods, fine-tuning to internalize new knowledge and latent memory methods.

Critical strengths
  • It's a good showcase that such type of methods can work. Decent set of experiments.
  • Clear motivation, paper is easy to read and follow
Critical weaknesses
  • The high level idea and the individual design decisions aren't novel. Taken together they might be. I don't think this is a big issue, but it's worth mentioning that the contribution is more a showcase that this can work rather than proposing an entirely new idea.
  • The method is sold as combining "the complementary strengths of the three paradigms" (non-parametric, parametric, and latent methods), but it has limitations of its own which aren't properly addressed in my opinion. For instance, the method requires fine-tuning a model, same as with parametric methods. It can also suffer from catastrophic forgetting and be vulnerable to retrieval problems. I believe the authors should be fair about the pros and cons.
  • Competitive with RAG methods, but doesn't necessarily outperform them.
Questions for authors
  • What's the cost of training the memory model and how does that affect the practicality of the method in contrast to other types of methods such as non-parametric ones?
  • You show that catastrophic forgetting happens in the memory model, isn't that potentially an issue?
  • Isn't the memory model also vulnerable to retrieval quality problems?
5 comments
AuthorsOpenReview ↗

Response [1/3]

We thank the reviewer for a careful and encouraging review. We appreciate the recognition that the paper is a good showcase that methods like MeMo can work, with a decent set of experiments, and that its motivation is clear and the paper is easy to read and follow. We have grouped and addressed related weaknesses and questions as follows:

W1 The high level idea and the individual design decisions aren't novel. Taken together they might be. I don't think this is a big issue, but it's worth mentioning that the contribution is more a showcase that this can work rather than proposing an entirely new idea.

We acknowledge that the idea of a parametric store of knowledge, augmented with various data synthesis techniques, has been proposed before. We appreciate the reviewer for noting this in their concerns, and we would like to take the opportunity to emphasize what we believe are the novel aspects of our work. The design for true plug-and-play integration with a parametric store while maintaining performance is novel, as all previous methods either rely on white-box access to the models or are only integrable within the same model family. Our empirical contributions showing that parametric stores of knowledge are competitive with non-parametric stores, when other prior works (to the best of our knowledge) are not able to be competitive, bring novelty.

In response to reviewer XL6C (Q2/3/4), we also included additional ablations on parts of the pipeline (i.e., data synthesis) to demonstrate the complexity of combining all the different decisions.

W2 The method is sold as combining "the complementary strengths of the three paradigms" (non-parametric, parametric, and latent methods), but it has limitations of its own which aren't properly addressed in my opinion. For instance, the method requires fine-tuning a model, same as with parametric methods. It can also suffer from catastrophic forgetting and be vulnerable to retrieval problems. I believe the authors should be fair about the pros and cons.

Regarding pros and cons. We agree with the reviewer that we highlighted only MeMo's complementary strengths in Tab. 1, which compares desirable properties across three existing memory paradigms. However, we have also highlighted the key limitations of MeMo in Sec. B and Sec. C of Appendix (Page 26).

Catastrophic forgetting and retrieval problems. We agree that MeMo is not immune to these issues. Forgetting of general capabilities in the MEMORY model is an expected trade-off when incorporating new knowledge, as the EXECUTIVE retains these capabilities. Similarly, the MEMORY model can experience knowledge-access failures, which MeMo's pipeline is designed to mitigate but cannot fully eliminate. Further details are provided in our response to Q2 below.

W3 Competitive with RAG methods, but doesn't necessarily outperform them.

We agree that, with Qwen2.5-32B-Instruct, MeMo is statistically on par with the strongest RAG baselines on BrowseComp-Plus and NarrativeQA (differences are not significant, p=0.32p = 0.32 and 0.280.28, respectively; Lines 351–353). Consistent with this, the paper claims competitiveness rather than universal superiority (Abstract, Lines 019–021). However, MeMo outperforms these baselines in several settings:

  • MuSiQue: +2.53 pp overall (Tab. 2) and +10.64 pp on 4-hop questions (Tab. 3).
  • NarrativeQA with Gemini-3-Flash: +20.82 pp (Tab. 2).
  • With 2N distractors: MeMo achieves the highest accuracy on both BrowseComp-Plus (52.33 vs. 50.78) and MuSiQue (47.83 vs. 45.37) (Tab. 4).
  • Against other parametric and latent-memory methods: MeMo is the only evaluated method to achieve performance in the range of strong RAG baselines. The other parametric and latent-memory baselines reach at most 19.68% (Tab. 2), and none supports a closed-source EXECUTIVE.

Our claim is therefore not that MeMo universally outperforms RAG, but that it is the first parametric knowledge store to achieve parity with strong RAG baselines.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [2/3]

Q1 What's the cost of training the memory model and how does that affect the practicality of the method in contrast to other types of methods such as non-parametric ones?

The cost of training the memory model includes both data synthesis and fine-tuning for up to 3 epochs. We report the costs associated with each in App G, in terms of GPU hours on Nvidia H200 hardware.

For data synthesis, generating the full reflection dataset for BrowseComp-Plus, NarrativeQA, and MuSiQue took approximately 440, 200, and 150 GPU-hours, respectively. For training, fine-tuning the MEMORY model (Qwen2.5-14B-Instruct) over 3 epochs for a single run on BrowseComp-Plus, NarrativeQA, and MuSiQue took approximately 320, 150, and 90 GPU-hours, respectively.

One advantage of a non-parametric system is that incorporating new knowledge incrementally is cheaper than with MeMo. We provide preliminary explorations of how incremental updates to parametric stores can be done in App P.

In terms of practicality, the current version of MeMo is suitable for settings where the knowledge does not change frequently, such that the cost of synthesizing data and training the memory model can be amortized over the queries served. Yet, we would also like to highlight that the development of this method is at its infant stages. This paper seeks to demonstrate that parametric knowledge stores can achieve competitive performance with non-parametric ones, while offering unique strengths (i.e., greater robustness to noise in the document corpus and better performance on 4- and multi-hop reasoning questions). The limitations highlighted are outside the scope of this paper and remain promising directions for future research.

W2/Q2 You show that catastrophic forgetting happens in the memory model, isn't that potentially an issue? Isn't the memory model also vulnerable to retrieval quality problems?

The results in Tab. 5 demonstrate that catastrophic forgetting of previously learned capabilities can occur during MEMORY model fine-tuning. This concern is one of the key motivations behind MeMo's modular design, in which the EXECUTIVE model remains frozen while the MEMORY model is fine-tuned to internalize corpus knowledge. Importantly, the MEMORY model is intended to serve as a specialized knowledge store rather than a general-purpose assistant. Its primary objective is to memorize corpus information and make it accessible to the EXECUTIVE at inference time. Consequently, degradation in general-purpose capabilities does not necessarily imply a corresponding degradation in its intended knowledge-storage function. By separating these responsibilities, MeMo preserves the EXECUTIVE's existing capabilities, such as safety alignment training, instruction following, tool use, and general reasoning, without exposing them to potential degradation from corpus-specific fine-tuning. In the case of a malicious prompt being sent as part of the query, the safety alignment training in the EXECUTIVE could act as a filter before determining whether it can be sent to the MEMORY, which has catastrophically forgotten its previously learned capabilities.

Beyond forgetting previously learned capabilities, we also acknowledge that catastrophic forgetting of previously learned knowledge can occur when the knowledge in the parametric store needs to be updated. With regard to these scenarios, we have provided preliminary explorations and a discussion of 2 methods, model merging and knowledge editing, in the paper. Model merging experiments are provided in App P.1 and knowledge editing in App P.2.

Crucially, we do not claim that MeMo is immune to knowledge-access failures. In fact, accurately accessing the required knowledge at test time for unknown queries, in multi-hop or narrative understanding over long contexts, is a key challenge we seek to address in the paper.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [3/3]

In our response to reviewer FL5d (Q2), we provide supplementary results on direct fine-tuning of the EXECUTIVE with the same reflection corpus used to train MEMORY. We report results for both full fine-tuning and LoRA fine-tuning from direct querying. Our results show that MeMo significantly outperforms the direct querying of the fine-tuned models, and we identify 4 failure modes of direct querying across our 3 datasets. These failure modes are as follows: (i) identifying the right entity but giving the wrong answer, (ii) identifying the wrong entity, (iii) stopping at an intermediate hop (e.g., a question asks for a character that an actor has played; direct query answers about the actor instead of the character), and (iv) not following instructions based on what the question is asking (e.g., a question asks for when; direct query responds with who). MeMo's inference pipeline thus represents our contribution to improving access to knowledge in parametric stores. While the inference pipeline helps to improve performance, we acknowledge that there is still more work to be done. Research directions such as exploring other post-training methods (e.g., Reinforcement Learning) are potentially worth pursuing but are out of scope for this paper and left to future work.

As detailed in our response to reviewer XL6C (Q2, Q3, Q4), given MeMo's inference pipeline, improving knowledge access can also occur during the data synthesis step. Through our Leave-One-Out ablations on the steps in MeMo's data synthesis pipeline (Tab. 9), we analyze why performance decreases when the entity-surfacing step (step 4) is removed. As we dive deeper into MeMo's inference pipeline, we would like to highlight the inference pipeline ablations (Tab. 15) and additional results requested by reviewer XL6C, in which the entity identification stage (stage 2) is removed. The analysis shows that the 2 components of entity-centric training and the entity identification stage at inference address entity identification errors in a complementary manner. Entity-centric training strengthens entity-fact associations within MEMORY during the entity-surfacing step and at the downstream cross-document synthesis. On the other hand, the entity identification stage guides the EXECUTIVE in pinpointing the target entity using the clues provided in the question. These findings support our design rationale of aligning training-data synthesis with the intended inference procedure to improve knowledge access.

One aspect of retrieval quality we mention in the paper is the addition of distractors to the document corpus, which causes non-parametric RAG performance to decrease as the distractor ratio increases. Unlike RAG with non-parametric stores, we demonstrate in Tab. 4 that access to knowledge is more robust to distractors in the corpus. Further analysis of these results shows that this trend can be attributed to two reasons: (1) by internalizing document knowledge inside its parameters, the MEMORY model can access relevant facts without requiring their source documents to compete with distractors for retrieval rankings or a limited retrieval budget, and (2) the structured interaction between the MEMORY and EXECUTIVE models allows the EXECUTIVE to probe multiple clues, identify inconsistencies, and issue follow-up queries to correct or refine misleading information, reducing the impact of incorrect MEMORY responses on the final answer. The structural advantages of such forms of retrieval motivate consideration of parametric knowledge stores, especially when the knowledge required for one question might serve as a distractor for another.

We thank the reviewer again for the careful review, which will help us improve the paper. We would be happy to provide any further clarification or results during the discussion period. We hope these additions address your concerns and strengthen your view of our work.

Reviewer_FL5dOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MeMo stores corpus knowledge in a small MEMORY model that is queried by a frozen EXECUTIVE LLM through a structured multi-turn protocol. The MEMORY model is trained on synthetic "reflections" produced by a five-step pipeline: fact extraction, consolidation, self-containment verification, entity surfacing, and cross-document synthesis.

Because MEMORY and EXECUTIVE communicate through natural language, the same trained memory can be reused across different LLMs, including closed-source models such as Gemini-3-Flash. Across BrowseComp-Plus, MuSiQue, and NarrativeQA, MeMo outperforms the evaluated parametric and latent-memory baselines, remains competitive with retrieval-based methods, and achieves a significant gain on MuSiQue.

Critical strengths

-True plug-and-play design. The same trained MEMORY model works with both Qwen2.5-32B-Instruct and Gemini-3-Flash. This cannot be the case for latent-memory approaches to achieve, because of different representations.

-Strong multi-hop reasoning performance. MeMo outperforms HippoRAG2 by 10.64 points on 4-hop MuSiQue questions.

Critical weaknesses
  • Ranking of benchmarked methods in accuracy results (Table 2) are very dependent on the EXECUTIVE model. For example if Gemini-3-Flash had not been included then HippoRAG2 would be best in 2/3 cases; then the introduction of Gemini-3-Flash, ranks MeMo best in 3/3 cases with massive relative improvements in NarrativeQA in particular.

  • Question-derived grouping can leak evaluation structure into training: Step 5 in generation, groups documents using each evaluation question's evidence set offering an advantage to MeMo.

  • Distractor robustness is not fully isolated. MeMo trails HippoRAG2 when no distractors are present, and increasing distractors also increases the size of MeMo's generated training set. This makes it difficult to attribute the flatter degradation curve solely to robustness.

  • The paper does not evaluate direct fine-tuning of the EXECUTIVE (or a single model) on the same reflection data. It is also not clear whether iterative retrieval baselines comparable to MeMo's multi-turn protocol are included in the benchmarks (assuming iteration offers critical advantage).

Questions for authors
  • How does MeMo perform when groups are constructed without access to question-level information (e.g., via clustering or retrieval-based grouping)?

  • What performance is obtained by fine-tuning the EXECUTIVE (or a single model) directly on the same "reflection" dataset?

7 comments
AuthorsOpenReview ↗

Response [1/5]

We thank the reviewer for a careful review. We appreciate the recognition of MeMo's true plug-and-play design, in which the same trained MEMORY model works with different executive models, something latent-memory approaches cannot achieve because of their different representations. We also appreciate the recognition of its strong multi-hop reasoning performance, outperforming HippoRAG2 by 10.64 points on 4-hop MuSiQue questions. We have grouped and addressed related weaknesses and questions as follows:

W1 Ranking of benchmarked methods in accuracy results (Table 2) are very dependent on the EXECUTIVE model. For example if Gemini-3-Flash had not been included then HippoRAG2 would be best in 2/3 cases; then the introduction of Gemini-3-Flash, ranks MeMo best in 3/3 cases with massive relative improvements in NarrativeQA in particular.

We acknowledge the reviewer's concern regarding how the EXECUTIVE model affects performance in MeMo and direct the reviewer to our full response to reviewer XL6C Q5/Q6. We provide additional results for 2 EXECUTIVE models, Qwen3.8-27B (no thinking) and Claude Haiku 5.5, showing that both the EXECUTIVE model's ability to access knowledge and its ability to make use of it are key contributors to overall performance. We also identify some possible contributing factors to Gemini's large improvement on NQA.

W3 Distractor robustness is not fully isolated. MeMo trails HippoRAG2 when no distractors are present, and increasing distractors also increases the size of MeMo's generated training set. This makes it difficult to attribute the flatter degradation curve solely to robustness.

We thank the reviewer for highlighting the potential influence of training set size on the observed robustness to noise. We agree that the increasing number of synthesized training examples may contribute to MeMo's robustness, but our current experiments do not isolate this effect. However, we believe that training set size should be considered alongside the underlying knowledge construction process. In MeMo, the increase in training set size arises naturally from its data synthesis pipeline as additional documents are introduced. Similarly, non-parametric methods such as HippoRAG2 also expand their constructed knowledge representations as additional documents are introduced. Beyond extracting additional knowledge triples, HippoRAG2 establishes connections between phrases across passages through shared entities and semantic similarity, potentially altering the graph structure and subsequent retrieval behavior.

Therefore, we view our experiments as an end-to-end evaluation of how different knowledge storage and retrieval approaches respond to increasing distractors, rather than an attempt to isolate the contribution of any single mechanism. While MeMo initially trails HippoRAG2 at 0N, it surpasses HippoRAG2 at 2N on BCP and at 1N on MSQ. Together with its flatter degradation curve, these results suggest that MeMo is less susceptible to performance degradation as distractors increase under the evaluated settings. We acknowledge that further controlled experiments would be needed to isolate the contribution of training-set size to the observed robustness, and leave that for future work.

We would also like to highlight that our response to reviewer XL6C (Q8) provides a more detailed analysis of why MeMo remains more robust to noise than the non-parametric methods.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [2/5]

W2 Question-derived grouping can leak evaluation structure into training: Step 5 in generation, groups documents using each evaluation question's evidence set offering an advantage to MeMo.

Q1 How does MeMo perform when groups are constructed without access to question-level information (e.g., via clustering or retrieval-based grouping)?

As mentioned in the paper (Sec. D.2 in the Appendix, Lines 1566-1567), we define the current grouping as the Upper Bound (UB) setting. This UB setting was initially selected to avoid the GENERATOR model processing the combinatorial space of document-group combinations. We explore 2 different clustering methods, (i) Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) and (ii) Entity-level grouping, to determine the difference between these methods and the upper bound. These methods are different from the post-hoc filter discussed in App D.1 and Tab 8.

For HDBSCAN, we embed each concatenated document, a combination of the document and its step 1-4 QA pairs, via NV-Embed-V2. The embeddings are reduced to 32 dimensions via Principal Component Analysis (PCA) and clustered with a minimum cluster size of 2. Notably, one advantage of this method is that there is no need to pre-specify the number of clusters, and documents in low-density regions can be classified as noise and left ungrouped rather than forced into a cluster. The clusters obtained from this process are used as the document groups in Step 5 of the data synthesis pipeline (cross-doc QA generation). Documents that were left ungrouped are not passed into Step 5.

For Entity grouping, we use the same concatenated document as in HDBSCAN and extract named-entity mentions. We index each entity by the documents that mention it and keep only entities with a document frequency of 2 or 3 to avoid common entities (e.g. United States). Each entity defines a group, with overlap allowed (a single document could belong to multiple groups). These entity groups are used as the document groups in Step 5. Documents that were left ungrouped are not passed into Step 5.

As the document groups from each method are built differently, we are able to take the union of the cross-document QA pairs generated from HDBSCAN groups and from Entity groups, yielding more diverse cross-document connections, where the near-duplicate of cross-doc QAs (>= 70% shared word trigrams) between both sets is approximately 3% and 8% for MSQ and BCP, respectively.

DatasetNum Non-cross-doc QAs (steps 1–4)Num Cross-doc QAs via HDBSCANCross-doc QAs via Entity groupingCross-doc QAs, totalTotal QAs
BCP 1N129,756624,490 (735 groups)2,916,945 (8,375 groups)3,541,4353,671,191
BCP 1N (UB)129,756--2,838,4582,968,214
MSQ 1N129,754956,101 (1,132 groups)1,091,424 (4,621 groups)2,047,5252,177,279
MSQ 1N (UB)129,754--535,008664,762

We train a MEMORY model for each of the 3 settings (HDBSCAN, Entity group, HDBSCAN + Entity group) and evaluate the checkpoints with the current pipeline. We report the performance of Qwen2.5-1.5B-Instruct for BCP and Qwen2.5-14B-Instruct for MSQ. This difference in MEMORY model size is due to time and resource constraints, as the BCP dataset has more QA pairs across the datasets, and fine-tuning Qwen2.5-14B-Instruct is still ongoing. The bootstrapping process for significance testing follows what was described in the paper in Sec. 5.

ModelDatasetMethodBest epochAccuracy (Mean ± std)
1.5BBCPUBep333.56 +- 0.77
1.5BBCPEntityep333.44 ± 1.84
1.5BBCPhdbscanep318.9 ± 0.5
1.5BBCPEntity + HDBSCANep333.78 ± 1.35
14BMSQUBep248.30 +- 1.25
14BMSQhdbscanep241.2 ± 1.8
14BMSQEntityep344.7 +- 1.00
14BMSQEntity + HDBSCANep346.68 +- 0.46
-MSQHippoRAG2-45.77 +- 0.49
-MSQNV-Embed-V2-40.00 +- 0.30
ModelDatasetMethodBest epoch2-hop Accuracy3-hop Accuracy4-hop Accuracy
14BMSQUBep253.60 +- 1.8346.41 +- 1.4435.34 +- 1.14
14BMSQHDBSCANep244.7 ± 1.240.1 ± 2.932.5 ± 2.6
14BMSQEntityep350.00+-1.8443.78+-0.6630.32 +- 0.92
14BMSQEntity + HDBSCANep350.19 +- 1.0747.83 +- 0.3733.53 +- 2.28
-MSQHippoRAG2-53.80 +- 0.3343.67 +- 1.4424.70 +- 0.49

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [3/5]

For MSQ, both HDBSCAN and Entity grouping perform significantly worse (p<0.05) than the UB performance, and do not beat the RAG baselines. However, when the 2 clustering methods are combined, the performance improves, and the difference between UB and Entity + HDBSCAN is not significant (p = 0.068). The breakdown by hop scores shows that the 4-hop performance continues to significantly outperform (p = 0.012) that of HippoRAG2.

For BCP, the results show that the individual clustering method of HDBSCAN performs worse than the UB. When Entity Grouping is used, the results are comparable to the UB, with the results within the noise of each other. The addition of HDBSCAN to Entity Grouping does not worsen the overall performance for BCP.

The observed performance indicates that Entity grouping outperforms the HDBSCAN clustering method on both BCP and MSQ, possibly for several reasons. One reason could be that part Step 5 of our data synthesis pipeline (cross-document synthesis) creates new QA pairs based on shared entities. Unlike embedding similarity, which does not guarantee that document groups share entities, the QA pairs generated by the Entity grouping method can capture more cross-document connections through shared entities. Furthermore, HDBSCAN leaves more documents ungrouped than Entity grouping (63% vs 95% on BCP), which further reduces the number of cross-document connections captured, resulting in a stark difference in the number of cross-doc BCP QA pairs generated. However, we acknowledge that the difference in QA pair counts may also contribute to the performance gap, where said effect is not isolated in the end-to-end evaluation of the grouping strategy. Hence, we do not claim that Entity grouping is always superior to HDBSCAN, and we leave a controlled comparison with matched QA pair counts for future work.

To conclude this question, while the UB does use question metadata to produce document groups, our preliminary results using HDBSCAN and Entity grouping to cluster and use these document groups show no significant difference (p=0.068) between the UB and the clustering combination of HDBSCAN and Entity grouping. As seen, different clustering methods can improve performance when combined; future work, such as exploring how different clustering methods affect one another, controlling for the number of training QA pairs, and deciding on methods when there is no ground truth to verify, is an exciting direction to consider. In the scope of the paper, the results currently presented in the paper remain applicable to the experimental setting investigated in this work.

W4 The paper does not evaluate direct fine-tuning of the EXECUTIVE (or a single model) on the same reflection data. It is also not clear whether iterative retrieval baselines comparable to MeMo's multi-turn protocol are included in the benchmarks (assuming iteration offers critical advantage).

Q2 What performance is obtained by fine-tuning the EXECUTIVE (or a single model) directly on the same "reflection" dataset?

For iterative retrieval baselines, we compare performance against 2 methods: IrCOT [1] and KiRAG [2]. We kept the EXECUTIVE (Qwen2.5-32B-Instruct) and retriever (NV-Embed-V2) the same as in other RAG experiments (HippoRAG2 and NV-Embed-V2). For IrCOT, which performs an initial retrieval and has the LLM reason to produce the next query, we set the total number of retrievable passages to 9, with up to 3 new passages per iteration. For KirAG, we extract the knowledge triples from corpus documents with the same GENERATOR (Qwen2.5-32B-Instruct). During retrieval via NV-embed-V2, the documents are retrieved, and their triplets are scored and ranked by the alginer (e5-large-v2) before the EXECUTIVE uses the top-ranked triples to decide the next query for retrieval. The answers from IrCOT and KiRAG are scored with the same LLM-as-a-Judge setup in the paper.

↓ ↓ ↓ Continued below ↓ ↓ ↓

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.