acceptodds
Under review as a conference paper at ICLR 2027

Memorilla: Latent Semantic Memory for LLMs

Abstract

Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108× fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLMs, Memory, Modality, Agents, RAG

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_REeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MEMORILLA, a latent semantic memory system for frozen LLMs. The main idea is to first encode all documents into reusable embeddings. For each query, several query-conditioned memory slots attend to the whole document embedding collection and generate a fixed number of soft memory tokens.

Critical strengths

The problem is important and practical. Long-term memory is important for personal assistants, long-document QA, and agents. Reducing the context size while keeping useful information is a meaningful problem.

The method is simple and clear. The architecture is easy to understand. The document embeddings can be stored once, and the query-dependent memory tokens can be generated for different queries.

Critical weaknesses
  1. The comparison with some baselines may not be fully fair because MEMORILLA uses downstream task training. This is my main concern. MEMORILLA has three training stages. In Stage 3, the authors directly fine-tune the memory module on the training splits of all eight evaluation benchmarks. PersonaMem-v2 and PersonalizationV4 also receive additional benchmark-specific fine-tuning. However, several important baselines, such as RAG, Full Context, Mem0, and MemoRAG, do not receive the same task-specific supervised training.

  2. The scalability to a very large memory collection is still unclear. MEMORILLA always gives only 16 memory tokens to the LLM, which makes the decoder context very small. However, the memory module still attends to all document embeddings for every query. Therefore, the memory access cost still increases with the number of documents. The current efficiency experiment only tests collections up to 512 documents.

  3. The “continual learning” claim may be too strong. In the TextWorld experiment, the memory module is trained with GRPO before evaluation. During evaluation, the model parameters are fixed, and new trajectory information is only added into the external document store.

Questions for authors

Please see above.

1 comment

Dear Reviewer REeN,

Thank you again for your review. As the discussion period ends tomorrow, we wanted to kindly check whether our responses have addressed your concerns. In brief, we added leave-one-benchmark-out experiments for all eight benchmarks, a matched LoRA control trained on the same supervision, scaling results up to 4.2M documents, and we have dropped the term continual learning. If these address your concerns, we would be grateful if you could consider reflecting this in your evaluation. We are happy to answer any remaining questions.

Best regards, The Authors

Reviewer_W851OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Memorilla, an external memory module for frozen LLMs. Each document chunk is encoded into one vector by a frozen Qwen3-Embedding-4B; 16 learned slots are first conditioned on the question vector, then cross-attend to all chunk vectors (a Perceiver-style read), and are projected into 16 soft tokens placed before the prompt of a frozen Qwen3-8B. Only the memory module is trained, in three stages, and the last stage (Stage 3) uses the training sets of all eight evaluation benchmarks: personalization (PersonaMem-v2 or PMv2, the authors' synthetic PersonalizationV4 or PV4, LaMP-4, LaMP-7), long-document QA (NarrativeQA, TriviaQA), and knowledge-intensive QA (FactKG, PubMedQA). Memorilla has the best average score (49.68) with 16 tokens and claims to beat full context and RAG, and a separate GRPO-trained variant improves a frozen Qwen3-4B agent on TextWorld games.

Critical strengths
  1. The architecture is simple, and letting the query shape the memory slots before reading the whole document collection is a natural way to avoid the top-k selection problem of RAG.

  2. Some of the analysis is informative. The breakdown by question type, the complementarity with RAG, and the noise-document control help explain when a compressed latent memory works, which is useful beyond this specific model.

Critical weaknesses
  1. The central claim, that a 16-token latent memory can beat both reading the full documents and text RAG methods (for example, "full context is not an upper bound"), is not well supported. I see some separate problems.
  • Task supervision is not controlled. The decoder is frozen in every method, but Memorilla's memory module (about 194M parameters) is trained in Stage 3 on the training sets of all eight evaluation benchmarks, while RAG, Full Context, Mem0, MemoRAG, and LLMLingua-2 receive no task-specific training. The 16 soft tokens can therefore also work as a learned task prompt that carries each benchmark's answer format, label distribution, or style, as in prompt tuning. Consistent with this, xRAG, the only other trained baseline, is also the strongest baseline on PMv2 and LaMP-7. A matched control is needed, for example Full Context or RAG with a soft prompt trained on the same data for the same frozen 8B decoder. The no-document LoRA control in Appendix E.4 goes in this direction, but it uses a 4B decoder and no documents, so it is not matched with the main model.
  • The only trained latent baseline is handicapped. xRAG follows its original design and maps only the top-1 retrieved chunk to one token, while Memorilla reads the whole collection into 16 tokens. The comparison therefore mixes the memory design with a 16 times larger token budget and access to every chunk. A matched variant, for example the top-16 chunks mapped to 16 tokens, should be reported on all benchmarks, beyond the cosine top-16 ablation on PMv2 in Table 13.
  • The full-context reference is weak. The largest decoder is Qwen3-8B, whose native window is 32K. Full context on PV4 (about 50K tokens) always needs YaRN, part of PMv2, TriviaQA, and NarrativeQA does too, and 10.8% of NarrativeQA books are still truncated. The low full-context scores may therefore reflect the limited long-context ability of an 8B model, and no larger open model (32B or above) or strong long-context API model is evaluated as a reference.
  • There is no account of why the method should win. The memory passes through several lossy steps (one embedding per chunk, one query vector, and 16 slots), yet it is claimed to beat the full text. Section 5 only offers a post-hoc explanation that compression keeps global structure, and this explanation is not tested. Together, these leave it open whether the gain over full context and text RAG comes from the memory design itself or from task supervision, an under-powered latent baseline, and a weak long-context reader.
  1. The module is close to the BLIP-2 Q-Former and the Perceiver Resampler used in vision-language models, but this link is not discussed in the method or related work sections and only appears as an ablation in Appendix E.1, where the Q-Former variant is already close to the full model (43.34 vs. 44.04 on PMv2), so the novelty of the architecture should be stated more carefully.
  2. In TextWorld, the memory module is trained from scratch with GRPO (8 latent tokens, the full text history kept in the prompt) and raises avg@3 from 0.186 to 0.242 over full-history in-context learning on 40 test games. The untrained-memory control rules out the effect of adding tokens but not the effect of a learned fixed prefix, so a trajectory-independent soft prompt trained by GRPO is needed, and calling this continual learning is not accurate since the module is not updated at test time.
  3. Efficiency is measured mainly by the number of document tokens, and Table 14 compares latency only with Full Context (HF generate without a paged KV cache) and not with RAG, while the encoding of every chunk with a 4B embedding model (about 2.6 s for 512 chunks) is counted as free because it is shared with dense retrieval.

Other issues that matter less for my rating:

  • All experiments pair a Qwen3 encoder with a Qwen3 decoder, and the memory module must be retrained for each LLM, so it is not shown whether the method works for other model families.
  • Appendix E.1 (Table 13 and its text) and Figure 9 report different numbers for what is described as the same setting, a 4B decoder trained from scratch with K=16 (PMv2 44.04 vs. 45.10, FactKG 82.56 vs. 81.87).
Questions for authors
  1. How does Memorilla perform when Stage 3 excludes the target benchmark (leave-one-benchmark-out)? This would show how much of the gain depends on in-domain supervision.
  2. What is the training cost of the three stages (GPU hours), given that every step back-propagates through the frozen 8B decoder?
2 comments

Dear Reviewer W851,

Thank you again for your detailed review. As the discussion period ends tomorrow, we wanted to kindly check whether our responses have addressed your concerns. In brief, we added a matched LoRA control on the same Qwen3-8B decoder, a budget-matched xRAG with 16 tokens, full-context results with Qwen3-30B and its native 256K window, a document scaling analysis, vLLM latency against RAG, training cost, and leave-one-benchmark-out results. If these address your concerns, we would be grateful if you could consider reflecting this in your evaluation. We are happy to answer any remaining questions.

Best regards, The Authors

My concerns were properly addressed. I maintain my positive score

Reviewer_qsZ9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MEMORILLA, a latent memory system for frozen LLMs. A frozen embedding model encodes each document chunk once into a persistent store. At query time, a trainable Perceiver-style module first conditions K=16 learned slots on the query embedding, then cross-attends from these slots over all stored embeddings. It projects the result into the decoder’s input space as soft tokens. The module is trained with a next-token prediction loss on answer tokens, backpropagated through the frozen LLM, under a three-stage curriculum. Across personalization, long-document QA and knowledge-intensive QA, MEMORILLA leads on five of eight benchmarks and outperforms full-context inference on seven while using far fewer document tokens. A second experiment trains the module from scratch with GRPO in TextWorld, where the memory supplements the agent’s full trajectory.

Critical strengths

The approach is sensible. The idea is incremental in nature, as the architecture largely adapts existing Perceiver/Q-Former-style query-conditioned cross-attention to document embeddings, but I find it interesting and of practical value. The memory store does not depend on the LLM and reuses the embeddings already computed for a dense-retrieval index. The decoder receives a fixed 16-token budget regardless of collection size, and both backbones stay frozen, which simplifies deployment and per-user memory management. The evaluation covers a broad set of benchmarks against a reasonably diverse set of baselines. The paper is clearly written and easy to follow.

Critical weaknesses
  1. The main comparison conflates the benefit of memory with supervised task adaptation. MEMORILLA is fine-tuned on the training split of every evaluation benchmark, whereas all baselines except xRAG are training-free. Gains over full context or RAG may therefore reflect task fitting rather than better use of the documents. Trained text-context baselines, such as LoRA on the same decoder with RAG-k or full context in the prompt, are needed to isolate the contribution of memory. The same issue weakens the generalization claim. The paper states that MEMORILLA handles unseen collections without further optimization, yet every evaluation benchmark is included in Stage 3 training. A zero-shot evaluation on a held-out task would support the claim far more convincingly.

  2. The evaluation would be stronger with results on established long-term memory benchmarks such as LongMemEval and LoCoMo. Both benchmarks are standard for conversational memory systems and are used by the agentic memory baselines the paper compares against, such as Mem0. They also test abilities the current suite covers only partly, including temporal reasoning, knowledge updates and multi-session synthesis. Results on them would make the comparison with prior memory systems more direct. If they are excluded from training, they would also serve as the held-out evaluation the generalization claim currently lacks.

  3. The technical novelty is moderate. Query-conditioned latent slots that cross-attend over frozen features and are projected into a frozen LLM closely follow the Perceiver and Q-Former designs. As noted above, I still find the application to persistent memory interesting and practically relevant. The contribution, however, lies mainly in the problem setup and empirical study rather than in the architecture.

Questions for authors

Please address the weaknesses listed above.

1 comment

Dear Reviewer qsZ9,

Thank you again for your thoughtful review. As the discussion period ends tomorrow, we wanted to kindly check whether our responses have addressed your concerns. In brief, we added a LoRA baseline with RAG-50 trained on the same data and parameter budget, ran held-out zero-shot evaluations on LongMemEval and LoCoMo, and clarified our claims on generalization and architectural novelty. We would be grateful for any further feedback and are happy to answer remaining questions before the deadline.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.