- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MEMORILLA, a latent semantic memory system for frozen LLMs. The main idea is to first encode all documents into reusable embeddings. For each query, several query-conditioned memory slots attend to the whole document embedding collection and generate a fixed number of soft memory tokens.
Critical strengths
The problem is important and practical. Long-term memory is important for personal assistants, long-document QA, and agents. Reducing the context size while keeping useful information is a meaningful problem.
The method is simple and clear. The architecture is easy to understand. The document embeddings can be stored once, and the query-dependent memory tokens can be generated for different queries.
Critical weaknesses
-
The comparison with some baselines may not be fully fair because MEMORILLA uses downstream task training. This is my main concern. MEMORILLA has three training stages. In Stage 3, the authors directly fine-tune the memory module on the training splits of all eight evaluation benchmarks. PersonaMem-v2 and PersonalizationV4 also receive additional benchmark-specific fine-tuning. However, several important baselines, such as RAG, Full Context, Mem0, and MemoRAG, do not receive the same task-specific supervised training.
-
The scalability to a very large memory collection is still unclear. MEMORILLA always gives only 16 memory tokens to the LLM, which makes the decoder context very small. However, the memory module still attends to all document embeddings for every query. Therefore, the memory access cost still increases with the number of documents. The current efficiency experiment only tests collections up to 512 documents.
-
The “continual learning” claim may be too strong. In the TextWorld experiment, the memory module is trained with GRPO before evaluation. During evaluation, the model parameters are fixed, and new trajectory information is only added into the external document store.
Questions for authors
Please see above.
Dear Reviewer REeN,
Thank you again for your review. As the discussion period ends tomorrow, we wanted to kindly check whether our responses have addressed your concerns. In brief, we added leave-one-benchmark-out experiments for all eight benchmarks, a matched LoRA control trained on the same supervision, scaling results up to 4.2M documents, and we have dropped the term continual learning. If these address your concerns, we would be grateful if you could consider reflecting this in your evaluation. We are happy to answer any remaining questions.
Best regards, The Authors