- Rating
- 1(−2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper aims to help AI remember past conversations and understand patterns across them, rather than just recall facts.
It introduces RefMem-Bench to test this ability and REMIND, a method that helps AI connect past information and make better inferences.
The results show improved accuracy compared with the base model.
Critical strengths
- It focuses on helping AI connect information across past conversations, not just recall facts.
- RefMem-Bench evaluates deeper memory abilities, such as identifying patterns and understanding user preferences.
- REMIND improves the model’s ability to reason using past conversations.
- It shows clear accuracy improvements over the base model.
- Better conversational memory can help AI provide more personalised and context-aware responses.
Critical weaknesses
- The method is mainly evaluated on Qwen3-VL-8B, so it is unclear whether it works well with other models.
- The benchmark relies partly on automatically generated questions and annotations, which may introduce errors.
- Some questions require subjective interpretation, making it difficult to judge whether the model’s inferences are correct.
- The efficiency claims need stronger evidence, including end-to-end inference time and computational costs.
- The paper should compare methods more fairly by considering differences in model backbones, training data, and computational budgets. 6.The method may require additional training and preprocessing, which could make it difficult to use in real-world applications.
Questions for authors
-
How does REMIND distinguish between factual memory retrieval and higher-level reflection?
-
How reliable is RefMem-Bench if some questions and annotations are generated automatically?
-
How do the authors verify that the model’s inferred user preferences or beliefs are actually correct?
-
Why is the method evaluated mainly on Qwen3-VL-8B? Does it generalise to other models?
8 comments
W2 & Q2: Reliability of Automatically Generated Questions and Annotations
The benchmark relies partly on automatically generated questions and annotations, which may introduce errors. How reliable is RefMem-Bench if some questions and annotations are generated automatically?
Thank you for raising this important concern. We appreciate the opportunity to further explain our annotation and quality-control procedures. RefMem-Bench employs LLMs for question generation, followed by independent verification of all generated questions, answers, and supporting evidence by three expert annotators. Specifically, annotators verify the correctness of answers and their supporting evidence against the original conversational histories, correct generation errors, and filter out ambiguous or low-quality questions with insufficient evidence. This process results in the removal of 8.31% of candidate questions, with inconsistent annotations further resolved through expert discussion.
To further quantify annotation reliability, we compute Fleiss' κ = 0.849 across the three annotators' initial judgments, indicating strong inter-annotator agreement. Detailed annotation procedures, quality-control criteria, and agreement analysis are provided in Appendices D.1–D.4, together with the annotation interfaces in Figures 7 and 8.
W3 & Q3: Reliability of Reflective Inference Evaluation
Some questions require subjective interpretation, making it difficult to judge whether the model's inferences are correct. How do the authors verify that the model's inferred user preferences or beliefs are actually correct?
Thank you for raising this insightful question. We would like to clarify that RefMem-Bench evaluates reflective inferences based on verifiable evidence from conversational histories, with expert annotators validating both the inferred answers and their supporting evidence. Questions with ambiguous interpretations or insufficient evidence are filtered during annotation, as detailed in our response to W2 & Q2.
To further ensure reliable evaluation, we adopt two complementary metrics: Answer Accuracy (Acc) and Memory Recall (MemR). Acc evaluates the correctness of model predictions against expert-verified reference answers, while MemR measures whether the model retrieves the supporting evidence required for its inference. For open-ended Direct-Answer questions, we additionally employ an LLM-as-a-Judge to assess semantic equivalence with acceptable reference answers. Its reliability has been verified through a meta-evaluation against expert human judgments, as reported in Appendix D.8, Table 3. Specifically, we evaluate 30 samples from each of the eight reflective-memory dimensions. The prompt-engineered GPT-4o judge achieves 99% average evaluation accuracy, compared with 93% for Qwen3-8B-Instruct, demonstrating strong agreement with human judgments.
W4 & W6: Computational Efficiency and Practical Applicability
The efficiency claims need stronger evidence, including end-to-end inference time and computational costs. The method may require additional training and preprocessing, which could make it difficult to use in real-world applications.
Thank you for raising such a professional and insightful question. To provide a more comprehensive evaluation of computational efficiency, we conducted additional experiments comparing LLM token consumption across different memory-processing stages and average per-query runtime. Specifically, we evaluate all methods on the Single-Choice task using Qwen3-VL-8B as the backbone, reporting LLM input/output tokens for segmentation and retrieval, memory construction, and final inference, together with the average processing time per question. The results are shown below:
| Method | Acc | MemR | Segment & Retrieval In | Segment & Retrieval Out | Construction In | Construction Out | Inference | Runtime (s) |
|---|---|---|---|---|---|---|---|---|
| Base Model | 44.9 | 37.8 | – | – | – | – | 29,074 | 25.8 |
| A-Mem | 52.1 | 41.9 | 173,787 | 45,157 | 11,611 | 4,225 | 10,387 | 31.7 |
| LightMEM | 52.0 | 42.0 | 28,773 | 17,300 | 98,059 | 1,324 | 2,866 | 16.7 |
| GAM | 50.1 | 42.0 | 159,319 | 4,040 | 25,623 | 1,063 | 29,992 | 24.7 |
| MemAgent | 53.5 | 43.9 | – | – | 22,041 | 1,371 | 22,041 | 53.2 |
| MEM1 | 49.2 | 36.9 | – | – | – | – | 4,622 | 90.4 |
| REMIND | 66.2 | 52.3 | 20,629 | 3,608 | 11 | 575 | 2,558 | 15.8 |
As observed, REMIND achieves the highest Acc (66.2%) and MemR (52.3%), while maintaining the lowest inference token consumption (2,558 tokens) and average runtime (15.8 seconds). Compared with Base Model, REMIND reduces inference token consumption by 91.2% and average runtime by 38.8%, while improving accuracy by 21.3 points. Furthermore, when accounting for LLM-based segmentation, retrieval, and memory construction, REMIND consumes 24,823 tokens in total, substantially fewer than A-Mem, LightMem, and GAM.
In real-world applications, however, reflective memory is primarily intended for offline reflection over accumulated factual memories, rather than immediate processing of individual interactions. For example, a long-term conversational assistant could periodically consolidate historical interactions to identify evolving user preferences, recurring behavioral patterns, and implicit needs, enabling more personalized responses in future conversations. In such scenarios, the additional preprocessing and training overhead can be amortized over long-term interactions, while the resulting reflective knowledge can be reused across subsequent queries. Considering the relatively low runtime and LLM token consumption demonstrated above, we believe that REMIND and future methods developed on RefMem-Bench hold considerable promise for practical long-term memory applications.
Thank you again for your highly professional question, it has provided us with valuable insights.
Dear Reviewer rT6s,
We are deeply grateful for your valuable time, great suggestion and affirmation! Below are our detailed responses to your concerns:
W1 & Q4: Generalization Across Backbones
Thank you for raising this insightful question. Following prior work [1], we adopt the Qwen family as our primary open-source multimodal backbone for evaluating REMIND. To further examine its generalizability, we conducted additional experiments with two models from different families,
InternVL3.5-8BandGemma-4-E4B, using the same REMIND framework. The results are shown below:As shown in table, REMIND achieves consistently strong performance across all three backbones. Notably, Gemma-4-E4B even surpasses Qwen3-VL-8B on Single-Choice accuracy by 3.8 points.These results demonstrate that REMIND maintains competitive performance across different model families, highlighting its generalizability in reflective-memory tasks.
We highly appreciate the constructive comments and insightful suggestions you have offered for our work. We will incorporate these additional experimental results into Appendix E of the revised manuscript.
References
[1] Yu et al. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent. ICLR 2026.