acceptodds
Withdrawn from ICLR 2027

Connecting-the-Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure the reflective memory required to synthesize fragmented, multimodal cues into high-level interpretations. To address this gap, we introduce **RefMem-Bench**, a benchmark for reflective memory in long-horizon dialogue. **RefMem-Bench** contains 26K annotated QA instances with eight reflective-memory dimensions and three task formats, requiring models to move beyond surface-level retrieval and infer latent meanings from evidence distributed across interaction histories. To enhance reflective memory capability, we propose **RE**flective **M**emory **IND**uction (**REMIND**), a hierarchical framework that treats reflective memory as progressive meaning construction. **REMIND** couples question-conditioned evidence retrieval, salience-aware grounding, and abstraction-level supervision, and uses Progressive Reflective Alignment to distill high-level reflective reasoning into the factual inference pathway. Experiments show **RefMem-Bench** poses a substantial challenge to current models, while **REMIND** consistently improves both answer accuracy and memory recall through progressive evidence perception, grounding, and abstraction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, LLM memory, Memory Agents Benchmark, Evaluation for Memory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rT6sOpenReview ↗
Rating
1(−2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to help AI remember past conversations and understand patterns across them, rather than just recall facts.

It introduces RefMem-Bench to test this ability and REMIND, a method that helps AI connect past information and make better inferences.

The results show improved accuracy compared with the base model.

Critical strengths
  1. It focuses on helping AI connect information across past conversations, not just recall facts.
  2. RefMem-Bench evaluates deeper memory abilities, such as identifying patterns and understanding user preferences.
  3. REMIND improves the model’s ability to reason using past conversations.
  4. It shows clear accuracy improvements over the base model.
  5. Better conversational memory can help AI provide more personalised and context-aware responses.
Critical weaknesses
  1. The method is mainly evaluated on Qwen3-VL-8B, so it is unclear whether it works well with other models.
  2. The benchmark relies partly on automatically generated questions and annotations, which may introduce errors.
  3. Some questions require subjective interpretation, making it difficult to judge whether the model’s inferences are correct.
  4. The efficiency claims need stronger evidence, including end-to-end inference time and computational costs.
  5. The paper should compare methods more fairly by considering differences in model backbones, training data, and computational budgets. 6.The method may require additional training and preprocessing, which could make it difficult to use in real-world applications.
Questions for authors
  1. How does REMIND distinguish between factual memory retrieval and higher-level reflection?

  2. How reliable is RefMem-Bench if some questions and annotations are generated automatically?

  3. How do the authors verify that the model’s inferred user preferences or beliefs are actually correct?

  4. Why is the method evaluated mainly on Qwen3-VL-8B? Does it generalise to other models?

8 comments

Dear Reviewer rT6s,

We are deeply grateful for your valuable time, great suggestion and affirmation! Below are our detailed responses to your concerns:

W1 & Q4: Generalization Across Backbones

The method is mainly evaluated on Qwen3-VL-8B, so it is unclear whether it works well with other models. Why is the method evaluated mainly on Qwen3-VL-8B? Does it generalise to other models?

Thank you for raising this insightful question. Following prior work [1], we adopt the Qwen family as our primary open-source multimodal backbone for evaluating REMIND. To further examine its generalizability, we conducted additional experiments with two models from different families, InternVL3.5-8B and Gemma-4-E4B, using the same REMIND framework. The results are shown below:

MethodMulti-Choice AccMulti-Choice MemRSingle-Choice AccSingle-Choice MemRDirect-Answer AccDirect-Answer MemRDirect-Answer B1Direct-Answer F1
Base Model33.245.044.937.820.937.917.420.9
w/ NaiveRAG33.645.947.638.619.838.717.019.8
MemoryOS41.248.953.040.918.935.618.621.2
LangMem41.2-57.8-21.1-19.922.3
A-Mem46.348.557.444.222.939.616.418.5
LightMem48.348.456.944.123.239.920.123.2
GAM4746.256.441.423.236.819.923.0
MemAgent-7B25.040.653.543.912.529.712.114.2
MEM1-7B19.833.949.236.912.624.610.712.9
Qwen3-VL-30B36.346.058.240.523.540.419.522.4
DS-Distill-Qwen-14B24.643.854.945.719.139.115.418.8
REMIND (Qwen3-VL-8B)59.458.166.252.332.955.427.630.4
REMIND (InternVL3.5-8B)58.456.466.451.231.854.227.830.7
REMIND (Gemma4-E4B)58.552.670.050.130.851.527.429.4

As shown in table, REMIND achieves consistently strong performance across all three backbones. Notably, Gemma-4-E4B even surpasses Qwen3-VL-8B on Single-Choice accuracy by 3.8 points.These results demonstrate that REMIND maintains competitive performance across different model families, highlighting its generalizability in reflective-memory tasks.

We highly appreciate the constructive comments and insightful suggestions you have offered for our work. We will incorporate these additional experimental results into Appendix E of the revised manuscript.

References

[1] Yu et al. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent. ICLR 2026.

W2 & Q2: Reliability of Automatically Generated Questions and Annotations

The benchmark relies partly on automatically generated questions and annotations, which may introduce errors. How reliable is RefMem-Bench if some questions and annotations are generated automatically?

Thank you for raising this important concern. We appreciate the opportunity to further explain our annotation and quality-control procedures. RefMem-Bench employs LLMs for question generation, followed by independent verification of all generated questions, answers, and supporting evidence by three expert annotators. Specifically, annotators verify the correctness of answers and their supporting evidence against the original conversational histories, correct generation errors, and filter out ambiguous or low-quality questions with insufficient evidence. This process results in the removal of 8.31% of candidate questions, with inconsistent annotations further resolved through expert discussion.

To further quantify annotation reliability, we compute Fleiss' κ = 0.849 across the three annotators' initial judgments, indicating strong inter-annotator agreement. Detailed annotation procedures, quality-control criteria, and agreement analysis are provided in Appendices D.1–D.4, together with the annotation interfaces in Figures 7 and 8.

W3 & Q3: Reliability of Reflective Inference Evaluation

Some questions require subjective interpretation, making it difficult to judge whether the model's inferences are correct. How do the authors verify that the model's inferred user preferences or beliefs are actually correct?

Thank you for raising this insightful question. We would like to clarify that RefMem-Bench evaluates reflective inferences based on verifiable evidence from conversational histories, with expert annotators validating both the inferred answers and their supporting evidence. Questions with ambiguous interpretations or insufficient evidence are filtered during annotation, as detailed in our response to W2 & Q2.

To further ensure reliable evaluation, we adopt two complementary metrics: Answer Accuracy (Acc) and Memory Recall (MemR). Acc evaluates the correctness of model predictions against expert-verified reference answers, while MemR measures whether the model retrieves the supporting evidence required for its inference. For open-ended Direct-Answer questions, we additionally employ an LLM-as-a-Judge to assess semantic equivalence with acceptable reference answers. Its reliability has been verified through a meta-evaluation against expert human judgments, as reported in Appendix D.8, Table 3. Specifically, we evaluate 30 samples from each of the eight reflective-memory dimensions. The prompt-engineered GPT-4o judge achieves 99% average evaluation accuracy, compared with 93% for Qwen3-8B-Instruct, demonstrating strong agreement with human judgments.

W4 & W6: Computational Efficiency and Practical Applicability

The efficiency claims need stronger evidence, including end-to-end inference time and computational costs. The method may require additional training and preprocessing, which could make it difficult to use in real-world applications.

Thank you for raising such a professional and insightful question. To provide a more comprehensive evaluation of computational efficiency, we conducted additional experiments comparing LLM token consumption across different memory-processing stages and average per-query runtime. Specifically, we evaluate all methods on the Single-Choice task using Qwen3-VL-8B as the backbone, reporting LLM input/output tokens for segmentation and retrieval, memory construction, and final inference, together with the average processing time per question. The results are shown below:

MethodAccMemRSegment & Retrieval InSegment & Retrieval OutConstruction InConstruction OutInferenceRuntime (s)
Base Model44.937.8––––29,07425.8
A-Mem52.141.9173,78745,15711,6114,22510,38731.7
LightMEM52.042.028,77317,30098,0591,3242,86616.7
GAM50.142.0159,3194,04025,6231,06329,99224.7
MemAgent53.543.9––22,0411,37122,04153.2
MEM149.236.9––––4,62290.4
REMIND66.252.320,6293,608115752,55815.8

As observed, REMIND achieves the highest Acc (66.2%) and MemR (52.3%), while maintaining the lowest inference token consumption (2,558 tokens) and average runtime (15.8 seconds). Compared with Base Model, REMIND reduces inference token consumption by 91.2% and average runtime by 38.8%, while improving accuracy by 21.3 points. Furthermore, when accounting for LLM-based segmentation, retrieval, and memory construction, REMIND consumes 24,823 tokens in total, substantially fewer than A-Mem, LightMem, and GAM.

In real-world applications, however, reflective memory is primarily intended for offline reflection over accumulated factual memories, rather than immediate processing of individual interactions. For example, a long-term conversational assistant could periodically consolidate historical interactions to identify evolving user preferences, recurring behavioral patterns, and implicit needs, enabling more personalized responses in future conversations. In such scenarios, the additional preprocessing and training overhead can be amortized over long-term interactions, while the resulting reflective knowledge can be reused across subsequent queries. Considering the relatively low runtime and LLM token consumption demonstrated above, we believe that REMIND and future methods developed on RefMem-Bench hold considerable promise for practical long-term memory applications.

Thank you again for your highly professional question, it has provided us with valuable insights.

Reviewer_W28WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a benchmark that measures the reflective memory required to synthesize fragmented, multimodal cues into high-level interpretations. The paper also proposed a method named REMIND, a hierarchical framework that treats reflective memory as progressive meaning construction.

Critical strengths
  • The paper studies reflective memory in multi-modal domain, which to my knowledge is novel.
Critical weaknesses
  • The core contribution of the work is unclear. The paper title is about the benchmark, then the paper content talks about a method for reflective memory as well and the benchmark contribution, which is supposed to be the core of the paper, is not sufficiently novel and sophisticated. This paper seems like a combination two half-baked papers. The benchmark side is not complete because when a new benchmark is constructed much more comprehensive analysis of the data is needed to inform the community about various aspects of the benchmark. And when a method is introduced, it is wise to also evaluate on a number of different memory benchmarks to demonstrate the generalizability of the method. Another issue is that some part of the dataset construction is automated like clustering and sampling and hierarchical methods will benefit from such decisions. Therefore, it is not clear whether the obtained improvements are real or just a biproduct of the dataset construction method.
Questions for authors

None.

6 comments

Dear Reviewer W28W, We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns:

W1: Clarification on Contribution Positioning.

The core contribution of the work is unclear. The paper title is about the benchmark, then the paper content talks about a method for reflective memory as well and the benchmark contribution, which is supposed to be the core of the paper, is not sufficiently novel and sophisticated. This paper seems like a combination two half-baked papers.

Thank you for raising this important concern. We appreciate the opportunity to clarify the positioning and contributions of our work:

Our primary contribution is RefMem-Bench, a benchmark for evaluating reflective memory in long-horizon dialogue, while REMIND serves as a complementary methodological contribution to explore this capability. This benchmark–method formulation is also adopted in long-horizon research, such as BEAM [1] (ICLR 2026), AMA-Bench[2] (ICML 2026), and TRIP-Bench[3] (ICML 2026). Similarly, our work aims to establish reflective memory as a distinct evaluation target and explore how models can acquire this capability. Let's clarify these contributions from three perspectives: benchmark novelty, benchmark difficulty, and methodological exploration.

1. From Factual Recall to Reflective Memory

Existing memory benchmarks primarily focus on factual recall, evaluating whether models can retrieve explicitly stated information from conversational histories. In contrast, RefMem-Bench evaluates whether models can go beyond retrieving isolated facts and perform higher-level inference over distributed interaction histories, such as latent preferences, recurring behavioral patterns, evolving beliefs, and implicit constraints.

As already presented in Figure 2(b), we have systematically compared RefMem-Bench with representative dialogue resources and memory benchmarks. For clarity, we present the comparison below:

BenchmarkMMEAHA|TDPIBMEGPIBMCMLIPVBTSVP
Dialogue Resources
DailyDialog✗✗✓|✗✗✗✗✗✗✗✗
MMDialog✓✗✗|✗✗✗✗✗✗✗✗
MSC✗✗✓|✗✗✗✗✗✗✗✗
Conversation Chronicles✗✗✗|✗✗✗✗✗✗✗✗
Memory-Oriented Benchmarks
DialSim✗✓△|✓△✗✗✗✗✗✗
LoCoMo✓△✓|✓△✗✗✗✗✗✗
LongMemEval✗✓✓|✓✗✗✗✗✗✗✗
REALTALK✓✗✓|△✗✗✗✗△✗✗
MMRC✓✗✓|△✗✗✗✗△✗✗
Mem-Gallery✓✓✓|✓✗✗✗✗△✗✗
LifeBench✗✓△|✓△✓✗✗✗✗✗
RefMem-Bench✓✓✓|✓✓✓✓✓✓✓✓

Benchmark properties: MM = Multimodal; EA = Evidence-Anchored; HA = Human-Annotated. Reflective-memory dimensions: TD = Temporal Dynamics; PI = Pattern Induction; BM = Belief Modeling; EGP = Experience-Grounded Planning; IBM = Implicit Boundary Modeling; CMLI = Cross-Modal Latent Inference; PVB = Personalized Visual Baselines; TSVP = Topic-Scoped Visual Prototypes. ✓ = Full support; △ = Partial support; ✗ = Not supported. Values marked with * are derived from the original papers.

As observed, RefMem-Bench establishes eight reflective memory dimensions grounded in cognitive theories and task requirements, supported by evidence-based annotations and human verification. We believe this establishes a distinct and complementary evaluation target for long-term memory research, as illustrated by the comparison between factual and reflective memory in Figure 1.

2. Persistent Challenges for Frontier Models

To further assess the difficulty of RefMem-Bench, we conducted additional experiments with four larger or frontier models, including DeepSeek-V4.1-Flash, Claude-Haiku-5.5, GPT-5.6-Luna, and Qwen3-VL-30B. The results are shown below:

MethodMulti-Choice AccMulti-Choice MemRSingle-Choice AccSingle-Choice MemRDirect-Answer AccDirect-Answer MemRDirect-Answer B1Direct-Answer F1
DeepSeek V4 Flash56.553.272.352.23048.72428.5
Claude Haiku 5.539.753.164.74830.451.321.924.8
GPT-5.6 Luna55.954.572.349.232.449.425.629.5
Qwen3-VL-30B36.34658.240.523.540.419.522.4

As shown in table, RefMem-Bench remains challenging even for frontier models. DeepSeek-V4.1-Flash and GPT-5.6-Luna achieve only 56.5% and 55.9% accuracy on Multi-Choice, and 30.0% and 32.4% on Direct-Answer, respectively. These results demonstrate that RefMem-Bench remains a valuable benchmark for evaluating reflective-memory capabilities in frontier models, with substantial room for further improvement.

3. REMIND: A Methodological Exploration of Reflective Memory

Beyond establishing the benchmark, we introduce REMIND as an initial methodological exploration of reflective-memory capabilities. Our experiments demonstrate the effectiveness of this approach. Notably, REMIND achieves competitive performance using only an 8B backbone, with 59.4% Multi-Choice accuracy and 32.9% Direct-Answer accuracy, outperforming all evaluated baselines on both metrics. It also achieves the highest MemR among the evaluated models on these two tasks (58.1% and 55.4%, respectively).

Importantly, REMIND is intended as a methodological starting point for reflective-memory research, not a complete solution to the challenges introduced by RefMem-Bench. The remaining performance gap, even among frontier models, highlights substantial opportunities for future research. We hope RefMem-Bench can serve as a shared evaluation foundation and encourage the community to develop more effective methods for reflective memory.

Thank you for your valuable feedback and constructive suggestions.

References

[1] Tavakoli et al. Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs. ICLR 2026.

[2] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. ICML 2026.

[3] Shen et al. TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios. ICML 2026.

W2: Clarification on Benchmark Analysis

The benchmark side is not complete because when a new benchmark is constructed much more comprehensive analysis of the data is needed to inform the community about various aspects of the benchmark.

Thank you for raising this important concern. We would like to clarify a potential misunderstanding: Section 3 and Appendix D already provide a comprehensive description and analysis of RefMem-Bench. We are happy to summarize these details below for clarity:

Appendix D.1 details the data sources and construction pipeline, including dialogue segmentation, evidence clustering, question generation, and expert review; Appendices D.2-D.4 describe the annotation interfaces, annotation guidelines, quality control procedures, and inter-annotator agreement analysis; Appendix D.5 provides detailed definitions and examples for all eight reflective-memory dimensions; Appendix D.6 reports dataset statistics, including conversation characteristics, task distributions, and data splits; and Appendices D.7-D.8 present the evaluation protocols and reliability analysis of the evaluation metrics.

We sincerely appreciate the reviewer for highlighting this important concern. We will further highlight and clarify the relevant analyses already provided in Appendix D in our revised manuscript to make these details more accessible to readers.

W3: Lack of Multi-Benchmark Evaluation for REMIND

And when a method is introduced, it is wise to also evaluate on a number of different memory benchmarks to demonstrate the generalizability of the method.

Thank you very much for your insightful feedback. To demonstrate the generalization of the REMIND method, We additionally add experiments using the the LongMemEval bench (specifically the LongMemEval-S split), and the results are as follows.

Baseline model: Qwen3-30B-A3B-Instruct-2507, judge model: gpt-4o-mini.

MethodSingle-UserSingle-AssistantTemporalMulti-SessionSingle-PreferenceKnowledge-Update
Base Model82.8687.5033.0835.6150.0076.92
LangMem78.5742.8637.6038.3570.0067.95
A-MEM90.0096.4351.8851.1240.0076.93
MemoryOS72.8692.8628.5736.8433.3361.54
Mem055.3226.0941.9428.1381.8228.57
Light-mem80.0031.2554.2051.9180.0066.67
REMIND(Ours)93.7596.4339.3746.2883.3383.33

As observed, REMIND achieves state-of-the-art performance on several dimensions. In particular, REMIND achieves strong results on Single-User, Single-Preference, and Knowledge-Update, which indicate that it learns transferable reflective-memory capabilities rather than benchmark-specific patterns.

Reviewer_LrPAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a benchmark of 26K question-answer designed to evaluate reflective memory in long-horizon conversations, where models must infer implicit user preferences, behavioral patterns, and latent states by integrating evidence scattered across multiple interactions. In addition, it introduces a strong baseline (REMIND), which is an hierarchical framework that combines question-conditioned evidence retrieval, salience-guided memory grounding, and reflective abstraction. Experiments on the benchmark show substantial improvements over existing conversational memory SOTA methods.

Critical strengths

The research line proposed on the paper follows previous work and expand them focusing on inferring recurring behaviors, latent preferences, and implicit constraints. This can be relevant and of interest as it gets closer to real-world problems. The benchmark itself, while is built from existing benchmarks, it represent a valuable resource for the community as it allows to experiment with new problems.

Critical weaknesses

The paper contributions would benefit from further experimental validation.While the benchmark itself represents a potentially valuable resource, my main concern is with REMIND. The method works on this benchmark, but it is unclear how it would generalize to other datasets. Another concern is on the number of distinct conversational histories. Although the number of examples originated from them is ok, the source remains limited and the paper does not discuss implications of this, if any (e.g. does it limit diversity?).

Questions for authors
  • What does it prevent you to apply REMIND to other datasets?
  • Based on my understanding, train/test split is at the evidence-cluster level rather than the conversation level. Does it mean that training and test examples may share underlying conversational histories?
  • The results are based on an 8B model, are the challenges of the benchmark present with larger models as well?
6 comments

Dear Reviewer LrPA,

We are deeply grateful for your valuable time and constructive feedback! Below are our detailed responses to your concerns:

W1 & Q1: Limited Cross-Dataset Generalization of REMIND.

my main concern is with REMIND. The method works on this benchmark, but it is unclear how it would generalize to other datasets. What does it prevent you to apply REMIND to other datasets?"

Thank you for your valuable suggestion. To assess cross-benchmark generalization, we evaluate REMIND on LongMemEval, one of the most widely used long-term memory benchmarks, using its LongMemEval-S split. We use exactly the same configuration as in our main experiments, without any further tuning. The results are shown below:

Baseline model: Qwen3-30B-A3B-Instruct-2507, judge model: gpt-4o-mini.

MethodSingle-UserSingle-AssistantTemporalMulti-SessionSingle-PreferenceKnowledge-Update
Base Model82.8687.5033.0835.6150.0076.92
LangMem78.5742.8637.6038.3570.0067.95
A-MEM90.0096.4351.8851.1240.0076.93
MemoryOS72.8692.8628.5736.8433.3361.54
Mem055.3226.0941.9428.1381.8228.57
Light-mem80.0031.2554.2051.9180.0066.67
REMIND(Ours)93.7596.4339.3746.2883.3383.33

As observed, REMIND achieves state-of-the-art performance on several dimensions. In particular, REMIND achieves strong results on Single-User, Single-Preference, and Knowledge-Update, which indicate that it learns transferable reflective-memory capabilities rather than benchmark-specific patterns.

W2: Data Source Diversity Analysis

Although the number of examples originated from them is ok, the source remains limited and the paper does not discuss implications of this, if any (e.g. does it limit diversity?).

Thank you for raising this insightful question. As shown in Figure 2(b), RefMem-Bench contains 35 long-horizon conversational histories, averaging 38.31 sessions per history and 52.99 turns per session. We further organize fine-grained topical segments into evidence clusters to capture distinct semantic contexts within these histories.

Inspired by [1], we conducted an additional cluster-level similarity analysis using all-MiniLM-L6-v2 to assess the semantic diversity of questions generated from different evidence clusters:

SourceClustersAvg. Questions per ClusterMeanMedianMax≥0.80 (%)≥0.85 (%)≥0.90 (%)
LoCoMo3764.250.69630.71620.866110.244.960
DialSim6234.310.71030.72850.865111.226.460
REALTALK2894.760.62410.63870.84397.2700
Overall12884.390.68690.70270.866110.054.570

As shown above, RefMem-Bench contains 1,288 evidence clusters, with an average of 4.39 questions per cluster. The overall mean cosine similarity is 0.6869, with only 4.57% of comparisons exceeding 0.85, and none exceeding 0.90. Similar patterns are observed across all three sources. These results indicate that the underlying conversational histories contain diverse semantic evidence rather than highly repetitive content, despite the limited number of source histories.

References

[1] Wu et al. Mapping Overlaps in Benchmarks through Perplexity in the Wild. ICLR 2026.

W3 & Q3: Benchmark Difficulty Beyond 8B Models

The results are based on an 8B model, are the challenges of the benchmark present with larger models as well?

Thank you for this constructive suggestion. We conduct additional experiments with four larger or frontier models, including DeepSeek-V4.1-Flash, Claude-Haiku-5.5, GPT-5.6-Luna, and Qwen3-VL-30B. The results are shown below:

MethodMulti-Choice AccMulti-Choice MemRSingle-Choice AccSingle-Choice MemRDirect-Answer AccDirect-Answer MemRDirect-Answer B1Direct-Answer F1
REMIND (Ours)59.458.166.252.332.955.427.630.4
DeepSeek V4 Flash56.553.272.352.230.048.724.028.5
Claude Haiku 5.539.753.164.748.030.451.321.924.8
GPT-5.6 Luna55.954.572.349.232.449.425.629.5
Qwen3-VL-30B36.346.058.240.523.540.419.522.4
GAM47.046.256.441.423.236.819.923
LightMem48.348.456.944.123.239.920.123.2
MemAgent-7B25.040.653.543.912.529.712.114.2

As shown in table, RefMem-Bench remains challenging even for frontier models. DeepSeek-V4.1-Flash and GPT-5.6-Luna achieve only 56.5% and 55.9% accuracy on Multi-Choice, and 30.0% and 32.4% on Direct-Answer, respectively. These results demonstrate that RefMem-Bench remains a valuable benchmark for evaluating reflective-memory capabilities in frontier models.

Notably, REMIND achieves competitive performance using only an 8B backbone, with 59.4% Multi-Choice accuracy and 32.9% Direct-Answer accuracy, outperforming these larger models on both metrics. It also achieves the highest MemR on both tasks (58.1% and 55.4%), further highlighting the effectiveness of REMIND in enhancing reflective-memory capabilities.

We highly appreciate the constructive comments and insightful suggestions you have offered for our work. We will incorporate these additional experimental results into Appendix E of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.