acceptodds
Under review as a conference paper at ICLR 2027

Customize Your Memory: Scenario-Adaptive and Evolvable Memory Modeling for LLM Agents

Abstract

Memory enables large language model (LLM) agents to learn continually across sessions and scenarios. Memory systems have attracted great attention as LLM agents are widely used in coding, office work, and other scenarios. However, memory modeling, i.e. how to represent flat text and what features to extract, is either ignored or hand-crafted and frozen at design time. We argue that memory is a process of identification, extraction, and synthesis, i.e. an offline cognitive capability whose core is memory modeling instead of a flat archive. We formulate the memory system as a modeling-guided optimization problem and present ScenarioModelMem, a scenario-adaptive memory-modeling system. We split memory modeling into a static memory structure and a dynamic memory policy to achieve efficient representation, flexible configuration, and optimizability. A validation-driven evolutionary algorithm refines the dynamic memory policy through error-informed mutation, crossover, and pruning. Under the same experimental configuration as the strongest published systems, ScenarioModelMem achieves state-of-the-art performance on LOCOMO (94.03) and PersonaMem (70.63) with a pre-defined memory policy. Furthermore, we evaluate automatic policy design on PersonaMem. Starting from a basic initial policy, the evolved policy outperforms a carefully crafted human-designed one. Memory modeling thus becomes scenario-adaptive and optimizable, rather than a fixed design choice.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agents, Memory System, Self-evolution, Memory Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_TMNhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ScenarioModelMem. It separates memory modeling into a fixed structure and a configurable memory policy, which is automatically optimized through an LLM-guided evolutionary algorithm involving mutation, crossover, and selection. Experiments on LOCOMO and PersonaMem demonstrate improvements over existing memory systems.

Critical strengths
  1. The paper provides a structured approach to memory management by separating a fixed Entity Property Time representation from a configurable memory policy. This design enables memory extraction and organization to be customized for different scenarios, offering a potentially useful direction for improving the flexibility of long-term memory systems.

  2. ScenarioModelMem achieves strong results on two memory benchmarks, LOCOMO and PersonaMem. The experiments also show that evolutionary policy optimization can modestly outperform manually designed policies, providing preliminary evidence that automatic memory configuration can be beneficial.

Critical weaknesses
  1. Limited novelty. The paper largely combines existing techniques, including Entity Property Time representations, LLM-based memory extraction, conventional retrieval methods, and evolutionary optimization. It fails to clearly distinguish its contributions from prior work on temporal knowledge graphs, memory schema optimization, and evolutionary search.

  2. The improvements over EverMemOS are relatively small (+0.98 on LOCOMO and +3.06 on PersonaMem). More importantly, these results rely on manually designed policies rather than the proposed evolution mechanism. Policy evolution is evaluated only on PersonaMem, yielding a modest +1.53 point improvement over the strongest human-designed policy. Can you evaluate the policy evolution method on more benchmarks?

  3. The framework introduces multiple expensive components, including LLM-based memory extraction, entity fusion, multi-round retrieval, and evolutionary optimization. It remains unclear whether the modest accuracy gains justify the additional complexity.

  4. The manuscript also suffers from excessive whitespace, further weakening its presentation quality.

Questions for authors
  1. Can the authors evaluate the proposed evolution method on additional benchmarks?

  2. Can the authors provide a detailed ablation analysis of the framework's design?

  3. Can the authors improve the manuscript formatting, particularly the excessive whitespace and figure layout?

Reviewer_LzdROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a 3D memory store (entity, property, time) with a declarative “policy” manifest, agentic search, and an evolutionary algorithm that automatically designs the policy. It claims SOTA scores of 94.03 on LOCOMO and 70.63 on PersonaMem, saying that the evolving policy of the method could beat the human designed score.

Critical strengths

The description of the static structure and the dynamic decomposition of the method is clear.

The result looks quite superior to the current SOTA methods.

The appendix demonstration looks informative to some extent.

The LOCOMO ablation and retrieval-truncation study are useful.

Critical weaknesses

The algorithm and motivation figure look AI-generated. The tables and the layout of the main text and appendix are poorly organized, and the writing reads like that of a beginner.

Novelty is overclaimed. “First to split memory modeling into structure and policy” is overclaimed as the EvolveMem, which the paper cites clearly does this work.

The paper mentions LLM-driven evolution methods such as EvoPrompt and ADAS. The directly relevant systems A-Mem, MIRIX, and EvolveMem are also discussed; however, a direct comparison with these systems is not presented.

Cost is not reported. There is no token or latency cost for memory generation, and no cost for the 24 generations × 10 individuals of evolution, which the paper itself calls a limitation.

The ablation is pretty thin and confounded. Only two components get tested. On top of that, “w/o 3D structure” also drops the policy-guided generation pipeline, so the effect can’t really be attributed to one thing. And search-field augmentation, entity fusion, higher-order properties, and temporal expansion are never ablated.

The LOCOMO margin is also small: +0.98 and +1.23 overall, with Open-domain and Multi-hop below EverMemOS under gpt-4o-mini. And the baseline numbers are just copied from another paper’s protocol.

Questions for authors

Can you report multiple evolution seeds with CIs?

Does the evolved policy transfer to LOCOMO or to gpt-4.1-mini?

How would the baselines do with a benchmark-specific schema?

I believe the paper falls well below the ICLR standard in terms of novelty, experimental design, writing quality, and figures. I would encourage the authors to substantially rework the paper; overall, I do not think it currently meets the ICLR bar.

Reviewer_VMbDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ScenarioModelMem, a memory system that combines an entity, property, and time representation with scenario-specific memory policies and evolutionary optimization to improve long-term memory for LLM agents.

Critical strengths

The paper presents an interesting perspective on making memory modeling optimizable, with a clear system design and competitive results on LOCOMO and PersonaMem.

Critical weaknesses

Indirect optimization signal for memory modeling. The proposed policy evolution relies on downstream QA accuracy, but answering errors may originate from memory generation, retrieval, or reasoning. It is unclear whether the error-informed mutation mechanism can accurately identify deficiencies in memory modeling. Moreover, optimizing for a fixed set of queries may overlook information that could be valuable for future tasks. Limited adaptability of the fixed memory representation. The proposed entity, property, and time structure is suitable for factual and temporal information, but its effectiveness for procedural knowledge, agent experiences, and complex causal relationships remains unclear. Although such information could be represented through entities and properties, it is not evident that this structure can organize and retrieve these memory types effectively. Limited expressiveness of the policy optimization space. The policy optimization primarily focuses on entity types and property definitions, while other important memory modeling decisions, such as episode segmentation, entity resolution, and information consolidation, remain fixed. This may limit the framework's ability to address memory modeling errors arising from these components. Insufficient reliability guarantees for higher-order memory synthesis. The system synthesizes higher-order properties from multiple observations, but it is unclear how the reliability of these inferred attributes is assessed or how they are grounded in supporting evidence. Incorrect abstractions may become persistent memories and affect subsequent retrieval and generation. Insufficient analysis of computational efficiency. The proposed system involves multiple LLM calls for memory generation, entity fusion, higher-order property synthesis, and iterative retrieval, which may introduce substantial computational overhead. However, the paper does not provide a detailed analysis of inference latency, token consumption, or memory construction costs compared with existing systems. Without such analysis, it is difficult to assess whether the reported accuracy improvements justify the additional computational complexity.

Questions for authors

Please refer to the questions I raised in the "weakness" section.

Reviewer_8ni3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ScenarioModelMem, a memory system that separates a fixed entity–property–time representation from a configurable memory policy specifying what information to extract and synthesize. The system combines policy-guided memory generation, entity fusion, temporal tracking, and dynamic retrieval. It also introduces an offline evolutionary procedure that optimizes memory policies using labeled fitting examples. With predefined policies, the system reports overall accuracies of 94.03% on LOCOMO and 70.63% on PersonaMem. In a separate PersonaMem setting, the evolved policy achieves higher test accuracy and lower storage usage than the evaluated human-designed policies.

Critical strengths
  1. The paper addresses a meaningful problem: deciding what an agent should remember, rather than treating memory extraction and organization as fixed design choices. Making the memory policy explicit and inspectable is useful for adapting a system to different applications.
  2. The architecture is coherent. Property-level histories, higher-order attribute synthesis, and bidirectional retrieval provide complementary mechanisms for organizing and accessing conversational information. The appendix includes algorithmic descriptions, evolutionary hyperparameters, and concrete policy examples.
  3. The empirical results are promising. The predefined-policy system improves overall accuracy over the strongest reported baseline on both benchmarks. In the evolution experiment, the evolved policy reduces stored text from 64.13k to 42.21k characters per scope relative to the human-designed PersonaMem policy, approximately a 34% reduction, while improving test accuracy.
Critical weaknesses
  1. The evidence for the superiority of automatic policy evolution is limited. In Figure 5, the evolved policy achieves 75.32% test accuracy versus 73.79% for the strongest human-designed policy. On 393 questions, this corresponds to a net improvement of six correct answers. No paired uncertainty estimates or variability across evolutionary runs are reported, making the robustness of this improvement difficult to assess. The scope-level split also does not establish whether fitting and test personas are disjoint. Furthermore, this experiment changes the answer model, disables fallback memory, and introduces book distractors. Although the policies are compared under matched conditions within this experiment, the results do not establish the benefit of evolution under the main evaluation setting.
  2. The ablations do not sufficiently isolate the proposed mechanisms. In Section 4.2, removing the 3D structure also removes detailed policy guidance and changes the representation to flat episodic memory. Consequently, the resulting performance decrease cannot distinguish the contribution of the storage structure from that of policy-guided extraction. Similarly, the dynamic-search ablation simultaneously removes planning, iterative retrieval, and reverse retrieval. The attribution of PersonaMem improvements to higher-order properties is also not supported by a dedicated ablation.
  3. The temporal visibility protocol for PersonaMem requires clarification. The official benchmark implementation restricts each question to the context prefix specified by end_index_in_shared_context. The paper describes constructing a store from a scope’s contexts and subsequently answering its questions, but does not explain how these question-specific cutoffs are enforced. This matters especially for entity fusion and higher-order synthesis: filtering retrieved records by timestamp would not necessarily prevent information from later messages from entering an earlier entity representation. This is an unresolved protocol question, rather than evidence that leakage occurred.
  4. Generalization across backbone models remains insufficiently evaluated. The main benchmark comparisons use GPT-4o-mini and GPT-4.1-mini, leaving it unclear whether the improvements over competing memory systems persist with newer proprietary models or open-weight models. Although Section 4.3 uses GPT-5.5, that experiment compares policy configurations within ScenarioModelMem under a modified evaluation setting, rather than comparing memory systems. A controlled comparison against the strongest baseline using a newer-generation backbone and a representative open-weight model would help establish whether the benefits extend beyond the evaluated GPT-4-family backbones.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.