- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ScenarioModelMem. It separates memory modeling into a fixed structure and a configurable memory policy, which is automatically optimized through an LLM-guided evolutionary algorithm involving mutation, crossover, and selection. Experiments on LOCOMO and PersonaMem demonstrate improvements over existing memory systems.
Critical strengths
-
The paper provides a structured approach to memory management by separating a fixed Entity Property Time representation from a configurable memory policy. This design enables memory extraction and organization to be customized for different scenarios, offering a potentially useful direction for improving the flexibility of long-term memory systems.
-
ScenarioModelMem achieves strong results on two memory benchmarks, LOCOMO and PersonaMem. The experiments also show that evolutionary policy optimization can modestly outperform manually designed policies, providing preliminary evidence that automatic memory configuration can be beneficial.
Critical weaknesses
-
Limited novelty. The paper largely combines existing techniques, including Entity Property Time representations, LLM-based memory extraction, conventional retrieval methods, and evolutionary optimization. It fails to clearly distinguish its contributions from prior work on temporal knowledge graphs, memory schema optimization, and evolutionary search.
-
The improvements over EverMemOS are relatively small (+0.98 on LOCOMO and +3.06 on PersonaMem). More importantly, these results rely on manually designed policies rather than the proposed evolution mechanism. Policy evolution is evaluated only on PersonaMem, yielding a modest +1.53 point improvement over the strongest human-designed policy. Can you evaluate the policy evolution method on more benchmarks?
-
The framework introduces multiple expensive components, including LLM-based memory extraction, entity fusion, multi-round retrieval, and evolutionary optimization. It remains unclear whether the modest accuracy gains justify the additional complexity.
-
The manuscript also suffers from excessive whitespace, further weakening its presentation quality.
Questions for authors
-
Can the authors evaluate the proposed evolution method on additional benchmarks?
-
Can the authors provide a detailed ablation analysis of the framework's design?
-
Can the authors improve the manuscript formatting, particularly the excessive whitespace and figure layout?