- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ReplayMem preserves original conversations and combines dual-view indexing, semantic expansion, and temporal neighbors to retrieve evidence for answering questions in a single LLM call. Experiments suggest competitive performance across three conversational memory benchmarks.
Critical strengths
ReplayMem presents a clear and practical design for long-term conversational memory. Preserving original conversations keeps source evidence available, while dual-view indexing and iterative expansion help retrieve information distributed across the history. The answering stage requires only a single LLM call, without continually maintaining a global memory state. The reported reduction in memory-construction token costs also makes the approach potentially useful in practical applications.
Critical weaknesses
Major Concern 1: Evaluation consistency remains unclear. For memory baselines reported for LoCoMo and LongMemEval-S, scores are nearly identical to those in EverMemOS (Hu et al., 2026) under different evaluation settings ( averaged three judges for EverMemOS vs. GPT-4.1-mini judge for ReplayMem). The authors should clarify whether these results were reused or reevaluated under identical protocols. Without this clarification, the comparisons do not adequately support the SOTA claim.
Major Concern 2: Single-session preference understanding remains a weakness. Despite its strong overall performance, ReplayMem falls noticeably behind several baselines on SS-Pref in Table 1. The paper does not explain whether this gap arises from missing preference evidence during retrieval or inadequate interpretation during replay. A targeted failure analysis would clarify this limitation.
Major Concern 3: The benefit of state reconstruction is not isolated. The prompt in Appendix A.2.1 mainly asks the model to extract relevant information and reason step by step. Although Table 5 fixes the retrieved evidence, it lacks a standard step-by-step reasoning baseline. Thus, the improvement may reflect general reasoning instructions rather than a distinct state reconstruction mechanism.
Minor Concern 1: The supplied artifact is incomplete. Configuration files, installation metadata, and raw experimental logs are missing from the provided package. These artifacts should be provided for better reproducibility.
[1] Hu, C., Gao, X., Zhou, Z., et al. (2026). EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning. arXiv:2601.02163.
Questions for authors
See above. My questions mainly revolve around the major concerns. I understand reusing statistics due to consumption restrictions, but I do require clarifications on the experimental setting. I am willing to improve my score if most of my concerns are resolved.