- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents graphmem, a durable, multi-agent graph memory system implementing A-Mem's Zettelkasten-style memory evolution. Its dual-engine architecture uses RustFS as the sole source of truth and FalkorDB as a rebuildable graph and vector index. A stateless coordinator supports shared access, while membership checks prevent links outside the retrieved neighbor set. Experiments compare With Evolution, No Evolution, No Memory, and A-Mem's reference implementation on LoCoMo, LongMemEval, and BEAM. Results show that memory evolution does not consistently outperform simpler retrieval.
Critical strengths
- The paper addresses a practical systems problem. Its dual-engine architecture supports durable, queryable, and shareable agent memory. Real graph edges and membership checks strengthen structural correctness and help prevent hallucinated links.
- The paper evaluates both answer quality and system behavior. It measures memory evolution costs, storage latency and throughput, and fault behavior. The authors also run A-Mem's reference implementation and include a Qdrant baseline to examine concurrency performance.
- The authors transparently report negative findings and limitations. They acknowledge asymmetric benchmark coverage, embedding-model differences, and the lack of repeated-seed runs. Additional analyses include paired statistical tests, multiple-comparison corrections, content-analysis metadata checks, 60 fault-injection trials, and two-agent concurrent memory insertion.
Critical weaknesses
- Some comparisons and statistical analyses weaken the conclusions. Table 1 compares BEAM's 1,600 No Evolution questions against 272 With Evolution questions, but Appendix E reports a matched difference of −0.0062 (No Evolution − With Evolution), reversing the apparent ordering. Although this difference is not significant, the headline comparison may be misleading. LoCoMo's 1,986 questions also come from only 10 conversations, so question-level bootstrapping does not account for within-conversation dependence. The matched BEAM comparison should be emphasized in the main results, and conversation-level cluster bootstrapping would strengthen the statistical conclusions.
- A-Mem ablation fidelity remains unclear. A-Mem's original "w/o ME" retains Link Generation, whereas graphmem's No Evolution excludes linked-neighbor retrieval expansion; embedding inputs also differ. Appendix N identifies missing seeding-time content-analysis metadata in the main With Evolution configuration. Restoring it improves LongMemEval F1 by 0.0382, narrowing the gap with No Evolution from 0.0430 to approximately 0.0048. Although the change is not statistically significant, the main conclusions should be reassessed by directly comparing the corrected With Evolution configuration against No Evolution.
- Concurrent revisions and fault recovery remain under-tested. Per-operation atomicity in RustFS and FalkorDB does not guarantee consistency across the complete memory evolution write path. Appendix L tests two-agent insertion of 20 memories, but not conflicting revisions to shared neighbor notes. Appendix K's Point B precedes the post-decision re-upsert/re-write pair, leaving failures between these revision writes untested. Conflict handling and recovery mechanisms require further validation.
- Evaluation coverage and end-to-end performance evidence remain limited. LongMemEval covers 58/500 questions, while BEAM's With Evolution covers 17/100 conversations, with truncation at the 10M scale. The thinking-mode comparison also changes generation-length caps and request timeouts, complicating attribution. Mem0 receives only a pilot-scale QA evaluation, while Graphiti is assessed mainly through ingestion costs. Storage-throughput tests exclude LLM costs and the complete memory evolution workflow. More controlled end-to-end comparisons are needed.
Questions for authors
See weaknesses.