acceptodds
Under review as a conference paper at ICLR 2027

Graphmem: Durable, Multi-agent Graph Memory for LLM Agents

Abstract

A growing family of systems gives LLM agents long-term memory by letting an LLM curate a structured note graph over time, tagging, linking, and revising entries as experience arrives, rather than using a static vector index (Xuet al., 2025; Park et al., 2023; Gutiérrez et al., 2024; Rasmussen et al., 2025). These designs report large gains in small, validation-scale evaluations. Two questions remain open: whether this curation can run durably as first-class, auditable structure, and whether its benefits hold at real scale under statistical scrutiny. This study examines both questions with graphmem, a durable, concurrency- safe graph memory architecture built on RustFS (durable storage) and FalkorDB (a graph database with a native vector index), instantiating A-Mem’s (Xu et al., 2025) Zettelkasten-style evolution loop while making every link a real, traversable edge that removes an id-hallucination failure mode present in A-Mem’s own reference implementation. Using graphmem’s reproducible harness, this paper reports a full-scale, statistically tested evaluation of this evolution paradigm across three long-conversation QA benchmarks (LoCoMo (Maharana et al., 2024), LongMemEval (Wu et al., 2025), BEAM (Tavakoli et al., 2026)). Memory beats no memory on nearly every well-powered comparison, but evolution’s edge over a simpler retrieval-only ablation reaches significance on only one of three benchmarks, and a category-level claim about its cross-session value does not survive scaling up or a controlled reasoning-mode toggle. Running A-Mem’s own code directly, not only its published numbers, yields the same pattern: its simplest configuration’s apparent edge over graphmem is also not statistically significant. A separate, LLM-cost-free storage comparison isolates the architecture’s own contribution: FalkorDB’s throughput increases under concurrent load, where A-Mem’s in-process store’s throughput decreases under the identical load pattern.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agent memory, Large language models, Long-term memory, memory evolution, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X4wEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents graphmem, a durable, multi-agent graph memory system implementing A-Mem's Zettelkasten-style memory evolution. Its dual-engine architecture uses RustFS as the sole source of truth and FalkorDB as a rebuildable graph and vector index. A stateless coordinator supports shared access, while membership checks prevent links outside the retrieved neighbor set. Experiments compare With Evolution, No Evolution, No Memory, and A-Mem's reference implementation on LoCoMo, LongMemEval, and BEAM. Results show that memory evolution does not consistently outperform simpler retrieval.

Critical strengths
  1. The paper addresses a practical systems problem. Its dual-engine architecture supports durable, queryable, and shareable agent memory. Real graph edges and membership checks strengthen structural correctness and help prevent hallucinated links.
  2. The paper evaluates both answer quality and system behavior. It measures memory evolution costs, storage latency and throughput, and fault behavior. The authors also run A-Mem's reference implementation and include a Qdrant baseline to examine concurrency performance.
  3. The authors transparently report negative findings and limitations. They acknowledge asymmetric benchmark coverage, embedding-model differences, and the lack of repeated-seed runs. Additional analyses include paired statistical tests, multiple-comparison corrections, content-analysis metadata checks, 60 fault-injection trials, and two-agent concurrent memory insertion.
Critical weaknesses
  1. Some comparisons and statistical analyses weaken the conclusions. Table 1 compares BEAM's 1,600 No Evolution questions against 272 With Evolution questions, but Appendix E reports a matched difference of −0.0062 (No Evolution − With Evolution), reversing the apparent ordering. Although this difference is not significant, the headline comparison may be misleading. LoCoMo's 1,986 questions also come from only 10 conversations, so question-level bootstrapping does not account for within-conversation dependence. The matched BEAM comparison should be emphasized in the main results, and conversation-level cluster bootstrapping would strengthen the statistical conclusions.
  2. A-Mem ablation fidelity remains unclear. A-Mem's original "w/o ME" retains Link Generation, whereas graphmem's No Evolution excludes linked-neighbor retrieval expansion; embedding inputs also differ. Appendix N identifies missing seeding-time content-analysis metadata in the main With Evolution configuration. Restoring it improves LongMemEval F1 by 0.0382, narrowing the gap with No Evolution from 0.0430 to approximately 0.0048. Although the change is not statistically significant, the main conclusions should be reassessed by directly comparing the corrected With Evolution configuration against No Evolution.
  3. Concurrent revisions and fault recovery remain under-tested. Per-operation atomicity in RustFS and FalkorDB does not guarantee consistency across the complete memory evolution write path. Appendix L tests two-agent insertion of 20 memories, but not conflicting revisions to shared neighbor notes. Appendix K's Point B precedes the post-decision re-upsert/re-write pair, leaving failures between these revision writes untested. Conflict handling and recovery mechanisms require further validation.
  4. Evaluation coverage and end-to-end performance evidence remain limited. LongMemEval covers 58/500 questions, while BEAM's With Evolution covers 17/100 conversations, with truncation at the 10M scale. The thinking-mode comparison also changes generation-length caps and request timeouts, complicating attribution. Mem0 receives only a pilot-scale QA evaluation, while Graphiti is assessed mainly through ingestion costs. Storage-throughput tests exclude LLM costs and the complete memory evolution workflow. More controlled end-to-end comparisons are needed.
Questions for authors

See weaknesses.

Reviewer_n7T7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents graphmem, a durable, concurrency-safe graph memory architecture for LLM agents that re-implements A-Mem's Zettelkasten-style memory evolution loop on a dual-engine storage stack: the S3-compatible object store RustFS as the sole system of record, and the graph database FalkorDB as a rebuildable query and vector index. Every curated link becomes a real graph edge filtered against the neighbor set actually shown to the LLM, structurally eliminating an id-hallucination bug the authors identify in A-Mem's reference implementation. The paper then runs a full-scale, statistically tested evaluation on three long-conversation QA benchmarks, namely LoCoMo, LongMemEval, and BEAM, using paired bootstrap confidence intervals, permutation tests, and multiple-comparisons correction. The findings are largely negative but informative: memory beats no memory wherever the test is well powered, yet evolution's edge over a retrieval-only ablation reaches significance on only one of three benchmarks, a cross-session category advantage does not survive scale-up or a thinking-mode toggle, and A-Mem's own code reproduces the same non-significant pattern. A storage-level benchmark shows FalkorDB's throughput rising under concurrent load where A-Mem's in-process store degrades.

Critical strengths
  1. Motivation: The two research questions are well chosen and timely. Agent memory is an active area in which prominent systems report large gains from small validation-scale evaluations, and this paper asks the two questions the field needs answered: whether LLM-curated memory structure can be made durable, auditable, and shareable as first-class system state, and whether the reported benefits survive real scale and statistical scrutiny. The concretely documented id-hallucination bug in A-Mem's reference implementation, where list positions are reused to index a differently ordered list so a link can point to a memory the model never saw, gives the durability question a sharp correctness motivation, and Appendix M measures that 2 of 1,240 real evolution proposals would have become exactly such hallucinated links.

  2. Method design: The architecture is sound and clearly described. Separating a durable source of truth, RustFS, from a rebuildable query index, FalkorDB, is the right systems decomposition, with a defensible write-ordering argument that a crash can only leave the index behind the store, never ahead of it. The membership check in Eq. 2 is a simple structural guarantee that no link can reference a memory outside the retrieved neighbor set, and the authors correctly scope it as a guarantee about where a mistaken decision can land, not about decision quality. Idempotent writes, per-backend graph isolation for embedding comparability, and a stateless coordinator enabling multi-agent sharing are all sensible, and the design goals of durability, structural correctness, shareability, and comparability are stated crisply in Section 3.1.

  3. Experiments: The evaluation methodology is the paper's strongest asset and sets a standard this area badly needs. LoCoMo is run at genuine full scale with 1,986 questions and five variants; every headline comparison carries a paired bootstrap 95 percent confidence interval and a permutation-test p-value from 10,000 resamples, with Benjamini-Hochberg correction across the 15 headline tests and explicit verification that question-level pairing is exact. Coverage shortfalls are quantified and cost-explained in Appendix D rather than hidden, cost projections are audited against actual wall time in Table 6, durability is verified by 60 fault-injection trials rather than argued from code, concurrency correctness is tested with two agents on independent LLM backends, the storage comparison includes a Qdrant control to separate client-server effects from graph-specific ones, and the thinking-mode toggle in Section 4.4 is a genuinely informative confound analysis. The willingness to report, prominently, that the paper's own system loses its own ablation is exemplary scientific honesty.

Critical weaknesses
  1. Limited novelty, and a system result that undermines itself. The system contribution is a competent integration of existing components, namely RustFS, FalkorDB, HNSW indexing, and A-Mem's unmodified evolution policy, so the research novelty rests almost entirely on the evaluation rather than the method. More damaging, the paper's own system is dominated by its simplest ablation: No Evolution has a higher point-estimate F1 than With Evolution on all three benchmarks in Table 1, significantly so on LoCoMo with p below 0.0001, and A-Mem's cheapest configuration, A-Mem Storage, has a higher point estimate than every graphmem variant on LoCoMo and LongMemEval. The durability and concurrency advantages are real, but they are properties a deployment could obtain by pairing a retrieval-only memory with the same storage layer, which is essentially what the paper's own numbers recommend.

  2. Systems claims are validated only at modest scale. The concurrency correctness test in Appendix L uses 2 agents and 20 notes in a single run; the fault-injection study in Appendix K interrupts execution at only two points over 60 trials; the latency and throughput tables stop at corpus size 2,000 and 8 threads; RustFS and FalkorDB run as single instances. The words durable and multi-agent in the title are thus supported by demonstration rather than stress testing, and the end-to-end suite in Appendix A had not been run at the time of writing. For a paper whose first contribution is a storage architecture, the scale of the systems evidence is thin relative to the scale of the QA evidence.

Questions for authors

Please refer to the weaknesses.

Reviewer_qi3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a durable and concurrency-safe graph memory system. With this system, this paper conducts a full-scale, statistically-tested evaluation of this evolution paradigm across three long-conversation QA benchmarks. The experiments shows that memory boost performance significantly across three long-conversation QA benchmarks, but evolution’s edge over a simpler retrieval-only ablation reaches significance on only one of three benchmarks.

Critical strengths
  1. The proposed graph mem system is well designed and scalable, with high-throughput.
  2. The large scale eval is strict and persuasive.
Critical weaknesses
  1. The experiment setting is mainly long qa rather than an real-world agent setting.
Questions for authors
  1. Is it possible to test the memory system on long agent tasks like edge bench, terminal bench v4, and rsi bench?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.