- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes MemState, a long-term memory framework for agents that keeps every successive state of a "matter" instead of overwriting it. At construction time (DMSG), an LLM attaches each new memory to a matter using both local nodes and a running chain summary, and labels relations as strong (state transition, kept acyclic) or weak (context). At query time (ASGR), initial hybrid retrieval results are grouped into per-matter units, a unit is flagged stale if its retrieved frontier has unseen strong-edge successors, and an agent issues up to four DAG_Search calls (forward/backward, latest/history/branch) to fill in the missing states. On RHELM, LoCoMo and LongMemEval-S the method roughly matches Zep/Graphiti, with 1 to 3 point gains on temporal and update subsets. On a separate state-focused evaluation it reports 76 CER and 85 history recall versus 65 and 72 for A-MEM, plus many construction ablations, robustness tests and cost accounting.
Critical strengths
Strength 1. The problem framing is useful. Separating "latest-only memory loses history" from "static retrieval misses later updates" (Figure 1) is a helpful way to think about memory for evolving matters, and the frontier/staleness definition in Eqs. (6) and (7) is simple and gives the agent a concrete signal for when to keep traversing.
Strength 2. The set of controls is well chosen in principle: a compute-matched multi-round text search (Table 16), a graph-structure by traversal factorial (Table 14), round-limit sweeps, edge and timestamp perturbations, and per-record construction cost. If these numbers hold up, the compute-matched comparison (82 vs 86 QA at the same 21K tokens) is the most convincing evidence that the graph matters and not just the extra agent calls.
Critical weaknesses
Weakness 1. The state-focused evaluation, which carries most of the claims, is not specified.** The headline numbers in the abstract (76 CER, 85 history recall, +11/+13 over A-MEM) come from Table 2, but I could not find where these questions come from. Section 4.1 says "state-focused QA uses a shared binary rubric, macro-averaged over query types and then over datasets", and Table 12 mentions "independently annotated supporting content", but there is no description of how gold latest/history evidence, transition paths and branches were annotated on RHELM/LoCoMo/LongMemEval-S, how many questions there are, or who annotated them. The same is true for the 32-hop chain test, the lexically-similar-matters test (1000 decisions) and the 100K-record scaling test. Without this I cannot judge whether the evaluation favours a system that builds exactly this kind of chain graph. I might be wrong here, but it looks like a custom benchmark was built around the method's own notion of matters and transitions.
Weakness 2. Baselines and statistics.** Zep/Graphiti is the strongest system in every native benchmark row (Table 1), and MemState only ties it on overall LoCoMo and LongMemEval-S accuracy (one point on RHELM). Yet Zep and MemU are absent from the state-evidence, trajectory and update-density tables, which is exactly where the claimed advantage lies. Zep is a temporal knowledge graph, so it is the natural comparator for "preserving state evolution". Also, three seeds are listed but every number is a rounded integer with no variance or significance test, so 1 to 3 point native-benchmark gains cannot be separated from noise. Hybrid RAG in Table 2 is literally MemState with zero rounds (identical rows in Tables 13 to 15), so it is not an independent baseline.
Other. The supplementary README states that the model-backed pipeline is not included and that the reported_results files are author-supplied summaries marked "independently_verified: false", with no per-question outputs or logs. So the numbers cannot be checked. Table 32's ASGR latency (36s p50, 80s p95, versus 10s for baselines) deserves more discussion than it gets.
Questions for authors
-
How was the state-focused QA set constructed (source dialogues, number of matters and questions per type, annotation procedure, annotator agreement on the gold evidence itself)? Was any of it written or filtered with the same LLM used in DMSG?
-
Can you report Zep/Graphiti (and MemU) on Tables 2, 11, 12 and 26 under the same reader and budget?
-
Please give per-seed results or confidence intervals for Table 1 and Table 2. With temperature 0, what exactly do the three seeds vary?
-
How does DMSG behave when the matter boundary is genuinely ambiguous in the raw conversation (e.g. LoCoMo, where matters are not pre-segmented)? Is component purity measured against gold matter labels on the native benchmarks or only on the constructed test sets?