- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a method for analyzing where memory system errors occur when evaluating on existing memory benchmarks. The work begins by proposing a formal definition of "sufficiency," a measurement of which memory operations are actually needed to solve a task. It then proposes a mechanism where one can hold everything fixed and ablate only individual memory operations in order to attribute credit. Specifically, the authors propose a mechanism where instead of providing the retrieved memories to the answer agent, the framework builds a graph over the memories, clustered by subject/predicate, and uses this as context. If the answer agent is able to get this right, the error can be attributed to a retrieval failure, otherwise it can be attributed to a write failure. The results on MemoryAgentBench show that different memory systems exhibit different failure modes; some fail to extract the facts in the first place, while some fail to retrieve them later on.
Critical strengths
None
Critical weaknesses
- The paper is extremely difficult to read; many terms and variables are frequently introduced without being defined or explained, making the method and contribution hard to follow. The empirical results and metrics are also disorganized and scattered throughout the paper in long paragraphs, rather than being organized into charts or tables, which makes them difficult to compare and understand. It is also unclear how the formal definition of sufficiency (a measurement of the minimal number of operations needed to answer a question) in Equation 1 is used empirically or what the high-level motivation for this formalism is.
- The novelty of the work is unclear; the paper does not clearly position itself against prior works nor provide sufficient motivation for its high level ideas and algorithms. It's unclear how the V/C transform algorithm is different from existing works that dump memories into context to analyze the errors, nor how the broader implications of this paper for memory systems are different from prior works.
- The method does not seem very general and only applies to particular types of memory benchmarks; in particular the V/C transform groups memories by subject/predicate; the V step also removes all memories that are older than the most recent one for a given topic, which may not be desirable for all memory tasks.
Questions for authors
- How does the diagnosis from the V/C transform method differ from dumping the entire memory into the context of the agent? It seems that both would tell us whether or not the memory relevant to the query being answered exists in the memory bank
- Are there any broader implications of the results? Prior works have already established that different memory systems and models can exhibit different failure modes/behaviors, what new insights does this work provide?
- The paper states in Section 8 that "earlier split-epoch estimate attenuates in the complete replication (11/58 changed arm outcomes; 8.62 pp interaction shift)" which I interpret as meaning 11/58 outcomes changed between identical runs. Could the authors provide confidence intervals or quantify the run-to-run variance? With almost 20% of the decisions flipping from run-to-run, it is currently difficult to trust that any of the single-seed results are statistically significant.
2 comments
Thank you for your additional experiments and clarifications. The author response seems to concede critical points such as a lack of novelty, significant run variability ("the concern is supported"), and that the method does not work over simpler alternatives ("Full MemReq did not improve repair selection over Store+Trace"); given these, I maintain my original score.
Clarifications on Failure Attribution, Dump-All Controls, Novelty, Generality, and Run Variability
We thank the reviewer for the detailed assessment. We revised the explanation of MemReq and added Dump-All controls, cross-history tests, human evidence audits, and repeated-run measurements.
1. Attribution and Dump-All (Q1). MemReq does not label every failed transformed answer as a write failure, or every successful one as a retrieval failure. It examines Store, Graph, and Reader Context evidence; V/C transforms test the response to changing readout, with UNKNOWN retained when support cannot be established.
Direct native-Store Dump-All controls are now in Appendix X (Tables 47–49). On the original MAB Stores, Qwen9 scores 5/58 with the full A-MEM dump versus 47/58 with Strict-V+C; for Additive, 5/58 versus 39/58. On five shorter histories, Full Dump versus V-dense scores 39/60 versus 52/60 (Additive), 50/60 versus 58/60 (A-MEM), and 55/60 versus 53/60 (RV-generic). Thus neither readout is universally better; lengths also differ.
The audit identifies a more specific boundary: 60/60 Stores and 45/60 Graphs contain complete support, but only 4/60 original Graph Contexts do. A gold-free intervention, frozen before five additional histories, raises complete Context support from 16/60 to 48/60 and accuracy from 12/60 under the original strict Graph readout to 47/60 (Table 4). Random seeding scores 33/60; V-dense remains higher at 52/60, with 37/60 seeded Contexts byte-identical to V-dense. This tests evidence delivery, not Graph superiority.
2. Novelty and broader implications (Q2/W2). Version selection and graph traversal are not new operators. Related Work now discusses Reddy and Challaram (2026), the concurrent Contract Memory Compiler (Song et al., 2026), and HippoRAG. MemReq instead links controlled operation responses to prerequisite formation and observed Store-to-Context evidence loss. Across five histories, changing only RV's extraction prompt raises literal update opportunities from 1/60 to 58/60 and native-readout accuracy from 7/60 to 52/60. With fixed dense readout, version filtering helps Additive by 38.33 pp and A-MEM by 43.33 pp, but hardly changes the RV variants (Appendix T). Consequently, a poor conflict-resolution score alone cannot establish that an update primitive failed.
3. Scope of V/C (W3). Strict-V/C uses MAB surface grammar and visible ingestion order; it is not validated for arbitrary natural-language memories. Latest-only filtering can also discard history needed for past-state questions. GPT-OSS scores 56/56 on SH6k with either AllVersion or LatestGlobal (a ceiling-bound result), so V is a tested operation, not a universal default. The prospective histories contain only 48–56 facts each. Their evaluated questions and full source chains are new, but 18–20 fact instances and 9–10 of 12 distinct MH answer strings per history overlap prior MQuAKE-T panels (Table 38). This is same-family, not cross-domain, validation.
4. Run variability (Q3). The concern is supported. Across five GPT-OSS runs, 40/232 arm–query cells (17.2%) vary under server defaults; 24/232 (10.3%) vary under a separate temperature-zero/seed-request contract. Most GPT-OSS runs fail strict completeness. In contrast, only 1/232 cells (0.4%) varies across three Qwen9 runs, whose V×C interactions range from +32.76 to +34.48 pp (Appendix Y, Table 50). Primary cross-history and prospective findings use Qwen9. Single-run GPT-OSS estimates are descriptive; model identity and reasoning settings are not isolated.
5. Equation 1 and presentation (W1). Equation 1 defines point-estimate sufficiency relative to an empirical reference and tolerance, with an inclusion-minimal frontier only among tested configurations. On CrossEp-Emb/Qwen9, the prior matches the reference at 75/121; retrieval scores 68/121. At a 5 pp margin, only the prior meets the point-estimate criterion; this is not established paired non-inferiority. We acknowledge the presentation problems; we aligned RQ wording, clarified prior-work distinctions, and consolidated prospective results in Table 4.
We retain the negative result that Full MemReq did not improve repair selection over Store+Trace (488/555 versus 486/555). The supported contribution is stage-specific diagnosis and bounded intervention testing, not general repair-policy superiority.
We welcome further questions.
Locations: Sections 2–3 and 6–8; Table 4; Appendices S, T, V, W, X, Y.