- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how arrival order can change extraction batches and produce different agent memories, even when merging is commutative. The paper proposes canonical replacement to form batches consistently and replace outdated outputs, with agreement conditional on repeatable extraction.
Critical strengths
Making agent memory consistent across different arrival orders is a research problem with some practical value.
Critical weaknesses
-
The significance of the problem is underdeveloped. Changing the inputs to an extractor can change its outputs. This observation is straightforward, and the paper provides limited evidence of how often it causes meaningful failures in existing memory systems.
-
The advantage of canonical replacement is unclear. Its guarantee still depends on repeatable extraction or shared outputs. Hash-based batches can group unrelated observations, and their benefit for memory quality is weakly demonstrated. In the small online study, canonical trie takes 4.19 times the request time of per-event extraction for a 2.74-point accuracy gain, with mixed results across questions.
-
The writing is repetitive and difficult to follow. The paper repeatedly explains what its results do not establish, while the main contribution remains diffuse. Similar terminology for different renderers, partitions, and construction policies also makes the pipeline unnecessarily confusing.
-
The experimental narrative is fragmented. Shared-artifact replay, fixed-atom readout, and fresh online construction test different questions, but the paper does not clearly explain how their findings fit together. In particular, the controlled readout experiment evaluates a presentation scheme that is not used in the online construction study, leaving its connection to canonical replacement unclear.
-
The distinction between consistency and usefulness offers limited insight. Consistent memory does not necessarily produce better answers, since accuracy also depends on extraction quality, retrieval, presentation, and reader behavior. Mixed accuracy results are therefore unsurprising. The paper provides limited explanation of when enforcing consistency improves task performance or when contextual extraction is worth its additional cost.
Questions for authors
No questions for the authors.
Response to Reviewer ndZn
Thank you. We take your points in order.
W1 (significance). We agree that "different extractor inputs give different outputs" is straightforward, and the revision keeps it only as the diagnosis (contribution (a), §3). It adds a measurement of how often this changes answers in an existing library, Mem0 2.2.1 driven by a local 4-bit Qwen3-8B, on all 10 LoCoMo conversations (§4).
Most of the questions that flip already flip between default re-ingestions, so the deterministic rates show fragility, not an arrival-specific excess (§8). Because mean accuracy barely moves, §8 argues that flip rates should be reported next to accuracy.
W2 (value of canonical replacement; repeatable extraction). Our new results agree with your concern about the construction. Hash-keyed batches answer 0.292 of correction questions, against 0.875 for one joint call, and retain 0.312 of the corrections bound to their subject, against 1.000 (§5; Table 18 in App. K.1). In the re-run cost study, the trie shows no measurable accuracy gain over per-event extraction (+0.8 points [−4.7, 6.3]) at 4.11× the extraction tokens (§6). The paper now presents canonical replacement only as the sufficient condition when joint context is needed (Proposition 2, §3.1). It states that a grouping key aligned with the dependency is untested.
You also note that the guarantee depends on repeatable extraction. The revision measures this directly. Extraction outputs were not repeatable by default in batched HuggingFace inference or in llama.cpp with prompt caching (§4, §6). A deterministic configuration made same-order Mem0 ingestions byte-identical in all 10 conversations, and a rebuilt stack reproduced the saved fingerprints of 21 of 21 earlier runs. Repeatability is achievable but must be configured, and §8 says so.
W3–W4 (writing; how the experiments fit). Limitations are collected in the Scope paragraphs of §8. Table 1 gives each experiment one question and marks the exploratory ones, and §8 opens with a paragraph on how the four studies fit. The readout factorial moved to App. F. The online construction study uses one common readout for every construction (§6).
W5 (consistency vs usefulness). We agree that consistency does not imply better answers. §5 quantifies what identity obtained by per-event extraction loses and what joint extraction recovers. For Mem0 we have only one indication of the cost of identity, and it is weak: one conversation (81 questions), no interval, and two factors changed at once. There, context-free one-turn extraction answered 0.395 of questions, against 0.506 with the default context (§4; App. K.3).