- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an append-only memory method that never deletes memories.
It organizes memories through three steps: record, curate, and store.
One distinctive feature is its endorsement mechanism, which assigns each memory a confidence-like score. This score is then used during subsequent memory retrieval
Critical strengths
-
The overall design is simple and intuitively‑structured, yielding substantial performance improvements over competing methods in the main experiments.
-
The component‑wise ablation presented in Table 8 is well‑organized, with confidence intervals provided to back its statistical findings.
Critical weaknesses
-
CuratorMem’s core performance gains hinge on memory ranking driven by ground-truth labels or environmental feedback, as demonstrated in Table 8. Such objective labels are largely unavailable in most real-world applications, and disabling this feedback eliminates the reported 7.41-point performance uplift. This renders the main-experiment comparison somewhat unfair: unlike CuratorMem, the A-Mem and Mem0 baselines have no access to ground-truth signals for memory construction.
-
Token consumption results are only partially reported. Given that LLM inference costs constitute a key practical deployment consideration, a full side-by-side comparison of input and output token usage across all memory approaches would facilitate a more thorough evaluation of their respective engineering trade-offs in the main experiment.
Questions for authors
Could authors also report the results for CuratorMem’s self-assessment and no-certificate variants in the main experiments, so they can be compared side by side with the other methods?
rebuttal
Here is a point-by-point response. Q:core performance gains hinge on memory ranking driven by ground-truth labels or environmental feedback.
True. This was a fatal flaw in both the experiments and the code that produced them, and there was no way to repair it without substantial rework, since label-driven ranking was a mechanism the earlier pipeline was built around. We therefore spent the week rebuilding those parts. Ranking is now driven exclusively by an outcome recorded by an independent, memory-free run of the same task; the benchmark's official scorer is consulted only to compute the final number and never enters the memory path; each chain is split so that the first 30% of tasks build the store and the remaining 70% are scored, with no label crossing the divide. Components that consulted the label were removed rather than disabled, and every number they had produced was discarded and re-run.
Q:main-experiment comparison somewhat unfair
True. The original submission offered a broad comparison across models, baselines and benchmarks, and attempted to show an advantage on every one of them. Where the effect was marginal, we followed it with a lengthy statistical analysis arguing that it was significant. We now narrow the claim. The method is no longer presented as superior everywhere; it is presented as no worse than the baselines in most settings, and as clearly better in the circumstances where the versioned lineage will take effect—namely, situations involving conflict resolution and temporal evolution. That advantage is bought with tokens, and we state the exchange rate rather than leave it implicit (§5.3). We believe the present version establishes our true contribution and novelty.
Q:self-assessment and no-certificate variants in the main experiments, so they can be compared side by side with the other methods
We agree, and we find the original choice of benchmarks faulty. GAIA is a single-passage lookup benchmark: each question is answered from one retrieved document, and no two answers are versions of one another. GAIA2 introduces asynchronous environment changes, but its tasks are independent and the change lives in the environment rather than in the stored answer. τ²-bench has episodes that are self-contained. None of the three contains the structure the design addresses—two statements about the same fact, of which only the later one still holds. We could inject synthetic version drift into these benchmarks, but the intervention would replace their official scoring protocol, which would make our numbers incomparable with the published ones and would measure our own perturbation rather than the benchmark. We therefore report those runs as a boundary (Appendix G) and evaluate where the structure is native: LoCoMo for the static case, and MAB, whose Test-Time Learning and Conflict Resolution splits are exactly the case in which a later statement overrides an earlier one.