- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether an agent should repair derived memory or re-read currently valid evidence after source documents are revoked or replaced. It introduces an evidence-revision evaluation protocol that compares re-reading, caching, rebuilding, and graph-local repair in terms of stale information, retention, replacement adoption, source compliance, and full-pipeline token cost. Local repair reduces revision-time recomputation, but all memory pipelines are more expensive than re-reading on short records. On longer records, memory can outperform full re-reading after repeated use, but remains more expensive than source-filtered re-reading. Re-reading also adopts replacement values more often, although the main confirmatory results are not statistically significant.
Critical strengths
-
Important and carefully structured evaluation problem. The paper evaluates agent memory under evidence revocation and replacement, jointly considering stale-information removal, retention of valid information, replacement adoption, and source compliance. This provides a more realistic evaluation of memory revision than measuring only final-answer accuracy.
-
Practically meaningful full-pipeline cost comparison. The paper accounts for memory ingestion, revision, and every subsequent use, and compares memory against both full and source-filtered re-reading. Its finding that local repair can reduce recomputation without reducing total cost provides a valuable and more rigorous standard for future memory-repair research.
Critical weaknesses
-
Limited statistical strength and generalizability. The main experiments use one ICU dataset, two 7B models, and relatively small patient samples, while none of the four pre-specified replacement tests reaches statistical significance. The problem-list results also depend on post-hoc lexical scorers whose choice substantially changes the conclusions, leaving the robustness of the main findings uncertain.
-
Important experimental confounds. Memory and re-reading differ simultaneously in representation and prompting, while the length sweep expands context mainly with documents that the task is not allowed to use and includes substantial extraction truncation or parsing failures. Consequently, the results cannot be attributed specifically to the repair mechanism, and the reported cost break-even points may not generalize to realistic settings where task-relevant evidence grows.
Questions for authors
-
In the length sweep, most added documents are not allowed sources for the medication task, and many extraction calls are truncated or fail to parse. Can the authors quantify how much each factor contributes to memory’s apparent cost advantage? Does a similar break-even point remain when only task-eligible evidence grows?
-
Memory and re-reading use different representations and prompts. Can the authors clarify how much of the observed difference can be attributed specifically to the repair strategy rather than extraction loss, summary formatting, or prompt differences?
-
The problem-list conclusions are highly sensitive to the lexical scoring rule. Do the authors have human or semantic evaluations showing that the main findings are not artifacts of the post-hoc scorer?
-
Can the authors repeat the replacement comparison on a larger unseen sample or additional models? None of the four pre-specified 7B primary tests is statistically significant, making it difficult to determine whether the observed adoption advantage is robust.