- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how long-term agent memory should preserve decision constraints after their originating events have ended. It introduces CCD, a controlled benchmark designed to fill a gap in existing memory evaluations by distinguishing event resolution from explicit constraint release. The paper further analyzes failures across memory writing, maintenance, retrieval, attribution, and final decision-making. Experiments on synthetic clinical cases show that suppressing resolved-event memories can remove still-binding evidence, while models often fail to associate release records with the correct case. The results validate CCD as a useful diagnostic benchmark, although evaluation on the English DevOps extension is limited.
Critical strengths
-
The paper identifies an important gap in existing agent-memory benchmarks and introduces CCD to evaluate long-range decision dependencies—specifically, whether earlier information continues to constrain later decisions after the originating event has been updated or resolved. This distinction between event status and the continuing validity of a decision constraint is consequential and is not explicitly tested by standard long-term memory benchmarks.
-
The validation is detailed and diagnostically useful. Rather than reporting only end-to-end answer accuracy, the paper separately examines memory writing, maintenance, retrieval, update attribution, and final answer generation. This multi-stage analysis helps localize where relevant constraint evidence is lost or misused, making CCD valuable for evaluating and improving agent-memory systems.
Critical weaknesses
-
Limited scale and external validity. CCD contains only 53 primary clinical cases, all synthetically generated. Although the paper mentions a 40-case English DevOps extension, it does not provide sufficiently complete experimental results for this portion of the benchmark.
-
Dataset construction required substantial repair. Several generated release records were missing explicit lifting evidence, while others contradicted the stated permanent constraint and had to be repaired or excluded from release evaluation. Although the authors disclose this process, the amount of post-generation correction raises concerns about the stability of the benchmark-construction procedure and the reliability of its gold labels.
-
The answer-level evaluation is underspecified and overly simplified. Decisions are reduced to binary retain/release outcomes and scored through deterministic text matching, which may not adequately handle conditional, uncertain, or partially correct answers. The paper also does not provide the exact answer prompt or complete matching rules. This is particularly consequential because the authors report substantial prompt sensitivity, making it difficult to separate memory failures from answerer caution and evaluation artifacts.
Questions for authors
-
Could the authors provide the complete results for the 40-case English DevOps extension? Most reported results appear to use only the 53 Chinese clinical cases.
-
Could the authors provide the exact answer prompt and decoding settings? The paper mentions a fixed “symmetric answer prompt,” but does not show it.
-
Could the authors add a clear metric for memory-writing quality? How do you distinguish an actual writing failure from a valid paraphrase missed by the term-matching scorer?