- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the reliability of diagnosing failures in agent memory systems through oracle interventions. The authors evaluate three interventions that correct upstream information, retrieval, or the solver's read context, and compare their effects individually and in combination. Experiments are conducted on 221 MemoryArena tasks across seven language models and multiple memory backends. Results show that correcting upstream information and directly providing the solver with gold context recover largely overlapping failures, while retrieval repair provides much smaller gains. The authors also find that different LLM judges can lead to inconsistent diagnostic conclusions.
Critical strengths
- The paper presents a well-controlled empirical study with good coverage across models and memory backends.
- The experimental analysis is thorough. The authors examine task-level error overlap, provide statistical significance tests, and investigate the sensitivity of the findings to different evaluation methods.
Critical weaknesses
- The paper is very difficult to read. The presentation introduces many abbreviations and mathematical formulations that make the core idea less accessible than necessary. I would suggest simplifying the presentation and using more concrete examples. I think the abstract/intro/related work writing is currently below the ICLR standard as well.
- The novelty and significance of the findings seem limited. The paper mainly presents an empirical comparison of different oracle interventions. The central finding that correcting earlier answers and directly supplying the same information to the solver recover overlapping failures is somewhat expected. While the empirical evidence is useful, it is unclear what new insights the proposed framework provides.
- The evaluation scope limits the generalizability of the findings. All experiments are conducted on one benchmark family with relatively small models, and the native accuracy is extremely low (1–9%). It would be helpful to evaluate stronger agents and more realistic memory tasks where failures are distributed across memory formation, retrieval, and utilization.
Questions for authors
Please refer to the weaknesses.