- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how memory construction and answering procedures affect an agent's ability to correct outdated premises while preserving accurate recall. It introduces LEDGER, a memory architecture that retains raw dialogue, attribute histories, and resolved current values, together with a query-dependent reader for recall, premise verification, and action planning.
Critical strengths
- The paper provides valuable controlled experiments and reveals an important failure mode in memory-based premise correction.
Critical weaknesses
W1. Improved premise resistance comes with substantial degradation in other capabilities. Table 5 (p. 8) shows that replacing R1 with R3 on GPT-4.1 improves stale-premise accuracy from 0.162 to 0.427, but reduces valid-premise accuracy from 0.855 to 0.427 and current-state accuracy from 0.521 to 0.325. This raises concerns about whether the proposed reader improves selective correction or primarily shifts behavior toward more aggressive premise rejection.
W2. Oracle-state interventions do not demonstrate an implementable correction mechanism. Table 6 (p. 9) shows that replacing stored values and transition records with ground-truth information substantially improves performance on GPT-4.1. However, these interventions assume access to correct state information and therefore establish sensitivity to representation rather than the effectiveness of an automatic memory-resolution method.
W3. The proposed approach is highly sensitive to the reader backbone and representation. Table 6 (p. 9) shows that replacing heads and transitions improves held-out valid-premise accuracy to 0.931 on GPT-4.1 but reduces it from 0.328 to 0.275 on GPT-4o-mini. This substantial difference suggests that the proposed state representation and verification procedure do not consistently support selective correction across models.
Questions for authors
Q1. Can the authors develop or evaluate a reader that improves stale-premise correction without substantially increasing false corrections on valid-premise questions, under the same memory state?
Q2. Can an automatic state-resolution method reproduce the improvements observed under oracle head and transition replacements, particularly across different model backbones?