acceptodds
Under review as a conference paper at ICLR 2027

Remember, Then Decide: How Stored State and Readers Shape Premise Correction in Agent Memory

Abstract

Agent memory must correct questions that assume outdated facts while answering valid questions and recalling current and historical information. We introduce LEDGER, an evidence-preserving memory that retains raw dialogue and attribute histories, and evaluate how stored state and answering procedures affect these goals across eight systems and three benchmarks. Holding LEDGER evidence fixed, replacing direct answering with its query-dependent reader raises GPT-4.1 premise resistance from 0.020 to 0.725, although effects vary across stores and backbones. Matched queries over 117 histories nevertheless reveal that higher stale-premise accuracy can accompany lower valid-premise accuracy. On 131 held-out histories, oracle replacement of current values improves both accuracies on both tested backbones. Further replacing transition records with canonical summaries improves valid-premise accuracy on GPT-4.1 but reduces it on GPT-4o-mini. These findings show that correction depends on both the supplied state representation and the reader interpreting it. Preserving evidence and scoring well on separate benchmarks do not establish reliable selective correction. We therefore advocate evaluating stale- and valid-premise accuracy alongside current and historical recall, distinguishing premise rejection from answer correctness, and separating memory-construction effects from reader effects.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Long-term memory, Knowledge updating, Memory conflicts, Premise resistance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f3HUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how memory construction and answering procedures affect an agent's ability to correct outdated premises while preserving accurate recall. It introduces LEDGER, a memory architecture that retains raw dialogue, attribute histories, and resolved current values, together with a query-dependent reader for recall, premise verification, and action planning.

Critical strengths
  • The paper provides valuable controlled experiments and reveals an important failure mode in memory-based premise correction.
Critical weaknesses

W1. Improved premise resistance comes with substantial degradation in other capabilities. Table 5 (p. 8) shows that replacing R1 with R3 on GPT-4.1 improves stale-premise accuracy from 0.162 to 0.427, but reduces valid-premise accuracy from 0.855 to 0.427 and current-state accuracy from 0.521 to 0.325. This raises concerns about whether the proposed reader improves selective correction or primarily shifts behavior toward more aggressive premise rejection.

W2. Oracle-state interventions do not demonstrate an implementable correction mechanism. Table 6 (p. 9) shows that replacing stored values and transition records with ground-truth information substantially improves performance on GPT-4.1. However, these interventions assume access to correct state information and therefore establish sensitivity to representation rather than the effectiveness of an automatic memory-resolution method.

W3. The proposed approach is highly sensitive to the reader backbone and representation. Table 6 (p. 9) shows that replacing heads and transitions improves held-out valid-premise accuracy to 0.931 on GPT-4.1 but reduces it from 0.328 to 0.275 on GPT-4o-mini. This substantial difference suggests that the proposed state representation and verification procedure do not consistently support selective correction across models.

Questions for authors

Q1. Can the authors develop or evaluate a reader that improves stale-premise correction without substantially increasing false corrections on valid-premise questions, under the same memory state?

Q2. Can an automatic state-resolution method reproduce the improvements observed under oracle head and transition replacements, particularly across different model backbones?

Reviewer_kMdwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how an agent should answer when facts change during a conversation. The purpose is to reject outdated premises without rejecting valid ones or losing access to current and past facts. The paper proposes, Ledger, which keeps the raw dialogue and a history of values for each (entity, attribute) pair. It selects a current value when new information is written. It can also infer changes to directly related attributes. At answer time, Ledger routes each query to factual recall, premise verification, or planning. For premise verification, a separate LLM call checks the query's premise against the stored history. If the premise is judged stale, Ledger uses the selected current value to correct it and excludes the stale value as a basis for the answer. Recall and planning use different answering procedures. Results on STALE, LongMemEval, and HaluMem are mixed. Controlled experiments show that the answering procedure can substantially improve stale premise resistance. However, tests also show losses in valid premise acceptance and recall. The results therefore support the value of premise checking, but do not establish a consistently better accuracy/cost balance.

Critical strengths
  • Good readability. The paper clearly describes the Ledger architecture and its query-dependent answering procedures. Figure 1 makes the distinction between memory construction, recall, premise verification, and planning easy to follow.
  • Interesting results on query-dependent answering. The results suggest that changing how the system answers a query can improve stale-premise resistance, even when the stored evidence is unchanged. On identical Ledger evidence, replacing R0 with R3 raises GPT-4.1 STALE Type 1 premise resistance from .020 to .725. This is a useful finding, although the reader changes several operations at once and does not improve every capability.
Critical weaknesses
  • Better stale-premise resistance does not establish selective correctness: On the same 117 histories and frozen Ledger memory, replacing R1 with R3 raises GPT-4.1 stale-premise accuracy from .162 to .427. However, valid-premise accuracy falls from .855 to .427, current-state accuracy from .521 to .325, and historical accuracy from .778 to .675. GPT-4o-mini shows the same directional losses. These results directly limit the central goal: correcting outdated premises without harming valid answers or recall. Lower valid-premise accuracy alone does not mean every failure is a false correction; both outcomes should be reported separately.
  • The remaining failures are not fully localized: Ledger combines memory construction with query routing, premise checking, evidence selection, and answer generation. The frozen-evidence comparisons and write/verifier factorial help separate broad contributions. However, replacing the complete reader changes several operations at once. These controls do not identify which reader steps cause the losses in valid-premise accuracy and recall. More focused ablations should measure these losses alongside stale-premise resistance.
  • The correction mechanism depends on unreliable stored state. On 131 additional histories, replacing selected current values with oracle values improves both stale- and valid-premise accuracy on both backbones. Replacing transition records as well raises GPT-4.1 valid-premise accuracy to .931 but lowers GPT-4o-mini accuracy to .275. The reader therefore depends on both state quality and its presentation. The oracle interventions diagnose this problem, but do not provide a deployable way to recover the correct state.
  • The complete system has no clear accuracy-cost advantage. With GPT-4o-mini, full Ledger's updated-fact accuracy is .68, compared with .74 for Ledger-base and .82 for RAG. The five-item cost sample reports 102 calls and 95.2K LLM tokens per item for full Ledger, versus 53 calls/45K tokens for Ledger-base and 3 calls/8K tokens for RAG. Higher stale-premise resistance must be weighed against these recall and cost differences. The small cost sample also cannot establish scaling.
Questions for authors
  1. Why does better state resolution not consistently improve policy adaptation? When SR improves but IPA declines, does the system recognize the updated fact but fail to apply it when planning? Could you report IPA conditional on correct SR for the same scenarios and inspect the remaining failures?
  2. Why does stronger stale-premise resistance reduce valid-premise accuracy? Is the reader becoming overly skeptical, relying on incorrect current values, or rejecting premises despite a correct verification result? Could you trace verification verdicts and final answers on the matched histories to distinguish these explanations?
  3. What causes the recall losses? Does the reader select insufficient evidence, incorrectly exclude useful historical facts, or fail to use evidence already present? Could you compare evidence availability and answer correctness before and after reader replacement?
  4. Why do the oracle transition replacements help GPT-4.1 but hurt GPT-4o-mini? Could you separate changes to transition content from changes to formatting and selection? This would help explain whether the difference comes from model capability or the representation.
Reviewer_UC8zOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces LEDGER, a memory system that combines raw dialogue, attribute histories, and resolved current values with query-dependent answering. Across three benchmarks, controlled reader comparisons, and matched queries on the same histories, the authors examine how stored state and answering procedures affect premise correction and recall. The results show that stronger stale-premise resistance can impair valid-premise handling, while representation interventions have effects that depend on the reader and backbone.

Critical strengths

The matched-history evaluation usefully tests whether gains in stale-premise resistance come at the expense of valid-premise handling and recall.

Holding upstream evidence fixed in reader comparisons and reusing memory snapshots within each write setting strengthen the experimental controls.

The observed interactions among stores, readers, and backbones offer useful engineering insights, and the authors generally qualify their conclusions appropriately.

Critical weaknesses

The component choices need stronger empirical justification. The paper does not sufficiently establish when additional memory views, write resolution, or propagation justify their complexity. For GPT-4o-mini, base+arch achieves Type 1 PR/KU of .76/.76 versus .78/.68 for the full system (Table 24). Although these independent builds do not establish superiority, the results motivate clearer guidance on component selection.

Failure attribution remains limited. R3 changes routing, extraction, verification, and prompting together, while head-plus-transitions changes both transition content and presentation. These interventions establish overall effects but provide limited guidance on their causes. The unspecified precedence between ACTION and PRESUPPOSE further complicates interpretation.

The evaluation protocol needs stronger validation. Appendix D.11 permits a correct stale-premise label for flagging OLD as outdated or indicating NEW without requiring full request completion. The scores therefore have a narrower meaning than complete answer correctness. The planned human audit has no reported results, and retaining 117 of 180 diagnostic candidates may introduce selection effects beyond the preserved conflict-type ratio.

Reproducibility and run variability need clarification. Complete reader prompts, baseline revisions, and some data-processing settings are missing. The unchanged GPT-4.1 R3 baseline shifts from .427 to .573 after regeneration and rejudging (Table 15), but the contributions of generation and judging variability are not separated.

Questions for authors

Which query or update types justify the full configuration over simpler alternatives? Could a targeted control with fixed extraction and reader clarify the roles of the memory views?

How are overlapping ACTION/PRESUPPOSE cases handled? Could existing traces and one focused intervention identify a principal source of R3’s false corrections?

Could the planned human audit validate the generated queries and separately annotate premise challenging, correct state use, and request completion?

Could the authors release complete prompts and settings, and separately assess judging variability on fixed responses and generation variability under a fixed evaluation protocol?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.