acceptodds
Under review as a conference paper at ICLR 2027

When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents

Abstract

When documents supporting an agent's derived facts are revoked or replaced, should it repair memory or re-read current evidence? We introduce an evidence-revision evaluation on medication- and problem-list tasks from public ICU records. Under revocation, replacement and control events, we compare full and source-filtered re-reading with caching, rebuilding and graph-local repair across two 7B models. Memory is supplied in full without retrieval, and costs include ingest, revision and every use. On short records, local repair uses 5–10× fewer revision tokens than rebuilding, yet every memory pipeline costs at least twice full re-reading in held-out conditions. In a small pre-specified development sweep, adding task-ineligible documents extended records to about 10,000 tokens; at that length, memory became cheaper than full re-reading after 2–14 uses on average, partly through truncated extraction, while source-filtered re-reading remained cheapest. In the replacement study, none of the four primary confirmatory tests reached statistical significance. These results show why the cost of agent memory after evidence revision must be assessed against source-filtered re-reading over the full pipeline.

Primary area:
datasets and benchmarks
Keywords:
agent memory, memory repair, provenance, re-reading baseline, token cost, full-pipeline accounting, pre-specified evaluation, LLM agents, clinical records, evidence revision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j3NHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an agent should repair derived memory or re-read currently valid evidence after source documents are revoked or replaced. It introduces an evidence-revision evaluation protocol that compares re-reading, caching, rebuilding, and graph-local repair in terms of stale information, retention, replacement adoption, source compliance, and full-pipeline token cost. Local repair reduces revision-time recomputation, but all memory pipelines are more expensive than re-reading on short records. On longer records, memory can outperform full re-reading after repeated use, but remains more expensive than source-filtered re-reading. Re-reading also adopts replacement values more often, although the main confirmatory results are not statistically significant.

Critical strengths
  1. Important and carefully structured evaluation problem. The paper evaluates agent memory under evidence revocation and replacement, jointly considering stale-information removal, retention of valid information, replacement adoption, and source compliance. This provides a more realistic evaluation of memory revision than measuring only final-answer accuracy.

  2. Practically meaningful full-pipeline cost comparison. The paper accounts for memory ingestion, revision, and every subsequent use, and compares memory against both full and source-filtered re-reading. Its finding that local repair can reduce recomputation without reducing total cost provides a valuable and more rigorous standard for future memory-repair research.

Critical weaknesses
  1. Limited statistical strength and generalizability. The main experiments use one ICU dataset, two 7B models, and relatively small patient samples, while none of the four pre-specified replacement tests reaches statistical significance. The problem-list results also depend on post-hoc lexical scorers whose choice substantially changes the conclusions, leaving the robustness of the main findings uncertain.

  2. Important experimental confounds. Memory and re-reading differ simultaneously in representation and prompting, while the length sweep expands context mainly with documents that the task is not allowed to use and includes substantial extraction truncation or parsing failures. Consequently, the results cannot be attributed specifically to the repair mechanism, and the reported cost break-even points may not generalize to realistic settings where task-relevant evidence grows.

Questions for authors
  1. In the length sweep, most added documents are not allowed sources for the medication task, and many extraction calls are truncated or fail to parse. Can the authors quantify how much each factor contributes to memory’s apparent cost advantage? Does a similar break-even point remain when only task-eligible evidence grows?

  2. Memory and re-reading use different representations and prompts. Can the authors clarify how much of the observed difference can be attributed specifically to the repair strategy rather than extraction loss, summary formatting, or prompt differences?

  3. The problem-list conclusions are highly sensitive to the lexical scoring rule. Do the authors have human or semantic evaluations showing that the main findings are not artifacts of the post-hoc scorer?

  4. Can the authors repeat the replacement comparison on a larger unseen sample or additional models? None of the four pre-specified 7B primary tests is statistically significant, making it difficult to determine whether the observed adoption advantage is robust.

Reviewer_YVJvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether an agent should repair its derived memory or re-read current evidence after a source document is revoked or replaced. The authors build an evaluation protocol around two tasks: 1) a medication handoff list and 2) an active problem list drawn from the eICU demo records. The key contribution is full-pipeline cost accounting: ingest plus revision plus every use, formalized through a break-even decomposition.

Critical strengths
  1. The paper's most consequential strength is its cost-accounting framing. Charging ingest, revision, and every subsequent use together — and making source-filtered re-reading rather than full re-reading the baseline — is the correct practical comparison, and prior work consistently missed it. The ΔC(k) break-even decomposition is clean enough to be reused directly by future work with different pipelines or record lengths.

  2. Pre-registered hashing before every batch, disclosed errata, explicit confound statements in the body rather than the limitations section, and transparent acknowledgment of post-hoc scoring amendments are all uncommon in evaluation papers. This makes the negative results trustworthy in a way that matters: a reader knows precisely what the numbers can and cannot support.

Critical weaknesses

1.The cost comparison conflates prompt verbosity with memory cost. Memory use-prompts render extracted claims plus summaries, which are longer than the raw documents, so the 2× overhead is partly a rendering choice rather than an intrinsic property of maintained memory. The information-matched ablation that would isolate this is explicitly flagged as missing in §9, which means the central result cannot distinguish "memory is expensive" from "we rendered it verbosely." 2. The one positive finding is an artifact. Memory beats full re-reading only at XL, and only because 225/520 extraction calls hit the token cap and 44 fail to parse meaning memory answers from less information than re-reading sees. Cost and correctness are not on equal footing in precisely the cells that favor memory. 3. Scoring contracts were written after outputs were seen; the significance count ranges from 0 to 12 of 12 depending on which of seven rules is applied. It should be reported as exploratory and not given the weight the tables imply. 4. The 32B supplement points the other way. The exploratory 32B run rejects the C2 primary in 2 of 3 cells. If the 7B nulls are a weak-model artifact — which this suggests — then the paper's practical conclusion that re-reading is preferable is premature rather than supported.

Questions for authors
  1. Batch A shows that facts-only (F) and summaries-then-facts (SF) branches produce nearly identical outcomes, but the prompt length difference was not controlled. Did you run a version of memory with use-prompt length matched to re-reading's raw document length? Without this, how do you rule out that the 2× cost overhead is purely a rendering choice?

  2. At XL, 225/520 added-document extractions hit the output cap and 44 fail to parse. How do the break-even crossings change if you exclude these truncated/failed trajectories, or impute what a complete extraction would have cost? Is the memory cost advantage at XL robust to complete extractions?

  3. The 32B supplement rejects the C2 primary in 2/3 cells — the opposite of the 7B result. Why is this treated as exploratory rather than as a primary validity concern? What is the theoretical reason the 7B conclusions should generalize to the model sizes relevant to production deployments?

  4. All memory pipelines share the system prompt written for the memory setting, which you acknowledge is "not a perfect description of the re-reading prompt." Did you run re-reading with a prompt written for re-reading? How much of the correctness gap on the problem list is attributable to this mismatch?

  5. The E4 leakage occurs because the extractor stores every drug mention as a medication fact regardless of source — the non-allowed drug enters memory as med | drug | from=D4:ehr alongside genuine pharmacy facts, and the model must then filter by a from= tag in the use prompt. Re-reading never faces this because source is structurally visible in the raw document. All memory pipelines fail at similar rates, which suggests the failure is in extraction typing, not maintenance strategy. Did you run a version where the extractor simply does not store facts from non-allowed sources at ingest? If E4 leakage disappears under that condition, the current E4 comparison is measuring a rendering artifact, not a property of memory repair.

Reviewer_YRRTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language-model agents should maintain and repair derived memory or re-read current evidence when source documents are revoked or replaced. The authors introduce an evidence-revision evaluation protocol using medication-list and problem-list tasks constructed from public eICU records. The protocol includes document revocation, replacement, independent-support retention, and irrelevant-source addition scenarios.

The paper compares re-reading, document-level caching, full memory rebuilding, and dependency-graph-based local repair using two 7B language models, with an exploratory 32B supplement. It evaluates correctness, retention, source compliance, and end-to-end token costs, including memory ingestion, revision, and subsequent uses.

The main findings are that local repair substantially reduces revision costs compared with full rebuilding, but memory-based pipelines remain more expensive than re-reading on short records. With longer records containing task-ineligible documents, memory pipelines can become cheaper than full re-reading after repeated uses, while source-filtered re-reading remains the cheapest approach. The replacement experiments show a directional advantage for re-reading in adopting updated values, although the primary confirmatory tests do not reach statistical significance.

Critical strengths

S1. A useful evaluation perspective with comprehensive cost accounting.

The paper raises an important question that is often overlooked: whether maintaining and repairing derived memory is worthwhile compared with directly re-reading current evidence. Instead of evaluating repair costs in isolation, it accounts for ingestion, revision, and subsequent uses. The comparison with source-filtered re-reading is particularly valuable, as it provides a stronger and more practical baseline than full-document re-reading alone. This perspective can help inform the design and evaluation of future memory systems.

S2. Careful evaluation design and transparent reporting of limitations.

The paper distinguishes multiple evidence-revision scenarios, including revocation, replacement, independent support, and irrelevant-source additions. It also separates adoption of updated information from retention of other valid information, which is important for understanding different failure modes. The authors provide detailed cost breakdowns, pre-specified experiments, statistical analyses, and sensitivity studies. I also appreciate the explicit disclosure of non-significant findings, extraction truncation, and post-hoc scoring changes, which helps readers interpret the evidence appropriately.

Critical weaknesses

W1. Insufficient justification for the problem formulation and the choice of memory architecture.

My primary concern is whether the experimental setting sufficiently motivates the proposed comparison between LLM-based memory repair and re-reading.

The paper considers explicitly versioned, source-attributed clinical records, where an authoritative system already identifies which documents have been revoked or replaced. The downstream tasks involve reconstructing medication or problem lists from currently valid evidence. In this setting, the central requirement is maintaining consistency with an authoritative evidence store, which seems closely related to traditional database view maintenance or cache invalidation.

I agree that dependencies between derived facts and summaries introduce additional maintenance challenges, especially when LLM-generated representations are involved. However, it remains unclear why an LLM-generated hierarchy of facts and summaries is necessary for these particular tasks in the first place.

For example, when a medication strength changes from 2 mg to 4 mg, a structured, versioned memory representation could potentially update the corresponding fact and its provenance directly, without re-extracting the document or regenerating multiple summaries. Similarly, revoking a document with independently supported facts can be handled through provenance and support-set tracking.

The paper compares several relatively expensive LLM-based maintenance pipelines, but does not include a lightweight structured-memory baseline with deterministic updates or field-level patches. Such a baseline would be particularly relevant because the evaluation already assumes structured document metadata, explicit revisions, and well-defined expected answers.

This omission makes it difficult to determine whether the observed costs reflect an inherent limitation of maintaining derived memory or the particular choice of representation and repair implementation.

More broadly, the paper's setting differs substantially from conversational or personalized agent memory, where information is accumulated across interactions, may be subjective or context-dependent, and can evolve without an explicit authoritative revision event. The paper does not need to address all these settings, but its motivation and implications should more clearly distinguish evidence-backed record maintenance from general-purpose long-term agent memory.

W2. The experimental design does not sufficiently establish when memory repair is preferable to re-reading, and important factors remain confounded.

The experimental evidence is narrower than the general question motivating the paper.

First, the main experiments use very short source records and relatively simple extraction tasks. These conditions naturally favor re-reading, since the original evidence is inexpensive to access and the required outputs can be reconstructed directly.

Second, the length-sweep experiment increases document length primarily by adding task-ineligible evidence. Consequently, source-filtered re-reading can discard nearly all the additional content, making its cost advantage largely a consequence of the experimental construction. This does not establish how the approaches compare when the amount of task-relevant evidence itself increases. Furthermore, a substantial fraction of extractions on added documents hit the output-token limit, meaning that some of the apparent memory savings arise from incomplete extraction rather than effective compression.

Third, the comparison changes multiple factors simultaneously, including evidence representation, extraction, summarization, prompt formatting, and maintenance strategy. This prevents a clear attribution of observed performance differences specifically to memory repair. The replacement results illustrate this concern: the updated value was already present in the memory prompt in every reported rebuild/local-repair non-adoption case in the development comparisons, suggesting that these failures arose during answer generation rather than revision propagation.

Finally, the confirmatory replacement experiments have small sample sizes, and none of the four primary tests reaches statistical significance. The directional findings are interesting, but they do not provide strong confirmatory evidence of a correctness advantage.

Together, these limitations make it difficult to draw conclusions about the conditions under which maintaining and repairing memory becomes preferable to re-reading, beyond the specific pipelines and scenarios evaluated in this work.

Questions for authors

Q1. What makes LLM-based derived memory necessary or preferable for the evaluated tasks?

Given that the underlying evidence is versioned, source-attributed, and directly accessible, could the authors clarify why a structured fact store with provenance tracking and deterministic updates would not be a more natural baseline? In particular, what additional functionality does the LLM-generated facts-and-summaries hierarchy provide for medication-list and problem-list reconstruction?

Q2. How would the main cost conclusions change if the additional documents in the length sweep contained task-relevant information?

The current length sweep primarily adds documents that the medication task is not permitted to use. Would the authors expect the same ordering when the allowed evidence itself grows substantially, so that source-filtered re-reading cannot simply discard the additional content? This distinction seems central to establishing the conditions under which memory becomes beneficial.

Q3. What is the intended scope of the conclusions regarding agent memory?

The evaluation focuses on maintaining derived facts under explicit authoritative evidence revisions. Could the authors clarify whether their conclusions are intended specifically for medical evidence-backed record assistants or for broader long-term agent-memory architectures? If the latter, what aspects of the current evaluation support generalization beyond structured record reconstruction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.