acceptodds
Under review as a conference paper at ICLR 2027

Not All Memory Updates Are Equal: A Benchmark for Typed Revision in Long-Term LLM Agents

Abstract

Long-term memory enables LLM agents to personalize assistance across interactions, yet evaluating memory updates requires more than checking whether the latest value can be recovered. We introduce RevisionBench, a diagnostic benchmark for the semantics of memory revision and use. It covers ten types of change, including factual correction, temporal replacement, preference evolution, temporary suspension, scope-limited overrides, goal lifecycle changes, and usage restrictions. Each target chain is paired with revision, candidate-discussion, and stable-confirmation histories, with queries probing current state, historical truth, and originally reported information. Our experiments show that systems can recover the correct current state while still failing to preserve the meaning of earlier evidence, and that competing candidates can be as challenging as actual revisions. Controlled studies further show that revision-aware interpretation improves state tracking, whereas explicit typed execution or graph storage does not consistently provide additional gains. RevisionBench provides a controlled testbed for diagnosing when memory systems conflate distinct revision semantics and for developing agents that model how, not merely whether, prior information is superseded.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Long-Term Memory, Memory Supersession, Privacy-Aware Memory, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xLWmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RevisionBench, a controlled benchmark for evaluating the semantics of memory revision in long-term LLM agents. It covers ten revision types and compares actual revisions with matched candidate-discussion and stable-confirmation histories. The primary evaluation measures operative-state and onset-date accuracy, while additional paired queries distinguish current state, historical truth, and originally reported information.

The paper also proposes Typed Revision Graph Memory (TRGM), which retains atomic source evidence and represents temporal, scope, and revision relations. Experiments with two local readers and two API readers report substantial improvements over several memory baselines. Controlled studies examine when revision-aware interpretation and explicit relation execution contribute to these improvements.

Critical strengths
  1. The benchmark targets meaningful distinctions that latest-value accuracy cannot capture. Factual correction and temporal replacement can produce the same current answer while implying different historical truths. Evaluating both changed and preserved information therefore provides a useful diagnostic perspective on memory quality. The matched candidate-discussion and stable-confirmation controls also help distinguish failures to apply a revision from failures caused by merely encountering competing information.

  2. The evaluation offers informative comparisons across representations and readers. The paper includes full-context, retrieval-based, and dynamic-memory baselines, together with controlled variants and relation-level diagnostics. These experiments show that access to relevant evidence alone does not always ensure correct revision interpretation, and provide narrower evidence that explicit execution can help on queries requiring particular semantic distinctions.

Critical weaknesses
  1. The incremental contribution of the proposed typed graph mechanism is not sufficiently established. The large gains over external baselines demonstrate the value of revision-aware memory processing, but do not clearly isolate the benefit of graph structure or explicit typed execution. The Entity Policy control matches or slightly exceeds the default TRGM results on the primary revision task across all four readers. The controlled diagnostics support benefits in selected settings, but do not establish a consistent advantage across the full benchmark. This interpretation is further complicated by contradictory descriptions of whether the default TRGM configuration is Compact or Typed. The paper should more clearly separate its useful benchmark findings from the evidence supporting a distinct methodological contribution.

  2. The benchmark's validity beyond controlled histories remains insufficiently characterized. The matched design is useful for diagnosis, but the evaluation is based on structured histories organized into only 60 history groups, and the most direct correction–replacement analysis uses 30 pairs. The construction description reports automatic checks and manual review of a subset without specifying the extent of independent semantic validation, observed disagreements, or linguistic diversity. Consequently, it is unclear how well the findings transfer to natural conversations with implicit, ambiguous, or overlapping revisions. The benchmark is promising, but its validation and the method's incremental contribution are not yet convincing enough for acceptance.

Questions for authors
  1. Which TRGM variant produced the principal results? Section 5.4 states that TRGM refers to TRGM-Typed unless otherwise specified, whereas Appendix C.3 states that the full 300-chain benchmark uses TRGM-Compact and that unlabeled TRGM results denote Compact. Please reconcile these descriptions and identify the configuration used in each main table.

  2. What additional benefit does the typed graph provide over Entity Policy? In Tables 10–13, Entity Policy matches or slightly exceeds TRGM on revision joint accuracy for every reader. Is there a full-benchmark comparison of Typed, Compact, and Entity Policy with matched source evidence, interpretation instructions, and computational budgets? Such a comparison would materially affect my assessment of the methodological contribution.

  3. How was the benchmark independently validated? How many histories and gold answers were manually reviewed, by how many annotators, and with what agreement or adjudication procedure? Please also clarify how the histories and gold semantic decisions were constructed and how ambiguity or dependence on recurring phrasing was assessed.

1 comment

W1 & Q1–Q2: TRGM variants and the contribution of explicit execution

The TRGM rows in Tables 2 and 3 use TRGM-Compact. Table 4 reports Compact and Typed separately, and Table 5 reports ablations of typed execution and scope relations. We will correct Section 5.4 to identify Compact as the configuration used for the main benchmark.

Full-benchmark comparison. Entity Policy and Compact achieve similar revision joint accuracy in the submitted results. The columns follow Table 2, with Date and Joint reported for actual revisions.

MethodSDRΔSΔDDateJoint
Qwen2.5-7B-Instruct
Entity Policy97.8987.5691.226.67−3.6797.3391.00
TRGM-Compact97.7887.8991.116.67−3.2297.6790.78
TRGM-Typed97.7886.2293.893.89−7.6797.6793.89
Llama-3.1-8B-Instruct
Entity Policy89.4478.6784.564.89−5.8996.7883.78
TRGM-Compact89.4478.0084.005.44−6.0096.7883.11
TRGM-Typed91.2286.4492.67−1.44−6.2297.4492.67

Typed improves revision joint accuracy over both Compact and Entity Policy on both backbones.

Controlled comparison on actual revisions. In a separate evaluation on the same 300 chains, TRGM-Typed and Entity Policy received the same source messages and revision-interpretation instructions, with the same 256-token generation cap. Entity Policy could write brief evidence notes before answering. On the 900 actual-revision questions per backbone, Typed increased joint accuracy from 75.22% to 80.44% on Qwen and from 75.56% to 82.89% on Llama. Across the full controlled evaluation, Typed used 12–13% more total tokens.

W2 & Q3: Benchmark construction, validation, and wording sensitivity

Dataset construction. RevisionBench contains 300 target chains across 60 conversation groups. Each chain has three versions with the same entity, dates, and candidate values. One adopts a change, one discusses it, and one reaffirms the current state. For each question, the gold answer gives the current state and the start date of its current uninterrupted period.

Data validation. We automatically checked timestamps, candidate values, aliases, and consistency across the matched histories. Before selecting the final dataset, two LLMs from OpenAI and xAI independently reviewed all 1,064 candidate pairs and their proposed answers. They agreed on which history contained the actual revision in 1,045 pairs (98.21%). We excluded all pairs on which they disagreed. We then sampled 20% retained pairs for manual review. Reviewers checked whether the histories were semantically consistent, whether each question was answerable at its cutoff, and whether both gold fields were correct. The two independent reviewers flagged three pairs for ambiguous wording. Then we regenerate pairs for that ambiguous cases.

Questions across all ten revision types. Our original study used 30 correction–replacement pairs to ask three questions: What is true now? What was true at an earlier date? What did the user originally say? We now ask these questions on 60 existing chains covering all ten revision types, for a total of 180 histories and 540 questions per backbone.

Does wording affect the results? We rewrote each of the 180 histories above. We kept 99 rewrites that preserved the original dates and values and that a separate model judged to have the same meaning at every turn. For each of these 99 histories, we asked the same current-state question on the original dialogue and its rewrite. The table shows how many answers were correct.

BackboneMethodOriginal dialogueRewritten dialogue
QwenDirect read88/9988/99
QwenTRGM-Typed91/9994/99
LlamaDirect read83/9981/99
LlamaTRGM-Typed89/9990/99

After rewriting, Typed still answered more questions correctly than direct reading on both backbones.

Reviewer_8ZgTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies revision semantics in long-term LLM memory, arguing that recovering the latest value does not capture how revisions affect historical truth and earlier evidence. It introduces RevisionBench, covering 10 revision types and 2,880 evaluation instances, and Typed Revision Graph Memory (TRGM), which preserves evidence and represents revisions through typed relations. TRGM achieves 91.11% and 84.00% revision-state accuracy on Qwen2.5-7B and Llama3.1-8B, respectively.

Critical strengths
  1. The paper addresses an important problem in long-term LLM memory. Distinguishing factual correction from temporal replacement motivates evaluating revision semantics beyond the latest state.
  2. RevisionBench includes three matched history conditions: revision, candidate discussion, and stable confirmation, allowing evaluation of both state changes and unchanged states.
  3. The evaluation covers multiple memory configurations and four LLM readers, with revision-type breakdowns, paired semantic evaluations, and controlled studies of typed and scope relations.
Critical weaknesses
  1. The benchmark’s contribution relative to Memora [2] and the STALE preprint [1] needs clearer articulation. Both evaluate memory updates beyond retrieval. A focused comparison should clarify which additional distinctions RevisionBench evaluates concerning historical validity and preserved evidence.
  2. TRGM’s distinction from Graphiti [3] needs clearer articulation. The scope of the evidence for explicit typed execution also needs clarification: Section 5.4 identifies TRGM-Typed as the default, whereas Appendix C.3 specifies TRGM-Compact for the full 300-chain benchmark. Although the semantic and fixed-evidence diagnostics demonstrate benefits in selected settings, a full-benchmark Compact-versus-Typed comparison is not reported, and Entity Policy achieves comparable primary-task accuracy. Consistent variant labels and a matched full-benchmark comparison would clarify whether the diagnostic gains generalize.
  3. The paired evaluation of current value, historical truth, and original report covers only correction and replacement. Additional diagnostics include 72 typed-relation queries and 12 relation-dependent scope queries per reader, providing limited further evidence. Broader change-and-preservation probes would better establish semantic coverage across the full taxonomy.
  4. The source of TRGM’s low candidate-discussion onset accuracy remains unclear. A breakdown by query position would help locate when these errors occur, particularly after candidate discussion or later mentions of obsolete values.
  5. The evaluation relies on structured revision histories. RevisionBench contains 300 target chains across 60 history groups, with automatic checks and manual review of an unspecified subset. The paper does not report the subset size or detailed validation results.

[1] Chao, Hanxiang, et al. "STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?." arXiv 2026. [2] Uddin, Md Nayem, et al. "From recall to forgetting: Benchmarking long-term memory for personalized agents." Findings of ACL 2026. [3] Rasmussen, Preston, et al. "Zep: A temporal knowledge graph architecture for agent memory" arXiv 2025.

Questions for authors

Please refer to the weaknesses.

1 comment

W1: A taxonomy of revision types.

RevisionBench classifies updates by what they change and preserve. A factual correction revises what was true in the past. A temporal replacement changes the state from a given date onward. A usage restriction changes whether a fact can be used. Memora [2] categorizes memory operations as additions, updates, or deletions. STALE [1] distinguishes implicit conflicts within a single attribute from conflicts that propagate across attributes.

In January, the user says, "I cycle to work." Consider two alternative February messages:

  • Temporal replacement. "Starting today, I'm taking the bus to work."
  • Factual correction. "I actually took the bus to work in January, and I still do."
BenchmarkIllustrative questionExpected response
MemoraPlan my commute for tomorrow based on my current routine.Plan a bus commute.
STALESince I still cycle to work, how should I get my bike ready for tomorrow’s commute?Flag that the user now commutes by bus.
RevisionBenchHow did I actually commute in January, and what did I say at the time?With replacement, the user cycled in January and said they cycled. With correction, the user took the bus in January and said they cycled.

W2: TRGM variants and full-benchmark comparison.

Thank you for pointing out the labeling error. Table 2 reports TRGM-Compact, as specified in Appendix C.3. Section 5.4 incorrectly names TRGM-Typed as the default. We will correct Section 5.4 and label each variant explicitly. Compact presents selected evidence for the reader to interpret. Typed applies explicit revision and scope rules before answering.

Relation to Graphiti. Graphiti [3] tracks fact validity and transaction time and invalidates conflicting edges. TRGM-Typed applies different rules depending on the revision type. For example, a February replacement preserves the old value as historically valid in January. A February correction can revise the answer about January while preserving what the January report originally said.

Full-benchmark results. We compare TRGM-Compact, TRGM-Typed, and Entity Policy on the same 300 chains, with 2,700 questions per backbone. As in Table 2, Date and Joint are reported for actual revisions.

MethodSDRΔSΔDDateJoint
Qwen2.5-7B-Instruct
Entity Policy97.8987.5691.226.67−3.6797.3391.00
TRGM-Compact97.7887.8991.116.67−3.2297.6790.78
TRGM-Typed97.7886.2293.893.89−7.6797.6793.89
Llama-3.1-8B-Instruct
Entity Policy89.4478.6784.564.89−5.8996.7883.78
TRGM-Compact89.4478.0084.005.44−6.0096.7883.11
TRGM-Typed91.2286.4492.67−1.44−6.2297.4492.67

Typed achieves the highest joint state-and-onset accuracy on actual revisions for both backbones.

W3: Change and preservation across all ten revision types

We extended the paired evaluation to all ten revision types in RevisionBench. We ask what is true now, what was true at an earlier date, and what was originally stated. TRGM-Typed reuses one evidence graph for the three questions, and the direct-reading baseline receives the same source text.

BackboneMethodCurrent valueHistorical truthOriginal report
QwenDirect read90.00%96.67%97.78%
QwenTRGM-Typed91.67%96.11%98.33%
LlamaDirect read88.89%66.11%88.89%
LlamaTRGM-Typed91.11%92.22%94.44%

On Llama, TRGM-Typed improves historical-truth accuracy by 26.11 percentage points over direct reading.

W4: Where do candidate-discussion onset errors occur?

We break down TRGM-Compact's onset accuracy in the candidate-discussion condition by query position.

Query positionQwen onset accuracyLlama onset accuracy
After the first candidate discussion65.00%62.33%
After the final candidate discussion57.67%76.00%
After an obsolete value is mentioned again56.00%69.67%
Overall59.56%69.33%

Onset accuracy drops after an obsolete value is mentioned again, from 57.67% to 56.00% on Qwen and from 76.00% to 69.67% on Llama.

W5: Benchmark validation.

We automatically checked timestamps, candidate values, aliases, and consistency across matched histories. Before selection, two LLMs from different providers (OpenAI and xAI) independently reviewed all 1,064 candidate pairs and their proposed gold answers, agreeing on which history contained the actual revision in 1,045 pairs (98.21%). We excluded all pairs on which they disagreed. We then sampled 20% retained pairs for manual review. The two independent reviewers flagged three pairs for ambiguous wording. Then we regenerate pairs for that ambiguous cases.

Reviewer_P4x8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RevisionBench, a diagnostic benchmark evaluating typed memory revision in long-term LLM agents. Rather than focusing solely on latest-value retrieval, RevisionBench categorizes memory updates into ten distinct semantic types (e.g., factual correction, temporal replacement, preference evolution, temporary suspension, usage restrictions) and probes current state, historical truth, and originally reported information. Experiments reveal that existing memory systems often conflate historical evidence with superseded states and are easily confused by non-adopted candidate discussions. The authors find that revision-aware interpretation improves performance, while explicit typed execution or graph-based stores offer marginal gains. However, the experiments seem not to be very sufficient in terms of a benchmark paper, especially in the main content.

Critical strengths

1 Expanding memory evaluation beyond naive latest-value recovery to ten revision semantics addresses a realistic and often-overlooked challenge in agent memory.

2 Testing across current state, historical truth, and original phrasing effectively diagnoses whether systems preserve or erase earlier evidence.

3 The ablations provide actionable guidance, showing that semantic interpretation prompts outperform rigid typed execution or complex graph schemas.

Critical weaknesses

The one major disadvantage of this paper remains to be the lack of its comprehensive experiments.

1 While standard baselines and graph stores are evaluated, the paper lacks investigation into dedicated lightweight memory controllers or fine-tuned routing modules designed specifically for state tracking.

2 Limited Model and Memory Baseline Diversity: The empirical evaluation lacks comprehensive testing across modern memory frameworks (e.g., MemGPT/Letta, HippoRAG) and a broader range of open/closed-source LLM families across scales. Furthermore, key experiments (such as the API Reader study) are evaluated only on a reduced 100-chain subset.

3 Lack of Downstream End-to-End Agent Tasks: Evaluation is strictly formulated as direct question-answering probes (current state, historical truth, original report). The paper would be much stronger if it demonstrated how misidentifying revision semantics concretely impacts long-horizon agent decision-making in downstream interactive workflows.

Questions for authors

Did the authors consider validating RevisionBench on downstream agentic tasks (e.g., tool-use or long-horizon task completion) to evaluate the impact of memory supersession errors beyond standalone QA probing?

1 comment

W1: Comparison with dedicated memory controllers.

We compared TRGM with CUPMem [a], the write-time memory controller introduced in STALE, and the bounded-memory rollout from Supersede [b]. All methods were evaluated on RevisionBench using Qwen2.5-7B-Instruct.

MethodState SState DState RΔSΔDDate RJoint R
CUPMem71.2244.1131.5639.6712.5658.2230.33
Supersede68.1151.0050.4417.670.5618.228.56
TRGM97.7887.8991.116.67−3.2297.6790.78

Following Table 2, accuracies are in percent and gaps are in percentage points. TRGM achieved higher joint accuracy on actual revisions than CUPMem and Supersede.

[a] STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?, arXiv:2605.06527.

[b] Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents, arXiv:2606.27472.

W2: Full-benchmark API evaluation and additional memory baselines.

We extended the GPT-4.1 study from 100 to all 300 chains. S, D, and R denote stable confirmation, candidate discussion, and actual revision, respectively.

MethodState SState DState ROnset SOnset DOnset RJoint SJoint DJoint R
Full Context97.1175.7886.4492.7860.7899.4490.7859.1186.22
BM25 (k=5)91.8973.7873.1185.3370.7883.0082.1166.8972.22
A-Mem94.6773.6781.6798.4470.7896.3394.3367.6781.11
TRGM96.6780.7889.7898.4480.3398.7896.6776.6789.00

Our TRGM achieved the highest joint accuracy across all three conditions.

We also evaluated native Letta and HippoRAG 2 on RevisionBench using Qwen2.5-7B-Instruct. Their joint accuracies were 73.9% and 72.8%, respectively. On the same questions, TRGM achieved 90.0% accuracy.

W3 & Q1: How memory updates affect downstream tool use.

We evaluated 180 workflows across six types of memory updates in booking, shipping, reminders, and document delivery. Each of scenarios has three versions: an adopted revision, a proposed change that was never adopted, and a no-change control.

Memory methodLlama-8BQwen-7B
A-Mem67.2%90.6%
Mem075.0%78.9%
TRGM95.6%96.1%

TRGM achieved the highest completion rate on both backbones. On Llama, it improved completion by 20.6 percentage points over the strongest baseline Mem0. In the document delivery cases where student access had been revoked, TRGM made no prohibited sends.

We also evaluated TRGM on Mem2ActBench [c], a public benchmark that tests whether agents can use earlier conversations to fill in tool arguments. With the same Qwen backbone, TRGM scored 121/400, compared with 117/400 for direct readout.

[c] Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents. ACL 2026.

Comments on the paper

1 comment

Summary for the Area Chair

  • Benchmark contribution. We clarified how RevisionBench differs from Memora and STALE, and extended the tests of current state, historical truth, and original reports to all ten revision types.
  • Method and baselines. We clarified TRGM’s distinction from Graphiti and corrected the Compact and Typed labels. On the full 300-chain benchmark, Typed outperformed both Compact and Entity Policy on joint state-and-onset accuracy for actual revisions. We also added dedicated controller baselines, tested individual execution rules with fixed evidence, and expanded the API-reader evaluation to all 300 chains.
  • Downstream tool use. We expanded the evaluation to 180 four-step workflows and added results on Mem2ActBench, showing how memory revisions affect tool calls.
  • Data validation and error analysis. We reported independent checks by two modelsi and human reviews. We also tested whether results held after rewording the dialogues and broke down onset errors by query position.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.