- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RevisionBench, a controlled benchmark for evaluating the semantics of memory revision in long-term LLM agents. It covers ten revision types and compares actual revisions with matched candidate-discussion and stable-confirmation histories. The primary evaluation measures operative-state and onset-date accuracy, while additional paired queries distinguish current state, historical truth, and originally reported information.
The paper also proposes Typed Revision Graph Memory (TRGM), which retains atomic source evidence and represents temporal, scope, and revision relations. Experiments with two local readers and two API readers report substantial improvements over several memory baselines. Controlled studies examine when revision-aware interpretation and explicit relation execution contribute to these improvements.
Critical strengths
-
The benchmark targets meaningful distinctions that latest-value accuracy cannot capture. Factual correction and temporal replacement can produce the same current answer while implying different historical truths. Evaluating both changed and preserved information therefore provides a useful diagnostic perspective on memory quality. The matched candidate-discussion and stable-confirmation controls also help distinguish failures to apply a revision from failures caused by merely encountering competing information.
-
The evaluation offers informative comparisons across representations and readers. The paper includes full-context, retrieval-based, and dynamic-memory baselines, together with controlled variants and relation-level diagnostics. These experiments show that access to relevant evidence alone does not always ensure correct revision interpretation, and provide narrower evidence that explicit execution can help on queries requiring particular semantic distinctions.
Critical weaknesses
-
The incremental contribution of the proposed typed graph mechanism is not sufficiently established. The large gains over external baselines demonstrate the value of revision-aware memory processing, but do not clearly isolate the benefit of graph structure or explicit typed execution. The Entity Policy control matches or slightly exceeds the default TRGM results on the primary revision task across all four readers. The controlled diagnostics support benefits in selected settings, but do not establish a consistent advantage across the full benchmark. This interpretation is further complicated by contradictory descriptions of whether the default TRGM configuration is Compact or Typed. The paper should more clearly separate its useful benchmark findings from the evidence supporting a distinct methodological contribution.
-
The benchmark's validity beyond controlled histories remains insufficiently characterized. The matched design is useful for diagnosis, but the evaluation is based on structured histories organized into only 60 history groups, and the most direct correction–replacement analysis uses 30 pairs. The construction description reports automatic checks and manual review of a subset without specifying the extent of independent semantic validation, observed disagreements, or linguistic diversity. Consequently, it is unclear how well the findings transfer to natural conversations with implicit, ambiguous, or overlapping revisions. The benchmark is promising, but its validation and the method's incremental contribution are not yet convincing enough for acceptance.
Questions for authors
-
Which TRGM variant produced the principal results? Section 5.4 states that TRGM refers to TRGM-Typed unless otherwise specified, whereas Appendix C.3 states that the full 300-chain benchmark uses TRGM-Compact and that unlabeled TRGM results denote Compact. Please reconcile these descriptions and identify the configuration used in each main table.
-
What additional benefit does the typed graph provide over Entity Policy? In Tables 10–13, Entity Policy matches or slightly exceeds TRGM on revision joint accuracy for every reader. Is there a full-benchmark comparison of Typed, Compact, and Entity Policy with matched source evidence, interpretation instructions, and computational budgets? Such a comparison would materially affect my assessment of the methodological contribution.
-
How was the benchmark independently validated? How many histories and gold answers were manually reviewed, by how many annotators, and with what agreement or adjudication procedure? Please also clarify how the histories and gold semantic decisions were constructed and how ambiguity or dependence on recurring phrasing was assessed.
W1 & Q1–Q2: TRGM variants and the contribution of explicit execution
The TRGM rows in Tables 2 and 3 use TRGM-Compact. Table 4 reports Compact and Typed separately, and Table 5 reports ablations of typed execution and scope relations. We will correct Section 5.4 to identify Compact as the configuration used for the main benchmark.
Full-benchmark comparison. Entity Policy and Compact achieve similar revision joint accuracy in the submitted results. The columns follow Table 2, with Date and Joint reported for actual revisions.
Typed improves revision joint accuracy over both Compact and Entity Policy on both backbones.
Controlled comparison on actual revisions. In a separate evaluation on the same 300 chains, TRGM-Typed and Entity Policy received the same source messages and revision-interpretation instructions, with the same 256-token generation cap. Entity Policy could write brief evidence notes before answering. On the 900 actual-revision questions per backbone, Typed increased joint accuracy from 75.22% to 80.44% on Qwen and from 75.56% to 82.89% on Llama. Across the full controlled evaluation, Typed used 12–13% more total tokens.
W2 & Q3: Benchmark construction, validation, and wording sensitivity
Dataset construction. RevisionBench contains 300 target chains across 60 conversation groups. Each chain has three versions with the same entity, dates, and candidate values. One adopts a change, one discusses it, and one reaffirms the current state. For each question, the gold answer gives the current state and the start date of its current uninterrupted period.
Data validation. We automatically checked timestamps, candidate values, aliases, and consistency across the matched histories. Before selecting the final dataset, two LLMs from OpenAI and xAI independently reviewed all 1,064 candidate pairs and their proposed answers. They agreed on which history contained the actual revision in 1,045 pairs (98.21%). We excluded all pairs on which they disagreed. We then sampled 20% retained pairs for manual review. Reviewers checked whether the histories were semantically consistent, whether each question was answerable at its cutoff, and whether both gold fields were correct. The two independent reviewers flagged three pairs for ambiguous wording. Then we regenerate pairs for that ambiguous cases.
Questions across all ten revision types. Our original study used 30 correction–replacement pairs to ask three questions: What is true now? What was true at an earlier date? What did the user originally say? We now ask these questions on 60 existing chains covering all ten revision types, for a total of 180 histories and 540 questions per backbone.
Does wording affect the results? We rewrote each of the 180 histories above. We kept 99 rewrites that preserved the original dates and values and that a separate model judged to have the same meaning at every turn. For each of these 99 histories, we asked the same current-state question on the original dialogue and its rewrite. The table shows how many answers were correct.
After rewriting, Typed still answered more questions correctly than direct reading on both backbones.