acceptodds
Under review as a conference paper at ICLR 2027

NoisyUpdateBench: Diagnosing Risk-Sensitive Revision in Long-Term Memory Systems

Abstract

Long-term memory benchmarks increasingly test knowledge updates, stale beliefs, and historical recall. Yet they rarely isolate two distinct questions: what state a system believes, and which revision action is warranted when applications assign different costs to false supersession, missed updates, and deferral. We introduce NoisyUpdateBench, a controlled diagnostic benchmark of noisy and retracting state streams with latent ground truth, versioned current and historical queries, and evidence-identical risk clones that hold observations fixed while varying only application costs. We formalize memory revision as utility-conditioned sequential decision making and evaluate transparent controllers that separate belief estimation, expected-risk action selection, and candidate quarantine. On a precommitted synthetic held-out suite of 120 base streams, dynamic Bayesian filtering with expected-risk decisions achieved the lowest pooled normalized risk cost (NRC) among ten configurations (0.676), improving over last-write-wins (ΔNRC = −0.655, 95% paired CI [−0.833, −0.482]); for risk-aware controllers, the clones preserved identical beliefs while inducing cost-dependent actions. On a sealed, independently authored synthetic external suite of 60 cases, the same controller again improved over last-write-wins and its matched fixed-policy variant, while comparisons with strong weighted-vote baselines remained inconclusive. In raw-language Track B, the same decision layer with automatic extraction reduced NRC relative to naive overwrite, full-context prompting, and Mem0 (ΔNRC = −0.770, −0.698, and −1.241; all paired CIs excluded zero) and achieved 0.962 exact-match extraction F1, while its NRC gap to a gold-extraction reference remained inconclusive. Scenario winners and quarantine effects varied across datasets, precluding universal-superiority claims. These results position NoisyUpdateBench as a diagnostic framework for utility-conditioned memory revision rather than another test of update accuracy alone

Primary area:
datasets and benchmarks
Keywords:
long-term memory, memory revision, evaluation benchmarks, risk-sensitive decision making, Bayesian filtering, uncertainty quantification, selective prediction, language agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_piPzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces NoisyUpdateBench, a controlled benchmark for memory revision under noisy observations and changing facts. It evaluates identical evidence streams under different costs for false supersession, missed updates, and quarantine, aiming to separate belief estimation from revision decisions. The paper compares overwrite, weighted-vote, and Bayesian controllers through structured and raw-language evaluations, reporting revision costs, coverage, and current and historical query accuracy.

Critical strengths

S1. The paper addresses a practically meaningful problem: a memory system must consider both the available evidence and the consequences of different revision errors.

S2. Reusing identical evidence under different cost settings provides a controlled way to examine how costs affect revision decisions.

S3. The paper reports coverage and query accuracy alongside revision costs, making the trade-offs between different controllers easier to assess. The independently authored synthetic suite and raw-language evaluation also extend the initial structured experiments.

Critical weaknesses

W1. The main novelty is to evaluate identical evidence under different error costs. However, the evaluated estimators do not receive cost information, while the decision policies explicitly use those costs to select actions. Unchanged beliefs and different actions are therefore expected under this setup. The experiments do not sufficiently establish what additional diagnostic insights this design provides over existing memory evaluations.

W2. NRC measures the cost of revision actions but excludes query errors. Although the main controller achieves lower NRC, its current-query accuracy, counting unanswered queries as incorrect, and its historical-query accuracy are both lower than those of LWW. The paper does not sufficiently establish when the reduction in revision cost justifies this loss in answer quality.

W3. In Track B, only RA-SBF receives the error costs, while the main competing systems remain cost-agnostic. This makes it difficult to separate the benefit of cost-aware decisions from that of the Bayesian revision mechanism. The evaluation also lacks a cost-aware Weighted Vote baseline using the same automatic extractor, which would provide a more direct comparison of the revision layer.

W4. Each sequence in the main test set contains only 3–15 observations about a single binary fact, and source reliabilities are directly supplied. The independently authored suite introduces different application domains but retains a similar task structure. The evaluation therefore leaves important practical challenges untested, including interference between multiple facts, retrieval over long histories, and unknown source reliability.

W5. Summing the query counts in Table 10 gives 3,795, whereas the paper states that the cloned canonical suite contains 2,808 queries. The historical accuracy calculated from that table also differs from Table 6. The authors need to clarify whether query roles overlap and explain how these tables relate to the reported overall results.

Questions for authors

Q1. Can the authors provide a concrete example where evidence-identical cost clones reveal a failure in an existing memory system that conventional memory evaluations would miss?

Q2. Can the authors provide a held-out comparison using the same cost-aware decision wrapper for the competing systems, including a Weighted Vote baseline with the same automatic extractor?

Q3. Beyond the development analyses in Appendix K, how do the held-out conclusions change when query errors and quarantine duration are included in the evaluation?

Q4. How should the query counts and historical accuracy in Table 10 be reconciled with the stated dataset size and Table 6?

Reviewer_KJNkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NoisyUpdateBench, a cost-sensitive benchmark for evaluating long-term memory systems.NoisyUpdateBench separates belief, action, and quarantine, showing how costs and stream structure change the lowest-action-loss controller.

Critical strengths
  1. Clear and meaningful problem formulation: The paper introduces a meaningful new perspective by separating belief estimation, action selection, and quarantine in long-term memory revision.
  2. Transparent evaluation framework: The paper provides interpretable, cost-sensitive evaluation metrics that separately assess belief quality, action loss, error types, and coverage.
Critical weaknesses
  1. Limited realism and generalization. The benchmark relies mainly on short, synthetic binary state streams and evaluates only one Qwen3-8B/Mem0 configuration in the raw-language setting.
  2. Strong dependence on manually specified assumptions. The methods rely on externally supplied reliability scores and author-defined cost profiles, while the reported advantage is highly scenario-dependent and Weighted Vote remains competitive.
Questions for authors
  1. Generalization: How does NoisyUpdateBench perform on longer histories, richer state spaces, real conversational data, and memory systems beyond the single Qwen3-8B/Mem0 configuration?
  2. How sensitive are the results to the manually specified reliability scores and cost profiles
Reviewer_qZZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a benchmark called NoiseUpdateBench, a controlled benchmark that test updates with focus on costs for false supersession, missed updates, and quarantine.

Critical strengths
  • The proposed benchmark tackles the under-explored aspects of existing benchmark in long-term memory, the sequential error costs for changes.
Critical weaknesses
  • Main concern. The overall writing of the paper includes numerous expressions or phrases that are written with no clear definition or chosen with ambiguous words, which make hard to follow for authors who are not familiar with long-term memory systems.
    • Ex) They leave a narrower diagnostic question: given identical evidence and belief, does a controller change its action when only the error costs change? -> what is controller? what is error cost?
    • Ex) Aggregate QA accuracy cannot answer this because it conflates state estimation, decision policy, and the consequences of directional errors. -> what is directional error?
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.