- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces NoisyUpdateBench, a controlled benchmark for memory revision under noisy observations and changing facts. It evaluates identical evidence streams under different costs for false supersession, missed updates, and quarantine, aiming to separate belief estimation from revision decisions. The paper compares overwrite, weighted-vote, and Bayesian controllers through structured and raw-language evaluations, reporting revision costs, coverage, and current and historical query accuracy.
Critical strengths
S1. The paper addresses a practically meaningful problem: a memory system must consider both the available evidence and the consequences of different revision errors.
S2. Reusing identical evidence under different cost settings provides a controlled way to examine how costs affect revision decisions.
S3. The paper reports coverage and query accuracy alongside revision costs, making the trade-offs between different controllers easier to assess. The independently authored synthetic suite and raw-language evaluation also extend the initial structured experiments.
Critical weaknesses
W1. The main novelty is to evaluate identical evidence under different error costs. However, the evaluated estimators do not receive cost information, while the decision policies explicitly use those costs to select actions. Unchanged beliefs and different actions are therefore expected under this setup. The experiments do not sufficiently establish what additional diagnostic insights this design provides over existing memory evaluations.
W2. NRC measures the cost of revision actions but excludes query errors. Although the main controller achieves lower NRC, its current-query accuracy, counting unanswered queries as incorrect, and its historical-query accuracy are both lower than those of LWW. The paper does not sufficiently establish when the reduction in revision cost justifies this loss in answer quality.
W3. In Track B, only RA-SBF receives the error costs, while the main competing systems remain cost-agnostic. This makes it difficult to separate the benefit of cost-aware decisions from that of the Bayesian revision mechanism. The evaluation also lacks a cost-aware Weighted Vote baseline using the same automatic extractor, which would provide a more direct comparison of the revision layer.
W4. Each sequence in the main test set contains only 3–15 observations about a single binary fact, and source reliabilities are directly supplied. The independently authored suite introduces different application domains but retains a similar task structure. The evaluation therefore leaves important practical challenges untested, including interference between multiple facts, retrieval over long histories, and unknown source reliability.
W5. Summing the query counts in Table 10 gives 3,795, whereas the paper states that the cloned canonical suite contains 2,808 queries. The historical accuracy calculated from that table also differs from Table 6. The authors need to clarify whether query roles overlap and explain how these tables relate to the reported overall results.
Questions for authors
Q1. Can the authors provide a concrete example where evidence-identical cost clones reveal a failure in an existing memory system that conventional memory evaluations would miss?
Q2. Can the authors provide a held-out comparison using the same cost-aware decision wrapper for the competing systems, including a Weighted Vote baseline with the same automatic extractor?
Q3. Beyond the development analyses in Appendix K, how do the held-out conclusions change when query errors and quarantine duration are included in the evaluation?
Q4. How should the query counts and historical accuracy in Table 10 be reconciled with the stated dataset size and Table 6?