- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the problem that existing memory systems often construct query-independent representations and make early decisions, which may lead to irreversible evidence loss. The paper introduces Distributed Evidence Memory (DEM), a framework that preserves potentially relevant evidence and defers irreversible decisions until query-conditioned information becomes available. The framework consists of three main components:
- Raw-memory Shards: Enable distributed access to original memory episodes and retrieve query-conditioned evidence through a shared workspace.
- Conflict Directed Acyclic Graph (CDAG): Preserves conflicting memory versions for query-time aggregation instead of prematurely discarding them.
- Good-Turing Coverage Estimation (GTC): Provides an evidence-based adaptive stopping mechanism to determine when sufficient evidence has been collected.
Experiments on FactCon, LoCoMo, and LongMemEval show that DEM outperforms existing retrieval and memory baselines. Additional analyses suggest that improved evidence coverage accounts for a substantial portion of the performance gains.
Critical strengths
- Interesting motivation and clear design principle. The paper identifies irreversible evidence loss as an important challenge in long-term agent memory and presents a coherent framework for deferring memory decisions until query-conditioned evidence becomes available.
- Comprehensive evaluation and strong empirical results. The paper evaluates DEM across three benchmarks and seven backbones, with substantial performance improvements. The extensive ablations and mechanism analyses provide useful insights into evidence coverage, multi-hop retrieval, and conflict preservation.
Critical weaknesses
- The statistical validity of GTC remains unclear, as adaptive multi-hop retrieval produces dependent and non-stationary evidence observations that may violate the assumptions underlying Good-Turing estimation.
- Although DEM aims to defer irreversible memory decisions, its initial shard activation remains irreversible, as subsequent evidence-guided retrieval is restricted to the fixed activated set. This may prevent the discovery of relevant evidence in previously inactive shards, particularly in multi-hop queries.
- Although DEM consistently improves SubEM across seven backbones, some models exhibit substantial declines in EM and F1 (e.g., Qwen3.5-9B: EM 31.0→10.0, F1 33.6→25.0), raising concerns about whether the reported SubEM gains consistently reflect improved answer correctness rather than more verbose or ambiguous outputs.
- DEM's gains are not sufficiently separated from its 2.7× compute cost, and the necessity of CDAG over simpler version-preserving methods remains unclear.
- CDAG's conflict definition is overly simplistic, as different values for the same subject–relation pair may represent compatible facts rather than genuine conflicts.
Questions for authors
- Could the authors clarify GTC assumptions and provide an experiment examining the relationship between GTC's estimated unseen mass and the actual remaining ground-truth evidence, particularly when the stopping criterion is triggered?
- Could the authors report the proportion of ground-truth evidence covered by the initially activated shards and compare against a dynamic shard-reactivation strategy under matched computational budgets?
- Could the authors provide compute-matched comparisons and evaluate CDAG against a simple timestamped multi-version baseline?
- Could the authors clarify how CDAG handles multi-valued relations and temporal validity, and evaluate its robustness to incorrectly identified conflicts?