- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Selective Evidence Reliance (SES), a metric that evaluates memory agents by measuring their sensitivity to required and irrelevant memories. Using a synthetic benchmark, the authors find that affect-augmented memory is particularly sensitive to irrelevant records, mainly through changes in cause attribution rather than actions. They further introduce a causal gate that constrains cause attribution, improving SES without affecting action accuracy.
Critical strengths
- The paper addresses a relevant problem in evaluating how memory agents use evidence beyond task success.
- The paper provides controlled comparisons that help examine how different memory designs affect agent behavior.
Critical weaknesses
-
Unclear presentation and overstated claims. The paper introduces multiple contributions, including a memory architecture, an evaluation metric, a benchmark, and a causal intervention, but their relationships and the central research focus are not clearly established. Key concepts such as decision dependence, cause attribution, and internal-state leakage are not sufficiently distinguished, making the overall argument difficult to follow. Moreover, several claims about causal mechanisms and repairing evidence dependence appear stronger than what the experiments actually demonstrate.
-
Limited methodological novelty. The proposed SES metric essentially relies on repeated inference with relevant and irrelevant memory ablations, followed by comparisons of output changes. The causal gate is a simple post-processing rule that directly constrains the cause-attribution field using oracle evidence. While the evaluation perspective may be useful, the technical contributions appear relatively incremental, and their distinctions from existing intervention-based evaluation methods are not sufficiently established.
-
Narrow research scope. Despite motivating a general problem of evidence dependence in memory agents, the paper primarily investigates cause-attribution instability in a specifically constructed affect-augmented architecture. Since the observed failures and proposed repair largely concern a single output field rather than actual decision-making, the broader significance of the findings remains unclear.
-
Limited experimental validation. The evaluation mainly uses synthetic scenarios with saturated action accuracy and evidence coverage, making it difficult to assess the practical value of the proposed metric and method. The comparisons also rely on simplified proxies of existing memory architectures, while cross-model evaluations are limited and not conducted under fully consistent configurations. Stronger baselines and more realistic agent tasks would help establish the generalizability of the findings.
Questions for authors
see Weakness