acceptodds
Under review as a conference paper at ICLR 2027

TraceMem: Diagnosing and Fixing Selective Evidence Dependence in Memory Agents

Abstract

TraceMem is a structured memory architecture that represents typed evidence, commitments, affect state, and causal attributions explicitly. This richer state makes memory use more inspectable, but also raises a control question: do affect and causal decisions remain tied to task-relevant evidence, or are they influenced by irrelevant records retrieved alongside it? Existing accuracy, coverage, and citation metrics often cannot distinguish these cases, particularly when retrieval quotas cause static metrics to saturate. We introduce selective evidence dependence and the SES score, which compares a decision's sensitivity to removing required evidence with its sensitivity to removing a matched placebo record. We evaluate TraceMem and matched alternative memory designs in a deterministic text-world benchmark using paired interventions, hard evaluator-side labels, and field-level decomposition over actions, affect, and evidence causes. Structured memory improves evidence availability and validity over recency. SES then exposes a distinct TraceMem failure mode: removing an irrelevant record changes emotion-cause attribution even when the action remains correct. An explicit graph-memory variant does not reproduce this instability, localizing the effect to affect-state coupling rather than graph structure. A minimal causal-gating intervention constrains cause attribution to relevant evidence and removes the diagnosed instability without changing action accuracy. The results position TraceMem as a richer but more demanding memory design: it improves evidence organization and auditability, while introducing a measurable selective-dependence trade-off. More broadly, SES complements conventional memory-agent metrics by revealing whether structured internal states depend selectively on the evidence required for a decision.

Primary area:
datasets and benchmarks
Keywords:
Memory Agents, Agent Evaluation, Evidence Grounding, Long-Context Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NonwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Selective Evidence Reliance (SES), a metric that evaluates memory agents by measuring their sensitivity to required and irrelevant memories. Using a synthetic benchmark, the authors find that affect-augmented memory is particularly sensitive to irrelevant records, mainly through changes in cause attribution rather than actions. They further introduce a causal gate that constrains cause attribution, improving SES without affecting action accuracy.

Critical strengths
  1. The paper addresses a relevant problem in evaluating how memory agents use evidence beyond task success.
  2. The paper provides controlled comparisons that help examine how different memory designs affect agent behavior.
Critical weaknesses
  1. Unclear presentation and overstated claims. The paper introduces multiple contributions, including a memory architecture, an evaluation metric, a benchmark, and a causal intervention, but their relationships and the central research focus are not clearly established. Key concepts such as decision dependence, cause attribution, and internal-state leakage are not sufficiently distinguished, making the overall argument difficult to follow. Moreover, several claims about causal mechanisms and repairing evidence dependence appear stronger than what the experiments actually demonstrate.

  2. Limited methodological novelty. The proposed SES metric essentially relies on repeated inference with relevant and irrelevant memory ablations, followed by comparisons of output changes. The causal gate is a simple post-processing rule that directly constrains the cause-attribution field using oracle evidence. While the evaluation perspective may be useful, the technical contributions appear relatively incremental, and their distinctions from existing intervention-based evaluation methods are not sufficiently established.

  3. Narrow research scope. Despite motivating a general problem of evidence dependence in memory agents, the paper primarily investigates cause-attribution instability in a specifically constructed affect-augmented architecture. Since the observed failures and proposed repair largely concern a single output field rather than actual decision-making, the broader significance of the findings remains unclear.

  4. Limited experimental validation. The evaluation mainly uses synthetic scenarios with saturated action accuracy and evidence coverage, making it difficult to assess the practical value of the proposed metric and method. The comparisons also rely on simplified proxies of existing memory architectures, while cross-model evaluations are limited and not conducted under fully consistent configurations. Stronger baselines and more realistic agent tasks would help establish the generalizability of the findings.

Questions for authors

see Weakness

Reviewer_xdSkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TraceMem, a structured-memory agent with typed evidence, commitments, affective state, and cause attribution, together with a deterministic text-world benchmark. Its selective evidence reliance score, SES = RS - PS, compares changes in structured decisions after removing required evidence versus a matched placebo record. The main experiments use 30 delayed-commitment scenarios and three seeds, showing that static coverage can saturate while the cause-attribution field remains sensitive to placebo removal. An oracle post-decoding gate overwrites the cause identifier with benchmark-annotated evidence, increasing SES from 0.289 to 0.778 in a separate paired run while leaving action accuracy at 1.0. The paper also reports a non-oracle evidence-consistency gate, integration probes, and supplementary cross-provider results.

Critical strengths
  1. The motivation addresses a meaningful evaluation gap. Distinguishing evidence visibility and correct actions from selective use of relevant information is a worthwhile objective for memory agents. The paper makes this distinction concrete through examples where the action remains correct while other decision fields change under removal of a competing record. This motivates inspecting more than accuracy and retrieval coverage, even though the proposed interpretation of those changes requires stronger validation.

  2. The paired audit and supporting documentation make the experiments inspectable. The factual, required-removal, and placebo-removal branches provide an accessible experimental framework, and the field-wise decomposition identifies which observable output fields contribute to the composite score. The equal-length substitution control is a useful attempt to separate record-content effects from shortening an individual prompt. The appendix supplies verbatim prompts, concrete decisions, per-seed and per-scenario summaries, and configuration and resource tables. The paper also distinguishes the oracle gate from the non-oracle variant and identifies the ActMem-style baseline as a proxy. These details help readers assess the experimental scope and limitations.

Critical weaknesses
  1. The score and gate do not establish the claimed causal diagnosis or repair. Equations (1)-(2) count a change in any one of action, emotion, or cause_id as a changed decision. Consequently, RS = 1 does not imply that the action depends on the required evidence, despite the interpretation in Section 4.2: an evidence-insensitive action with a changing cause label can satisfy this criterion. Likewise, the placebo is a competing instruction about the same item; its irrelevance to the correct destination does not automatically establish irrelevance to the agent's emotional state or explanation. The paper needs to validate the intended invariance of each scored field.

The oracle gate then overwrites cause_id with a benchmark-provided identifier after decoding. Since cause_id is itself part of the change indicator, eliminating its placebo variation directly removes a component of the measured instability. Preserving action accuracy is guaranteed by leaving the action untouched. I recognize that Section 3.3 explicitly presents this as an oracle diagnostic probe; nevertheless, the intervention demonstrates stabilization of a reported output field without establishing that the underlying evidence-use mechanism has been repaired or that affective coupling is the unique source of the instability. A constant-cause control, independent cause-correctness evaluation, and evidence of consequences for subsequent decisions are needed to distinguish meaningful grounding from improved output consistency. This is a foundational concern about what the central result establishes.

  1. The comparisons do not isolate affective coupling, and the supporting validation is insufficient. Appendix C.5 shows that the proxy, graph-memory, and TraceMem prompts differ in serialization and state presentation. Equal retrieval quotas do not ensure matched prompts: Table 22 reports 356 prompt tokens for the proxy versus 232 for TraceMem. The equal-length placebo substitution controls a within-prompt deletion effect, but does not isolate the between-architecture differences. The paper therefore needs an affect-on/off comparison with the remaining serialization and verification behavior fixed before attributing the gap specifically to affective coupling.

Endpoint variability is another unresolved confound. Section 4.4 acknowledges changes between repeated runs even at temperature 0, and Appendix B describes a repeated-call noise-floor procedure, but the supplied PDF provides no numerical unchanged-prompt flip rates. Those rates are necessary to separate intervention-induced changes from ordinary output variability. Furthermore, Appendix C.1 reports the opposite descriptive architecture ranking on Kimi: TraceMem SES is 0.594 versus 0.400 for the proxy, compared with 0.233 versus 0.711 on the main endpoint. The non-oracle gate only changes the Kimi mean from 0.594 to 0.606, with no reported paired interval for that improvement. The authors acknowledge configuration changes, so these results are not a controlled refutation of the main experiment; they also do not validate a stable architectural diagnosis or a reliable deployable repair. Resolving these issues requires additional controlled experiments.

  1. The figures and typesetting are insufficiently polished and hinder interpretation. The visual presentation also needs substantial revision. Figures 3, 5, and 8 use small axis labels, legends, and annotations relative to the body text, with crowded multi-panel layouts. In Figure 3, the caption mentions 95% bootstrap intervals, but the uncertainty encoding is not clearly distinguished from the RS/PS endpoints and connecting segments. Figure 8's caption describes mean latency and token usage, whereas the panels show distributions and several statistical markers whose meanings are not explained. Several small tables in Appendix C.1 (page 14) lack individual numbered captions. Appendix D.5 (page 27) also contains joined headings and body text, such as "Datasets.Deterministically" and "Decoding.Non-thinking". These legibility, caption-consistency, and typesetting problems make the experimental evidence harder to inspect
Questions for authors
  1. What independent criterion establishes that the placebo should leave the emotion and cause fields unchanged, as well as the action? Can the oracle and non-oracle gates be compared against a constant-cause control using cause correctness or subsequent decision quality, in addition to SES? Please also clarify how RS over the joint output schema supports the action-dependence claim in Section 4.2.

  2. Please report the numerical unchanged-prompt noise floor for each method and field, and any affect-on/off ablation that holds the remaining prompt and verifier fixed. Under a frozen endpoint and decoding configuration, does the architectural ranking persist, and is the non-oracle gate's paired improvement distinguishable from variability? These results could materially affect the assessment.

  3. Like a Vibe Research project without human polishment and review.

Reviewer_xzkNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether memory agents selectively depend on relevant evidence, even when accuracy and retrieval coverage are saturated. It introduces selective evidence reliance, SES = RS − PS, comparing changes in structured decisions after removing required evidence. On 30 delayed-commitment scenarios evaluated across three seeds, TraceMem exhibits substantially higher placebo sensitivity than an ActMem-style proxy, primarily through changes in its reported cause identifier. An oracle post-processing gate reduces this sensitivity while preserving action accuracy.

Critical strengths

In the main DeepSeek experiment, placebo sensitivity differs markedly between TraceMem and the proxy: 0.767 versus 0.289. The cause-field decomposition is informative even if the mechanism remains unresolved.

Critical weaknesses
  1. Equation 1 treats any change in action, emotion, or cause identifier as a changed decision. However, a record irrelevant to the correct delivery destination need not be irrelevant to an agent’s emotion or explanation. The “placebo” is a competing destination statement about the same item; its presence can reasonably introduce conflict or uncertainty without changing the correct action.

  2. The experiments do not isolate affective coupling as the causal mechanism. The compared conditions differ in several respects, including affect-state text, graph headers and edges, serialization, and verification. Matching record content and retrieval quotas does not hold these factors constant. Indeed, Table 22 reports 356 prompt tokens for the proxy and 232 for TraceMem, contradicting the suggestion that equal quotas establish comparable prompt length.

Questions for authors
  1. What are the identical-prompt repeat flip rates for each condition and output field, and how large are the intervention effects relative to those rates?

  2. Does disabling only affective coupling reproduce the proxy’s behavior when all other prompt and verifier details remain fixed?

  3. How is placebo irrelevance established separately for action, emotion, and cause attribution?

  4. How do constant-cause, oracle, and evidence-only gates compare on identical saved completions, measuring both stability and attribution correctness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.