TraceMem: Diagnosing and Fixing Selective Evidence Dependence in Memory Agents

TraceMem is a structured memory architecture that represents typed evidence, commitments, affect state, and causal attributions explicitly. This richer state makes memory use more inspectable, but also raises a control question: do affect and causal decisions remain tied to task-relevant evidence, or are they influenced by irrelevant records retrieved alongside it? Existing accuracy, coverage, and citation metrics often cannot distinguish these cases, particularly when retrieval quotas cause static metrics to saturate. We introduce selective evidence dependence and the SES score, which compares a decision's sensitivity to removing required evidence with its sensitivity to removing a matched placebo record. We evaluate TraceMem and matched alternative memory designs in a deterministic text-world benchmark using paired interventions, hard evaluator-side labels, and field-level decomposition over actions, affect, and evidence causes. Structured memory improves evidence availability and validity over recency. SES then exposes a distinct TraceMem failure mode: removing an irrelevant record changes emotion-cause attribution even when the action remains correct. An explicit graph-memory variant does not reproduce this instability, localizing the effect to affect-state coupling rather than graph structure. A minimal causal-gating intervention constrains cause attribution to relevant evidence and removes the diagnosed instability without changing action accuracy. The results position TraceMem as a richer but more demanding memory design: it improves evidence organization and auditability, while introducing a measurable selective-dependence trade-off. More broadly, SES complements conventional memory-agent metrics by revealing whether structured internal states depend selectively on the evidence required for a decision.