Attention-Derived Memory-Use Traces for Agent Memory Refinement

Self-evolving agent memory is usually updated from task outcomes, execution trajectories, and post-hoc textual reflections. These signals can guide memory writing, but they provide limited evidence about how retrieved memory was actually used in the decisions made during execution. Without such read-side evidence, it is difficult to distinguish among different causes of memory-related errors. We introduce execution-time memory utilization as a complementary signal for persistent memory refinement. To estimate this utilization, we turn to attention as an indicator of memory access. Because raw attention is distributed across layers and heads, remains at the token level, and is not aligned with semantic memory segments or agent decisions, we identify model-specific retrieval heads and aggregate their attention into a segment-by-decision memory-use trace. Intervention experiments provide temporal and segment-level evidence that this trace reflects memory utilization. We then apply this signal in Attention-Guided Memory Refinement (AGMR), a simple refinement procedure that compares the utilization expected from reflection with the utilization observed in the trace to propose targeted edits. Across three interactive benchmarks and two models, AGMR improves task performance while generally reducing memory length.