- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies memory refinement for LLM agents from a read-side perspective. The key idea is to use execution-time attention patterns to estimate how retrieved memory segments are actually utilized during decision making. The authors first identify a small set of retrieval-sensitive attention heads from transition cases where adding trajectory memory changes task outcomes from failure to success. Their attention is then aggregated into a segment-by-decision memory-use trace. Based on the discrepancy between semantically expected memory use and observed attention-derived utilization, the method categorizes failures into accessed-but-ineffective, missed-memory, and distraction cases, and applies different textual memory-editing strategies. The resulting AGMR framework is evaluated on ALFWorld, ScienceWorld, and WebShop.
Critical strengths
-
The paper introduces a useful read-side perspective on agent memory refinement. Existing memory refinement methods mainly rely on task outcomes, trajectories, and post-hoc semantic reflection, while this work explicitly incorporates information about how memory is accessed during execution. The idea of connecting memory reading and persistent memory writing is conceptually interesting.
-
The paper makes a serious effort to validate the attention-derived signal. The intervention experiments provide evidences that the selected heads are functionally related to memory-assisted decision making rather than being arbitrary correlates.
-
The empirical evaluation is reasonably comprehensive within the studied setting. AGMR consistently improves over trajectory-memory and text-only refinement baselines across multiple agent environments and two open-weight backbones, while often reducing memory length. The factor-matched text-only and random-head controls are particularly useful for isolating the contribution of the utilization trace.
Critical weaknesses
-
The necessity of the attention-based failure classification is not sufficiently established. The main novelty of the paper lies in identifying retrieval-sensitive heads and using their utilization trace to distinguish among several failure modes. However, after this diagnosis, all branches ultimately reduce to textual memory rewriting: correcting misleading content, increasing the salience of relevant content, or clarifying competing memories. The current experiments show that adding the utilization trace improves over the authors' text-only control, but they do not establish that explicitly classifying a failure into these cases is necessary. The current results demonstrate that attention-conditioned refinement is useful under the proposed updater, but do not yet show that the proposed diagnosis-and-routing mechanism is itself necessary or offers a fundamental advantage over a stronger diagnosis-free rewriter.
-
The reliability and generality of the identified retrieval heads remain unclear. The heads are selected from relatively small LLM-annotated supervision sets (47 cases for Qwen and 34 for Llama). Although the paper provides bootstrap stability and cross-benchmark transfer evidence, it is still unclear how robust the selected heads are to different supervision data, annotation procedures, memory structures, or model families.
Questions for authors
-
Could the authors comment on how AGMR would compare with a more flexible unified memory rewriter that can jointly revise relevant and potentially conflicting memories without explicitly assigning one of the three failure types?
-
It would also be useful to better understand the robustness of retrieval-head selection. Have the authors observed similar head sets or downstream performance when changing the supervision subset or the analyzer used to identify supporting memory segments?
2 comments
Part Ⅱ
W2 and Q2: Robustness of Retrieval-Head Selection
We thank the reviewer for asking about the robustness of retrieval-head selection to the supervision data and analyzer.
Human verification. The original supervision sets, containing 47 Qwen cases and 34 Llama cases, were manually checked by the authors and co-authors. Ambiguous or incorrect annotations were excluded or corrected before head selection.
Stability experiments already included in the paper. The original manuscript evaluates retrieval-head selection with 10,000 bootstrap resamples and 1,000 random disjoint half-splits. Bootstrap Jaccard measures overlap with the full-data Top-5 set, while half-split Jaccard and overlap counts compare the Top-5 sets selected independently from two non-overlapping halves of the supervision data.
| Stability metric | Qwen | Llama |
|---|---|---|
| 10,000 bootstrap resamples: mean Jaccard | 0.867 | 0.811 |
| 1,000 disjoint half-splits: mean Jaccard | 0.739 | 0.662 |
| Disjoint half-splits: mean number of shared heads | 4.22/5 | 4.01/5 |
To further address the reviewer's questions, we additionally tested a different analyzer and three new supervision sets, as detailed below.
Changing the analyzer. We replaced the supporting-segment analyzer with GPT-5.4, keeping the original cases fixed. GPT-5.4 re-annotated the supporting memory segments without seeing the original labels. For both Qwen2.5-7B and Llama-3.1-8B, the selected Top-5 heads and their rankings remained unchanged.
Changing the supervision set. We additionally randomly sampled three mutually disjoint sets of 47 ALFWorld training cases, all without overlap with the original Qwen supervision set. Reselecting Qwen's retrieval heads from each of the three sets recovered exactly the same Top-5 head set as the original selection, with only their relative rankings changing.
| Backbone | Supervision data | Analyzer | Top-5 overlap with original selection | Head ranking |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | Original 47 cases | GPT-5.4 | 5/5 | Unchanged |
| Llama-3.1-8B-Instruct | Original 34 cases | GPT-5.4 | 5/5 | Unchanged |
| Qwen2.5-7B-Instruct | New set 1: 47 cases | GPT-5.4 | 5/5 | Changed |
| Qwen2.5-7B-Instruct | New set 2: 47 cases | GPT-5.4 | 5/5 | Changed |
| Qwen2.5-7B-Instruct | New set 3: 47 cases | GPT-5.4 | 5/5 | Changed |
Part Ⅰ
Response to Reviewer 2Vfr
We sincerely thank the reviewer for recognizing the read-side perspective and the factor-matched controls, and for the constructive questions about explicit diagnosis and retrieval-head robustness.
W1 and Q1: Necessity of Three Explicit Failure Categories
Our existing text-only control already removes both the memory utilization trace and the explicit three-category diagnosis. Its comparison with AGMR, together with the intervention results in Figure 5 and the case studies in Appendix F (Figures 12 and 13), demonstrates the effectiveness of attention-guided, diagnosis-based memory refinement.
So we understand the reviewer's question as whether explicitly classifying failures is necessary when the memory utilization trace is already available. In other words, could a unified rewriter directly use the attention-derived utilization trace to jointly revise relevant and potentially conflicting memory segments, without explicitly assigning one of the three failure categories?
To investigate this, we introduce an Attention-conditioned Unified Rewriter and compare it with our existing Text-only control and AGMR:
The Attention-conditioned Unified Rewriter retains the same two-stage pipeline and Semantic Reflector as AGMR. The only modification is in the Refiner: instead of explicitly classifying the failure and applying category-specific editing strategies, we directly provide the full memory utilization matrix U to the LLM and allow it to freely determine how to jointly revise the memory.
The following comparison uses Qwen2.5-7B-Instruct on ALFWorld and ScienceWorld.
As shown above, the Attention-conditioned Unified Rewriter performs substantially worse than AGMR and even falls below the text-only control. This is perhaps unsurprising: providing an LLM with a raw utilization matrix alone does not necessarily tell it how to interpret the attention patterns or translate them into effective memory edits. Without an explicit diagnostic framework and corresponding editing strategies, the utilization trace may be underutilized or even introduce distracting signals.
In contrast, AGMR transforms the use of memory utilization traces from generation to discrimination. By explicitly classifying the observed attention patterns and mapping each category to a corresponding editing strategy, AGMR substantially reduces the burden on the LLM to interpret raw read-side evidence and independently derive appropriate editing actions. This structured diagnosis mechanism makes the utilization trace more actionable, leading to more effective and efficient memory refinement than unrestricted, attention-conditioned rewriting.