acceptodds
Under review as a conference paper at ICLR 2027

Beyond Query-Memory Matching: Path-Aware Anchor-Guided Retrieval for Long-Term Agent Memory

Abstract

LLM-based agents often rely on Retrieval-Augmented Generation (RAG)-style memory systems to leverage historical experiences. However, existing methods typically rely on direct query–memory similarity, implicitly assuming that gold evidence memories are close to the query in embedding space. This assumption overlooks the semantic gap between user queries and fine-grained agent memories, making query-memory similarity alone insufficient for reliable evidence recall. To address this limitation, we propose **PAGR**, a **P**ath-Aware **A**nchor-**G**uided Evidence **R**etrieval framework. Inspired by the spreading activation mechanism in human associative memory, **PAGR** first retrieves a small set of seed memories as initial anchors. It then expands from these anchors through both global semantic associations and local temporal neighbors. A path-aware cross-encoder selects candidate memories that complement the current retrieval path. The selected memories are accumulated as evidence and reused as anchors for subsequent hops, enabling iterative memory exploration. Extensive experiments on the LoCoMo benchmark demonstrate that **PAGR** consistently outperforms all baselines, achieving the highest overall F1 scores of 45.2 with GPT-4o-mini and 41.8 with Llama-3.1-8B-Instruct, while maintaining token efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LLM Agent, long-term memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ibHYOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that query–memory embedding similarity fails to recall gold evidence in long-term agent memory because of a semantic gap between high-level queries and fine-grained episodic memories. It proposes PAGR, a training-free retrieval procedure:

  • Retrieve top-K seed memories as anchors.
  • Expand each anchor to its semantic neighbors (memory–memory cosine) and temporal neighbors (a same-session window).
  • Score each candidate with a cross-encoder conditioned on the parent memory, CE(q, p ⊕ c).
  • Keep the top-B candidates as evidence and as the next-hop anchors, for up to T hops.

On LoCoMo it reports 45.2 / 41.8 overall F1 with GPT-4o-mini / Llama-3.1-8B. It also includes smaller comparisons against A-MEM on GVD and PerLTQA, plus ablations, a case study and an efficiency analysis.

Critical strengths
  • The motivation is clear and well-grounded (Fig. 2): even top-50 retrieval misses gold memories for many LoCoMo queries.
  • The method is simple, training-free, and easy to plug into existing memory systems. Temporal-window expansion is a cheap and sensible inductive bias for conversational memory.
  • The ablations go beyond end-task F1.
  • The open-domain failure analysis (Table 3) and the qualitative traces (Fig. 4) are informative.
Critical weaknesses
  • Limited novelty and missing related work: Seed retrieval → neighbor expansion → reranking → iterate is close to beam-style multi-hop retrieval: PathRetriever (Asai et al., 2020), MDR (Xiong et al., 2021), Baleen, IRCoT. Graph-based propagation (HippoRAG, GraphRAG) is another close relative. Contribution 1 is already discussed in prior work.
  • Missing or weak baselines: MemoryOS, Nemori, Memory-R1, and many other related papers.
  • Limited evaluation: Main results use only LoCoMo (10 dialogues), with lexical F1/BLEU only and no LLM-as-judge. There are no multiple runs, variance, or significance tests.
  • The abstract and conclusion mention only LoCoMo, but the performance on that dataset is not as good as recent papers.
Questions for authors
  • Can you report results with LLM-as-judge metrics?
Reviewer_KFyYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PAGR, a path-aware retrieval method for agent memory. The core idea is to expand from seed memories through semantic and temporal neighbors, then score paths with a cross-encoder. Experiments raise overall LoCoMo F1 and beat A-MEM on some extra sets, with ablations on expansion and path scoring.

Critical strengths
  1. The paper target to solve a common problem: similarity-only retrieval often misses the most direct evidence. It proposes a multi-round retrieval method, and the gains look reasonably solid.
  2. The evaluation is fairly broad and includes token cost. That makes the method can be used in a real agent system.
Critical weaknesses
  1. The paper mainly compares against similarity-only retrieval. Real systems often mix sparse, dense, keyword, or graph retrieval. There is no analysis of hybrid first-stage retrievers, so it is unclear whether PAGR still helps when the base retriever is already stronger than plain cosine top-k.
  2. Multi-step expansion is common used, some typical errors exist: bad seed initialization that traps the search, path drift when new neighbors pull away from the goal, or noise when new results are merged into the evidence set. Without thorough discussion on these failure mode, it is hard to know when the method fails and why.
Questions for authors
  1. if the seed retriever is hybrid (e.g., BM25 + dense, or keyword + embedding), does path expansion still improve recall and answer F1?
  2. How often do failures come from bad seeds, path drift, or noisy evidence aggregation?
  3. Is there a deletion mechanism in the proposed method? and if drifted memories are deleted or down-weighted, how does the method prevent a removed item from being expanded again in a later hop?
Reviewer_3SxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When retrieving past memories to answer a question, retrieval based on direct query–memory similarity may miss useful evidence due to the semantic gap between user queries and specific memories. This paper proposes PAGR, a retrieval method that uses initially retrieved memories as anchors to find additional evidence. For each anchor, it expands the candidate pool with semantically similar memories and nearby turns from the same conversation. A cross-encoder ranks candidates by considering the question together with the anchor and candidate memories. The selected memories are added to the evidence set and used as anchors for further retrieval. The authors evaluate the method on LoCoMo using GPT-4o-mini and Llama-3.1-8B-Instruct, with additional comparisons on GVD and PerLTQA.

Critical strengths

Novel memory retrieval for long-term memory. PAGR uses semantic and temporal connections between memories to recover evidence missed by direct query–memory matching. Specifically, it uses retrieved memories as anchors to explore semantically related memories and nearby conversational turns over multiple rounds. This can help recover evidence scattered across long conversation histories that direct query–memory matching may miss.

Critical weaknesses

Limited validation of complementarity and redundancy reduction.

  1. The ablations show performance gains, but do not clearly establish whether iterative exploration finds complementary evidence or mainly provides more context for reranking. A high anchor–candidate score may also reflect the anchor’s relevance, even when the candidate adds little useful information.
  2. Existing metrics like coverage and rescue rate do not directly show that the selected memories contribute distinct answer-relevant information or contain less repeated content.
Questions for authors

Source of the gains.

  1. Could you compare the candidate pool collected through iterative exploration with the topK memories retrieved directly by query–memory similarity, keeping the number of unique memories the same? How much do the two pools overlap, and does iterative exploration recover more gold evidence?
  2. Could you also compare anchor-only scores with anchor–candidate scores to check whether high scores reflect useful information added by the candidate?
  3. Could you also compare redundancy in the final selected memory sets selected by PAGR and the baselines?

Cross-encoder.

  1. Which cross-encoder model is used, and was it fine-tuned for this task?
  2. How does the cross-encoder distinguish a candidate that repeats facts already contained in its anchor from one that adds useful new evidence?

Case studies.

  1. Could you extend the examples in Figure 4 to compare the memories retrieved by PAGR and the baselines for the same questions, highlighting useful evidence, repeated information, and missed evidence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.