acceptodds
Withdrawn from ICLR 2027

TAGGRAPH: Tag-Augmented Graphs for Graph Retrieval of Agent Persistent Histories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term memory lets LLM agents recall past interactions and remain consistent across sessions, but memory systems are hard to compare because they often vary in representation, indexing, retrieval, and evaluation. We present a controlled evaluation framework based on shared 5W-style conversational memories. Localized graph configurations traverse a common base graph; AdaptiveGraph adds chronological edges and Personalized PageRank diffusion. We also evaluate BM25 over the same extracted notes and OpenClaw as a raw-input external reference. Retrieval rankings vary across memory settings. On LongMemEval-S, AdaptiveGraph is the strongest graph configuration at 0.844 MRR, but BM25 reaches 0.867 and OpenClaw 0.880. On ATANT Core, localized graph traversal outperforms diffusion and BM25, whereas BM25 leads the stress rounds. Reducing LongMemEval-S within the tested range does not reproduce the ATANT diffusion penalty, but the smallest tested store remains larger than ATANT Core, so store size cannot be ruled out. The penalty also persists under a permissive content-match criterion. Vocabulary normalization and extraction quality substantially affect graph retrieval, and missing extraction tags are common among top-five misses. Retrieval strategies should therefore be evaluated jointly with the memory setting and against strong lexical baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
agents, retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LchMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript aims to address the lack of controlled experimental comparisons among different agent memory systems. To this end, the authors introduce an interpretable 5W-style representation of memory systems, which provides a unified interface for comparing different retrieval configurations over shared memory stores.

Critical strengths

The proposed 5W-style representation is reasonable, intuitive, and easy to follow. It provides a structured perspective for analyzing different memory configurations.

Critical weaknesses
  1. Limited depth of analysis and insufficiently insightful findings.

Due to the relatively shallow experimental analysis, the conclusions drawn from the study remain limited. For example, the observation that extraction quality constrains the performance of different graph-based configurations is somewhat expected.

A potentially valuable extension would be to implement and analyze several existing published agent memory architectures, categorize them under the proposed framework, and, if possible, incorporate them into the unified representation for systematic comparison. Such an analysis could reveal deeper insights into the design space of agent memory systems and strengthen the contribution of the work.

  1. The novelty of the 5W representation is unclear.

The 5W-style representation does not appear to be a novel way of modeling agent memory. For example, MemoCue: Empowering LLM-Based Agents for Human Memory Recall via Strategy-Guided Querying (2025) has already adopted a similar perspective for characterizing memory-related components. The authors should clarify the distinction between the proposed framework and existing formulations and discuss whether the contribution lies in the representation itself or in the systematic evaluation enabled by it.

Questions for authors
  1. Please address the critical concerns raised in the Critical Weaknesses section, especially regarding the limited analysis depth and the relationship to prior 5W-style memory representations.

  2. Please provide more details about the extraction models used in the experiments. Specifically:

  • Does GPT-OSS refer to the 20B model or the 120B model?
  • What exactly does "Qwen-Small" refer to? Is it from the Qwen3 series or the Qwen2.5 series? What is the parameter scale?
  • Which specific Gemma model is used?
  1. There appear to be two missing references in the REPRODUCIBILITY STATEMENT section. Please verify and complete the citations.
Reviewer_iJJMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an empirical results showing that graph generated memory doesn't necessarily benefit memory retrieval. It uses an LLM to extract 5W-style information—Who, What, When, Where, and Why—and constructs a graph linking hierarchical tags, topics, and interactions. The evaluation compares localized graph traversal, vocabulary-normalized and TF-IDF-weighted variants, and an AdaptiveGraph configuration incorporating chronological links and Personalized PageRank, alongside BM25 over the same extracted notes. The results seem to show that BM25 is a strong baseline overall.

Critical strengths
  1. evaluation framework that compares graph-based memory effectiveness.
Critical weaknesses

While the paper presents a tag-augmented graph framework, its empirical finding is that lexical retrieval remains competitive or superior across most evaluated settings(?). This could support a empirical study, but the analysis does not yet sufficiently explain when graph structure helps, when it hurts, or why. The presentation would benefit from a sharper distinction between established findings, descriptive hypotheses, and implementation choices.

Questions for authors

n/A

Reviewer_KcgJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies retrieval from long-term conversational memory for LLM agents and argues that published graph-memory systems vary representation, indexing, traversal, and scoring together, which makes it impossible to tell whether a gain comes from graph structure or from a strong lexical component. It proposes a controlled comparison over a shared memory representation. Every user-assistant interaction is stored as one Markdown document, which remains the atomic retrieval unit. A language model extracts a 5W+Extra representation (Who, What, When, Where, Why, Extra, plus a summary) and short hierarchical ontology tags; these tags, together with topic nodes and file nodes, form a directed base graph with tag-topic, topic-file, topic-similarity, and ontology edges. Four graph configurations traverse this graph: TagGraph-Basic uses raw lexical tag overlap; TagGraph-Stopwords removes query tokens appearing in more than 15% of a case's files before ontology matching; TagGraph-TFIDF weights overlap by corpus TF-IDF; and AdaptiveGraph adds chronological file edges, seeds both tag and file nodes, and ranks with Personalized PageRank. The controlled lexical baseline is Okapi BM25 over the same extracted notes, with no ontology or graph; OpenClaw, which indexes raw conversations with a dense-plus-BM25 hybrid, is an external reference rather than a controlled ablation. Evaluation uses LongMemEval-S (500 questions, distractor-heavy multi-session histories, session-level MRR) and ATANT (compact persistent narratives, Core split and Stress R2-R5, substring-matched relevance), with three extraction models (Gemma, GPT-OSS, Qwen3-Small). On LongMemEval-S, AdaptiveGraph is the strongest graph configuration at 0.844 MRR with Gemma, but BM25 over the same notes reaches 0.867 and OpenClaw on raw text reaches 0.880. On ATANT Core, localized traversal leads at about 0.70-0.73 while AdaptiveGraph falls to about 0.55 and BM25 sits at 0.63; BM25 leads the stress rounds. The paper reports that stopword filtering and TF-IDF weighting improve localized traversal substantially, that extraction-model choice changes MRR more than the Basic-to-TFIDF progression, that a distractor sweep shows diffusion's benefit is distractor-dependent on LongMemEval-S, and that the ATANT diffusion penalty persists under permissive matching but is not reproduced by reducing LongMemEval-S store size within the tested range. A failure diagnostic finds that 651 of 658 pooled top-five misses lack a helpful tag under at least one extractor. The paper explicitly labels "topic dilution" a descriptive hypothesis and states that the mechanism of the ATANT penalty remains unidentified.

Critical strengths
  1. Since extraction is a large source of variance, showing that the ordering is stable across three extraction models is the right way to make the result credible.
  2. The distractor sweep and the component ablation produce a non-obvious interaction. With file seeding disabled at 36 distractors, adding diffusion raises MRR from 0.101 to 0.678, whereas its marginal contribution with seeding present is about 0.020 (lines 362–366; Table 3). The finding that diffusion and lexical file seeding are strongly substitutable in aggregate MRR is a useful design lesson, and the paper is explicit that it did not test whether they recover the same cases.
  3. The submission is reproducible by the standards of an empirical retrieval paper.
Critical weaknesses
  1. The main table omits the strongest controlled baseline, which is relegated to an appendix and often beats the graph configurations. Appendix F reports an "OC-notes" column: OpenClaw's hybrid retriever applied to the same 5W-extracted notes used by the graph methods, which the paper itself calls a controlled comparison with the same memory representation. In that table OC-notes outperforms AdaptiveGraph on several question types by large margins — for example 73.3/86.7/93.3 recall@1/3/5 versus 36.7/60.0/66.7 on single-session-preference under GPT-OSS extraction (Table 6, p. 16). Yet Table 1 compares the graph configurations only against BM25 over notes and OpenClaw over raw text, so the headline "graph versus lexical" comparison excludes the hybrid-on-notes baseline that is most informative about whether graph structure adds anything beyond dense-plus-BM25 on the same store. Including OC-notes in Table 1 would make the negative result far more convincing; omitting it leaves the impression that the strongest controlled alternative was measured but not reported at headline level.

  2. The paper does not establish that its graph pipeline is a competitive representative of graph memory, so the null result is ambiguous between "graph structure does not help" and "this graph implementation does not help". The four configurations differ only in lexical normalization and in whether PageRank diffusion and chronological edges are added over one hand-built ontology (lines 174–204). There is no tuning of the ontology construction, no replication of a published graph retriever, and no measurement showing that the graph's retrieval quality is in the range reported by HippoRAG, Zep, or MAGMA. The paper acknowledges that those systems are outside scope (lines 399–408), which is fair as a scope statement, but it does not acknowledge the inferential consequence: a reader cannot tell whether the BM25 win reflects a property of graph memory or of this particular extraction-and-traversal stack. A single reference implementation of one published graph retriever, run through the same interface, would resolve the ambiguity.

Questions for authors
  1. How competitive is the graph implementation? Would a reference implementation of HippoRAG or Zep, run through the same remember()/query() interface and evaluated on the same splits, reproduce the BM25 ordering? If not, the null result should be scoped to "this 5W tag graph" rather than to graph memory.
  2. What is the mechanism of the ATANT Core diffusion penalty? You say topic dilution is only a hypothesis and that you measured neither topic concentration nor score redistribution. Can you report at least one direct structural measurement — for example the distribution of PageRank mass across topics, or the number of similarly connected memories — to distinguish topic dilution from a store-size or degree-distribution effect?
  3. How sensitive are the results to the 5W extraction schema and the 15% stopword threshold? The paper acknowledges that extraction dominates retrieval performance. Did you vary the stopword threshold, the ontology prompt, or the number of extraction passes, and does the BM25-versus-graph ordering survive?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.