acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Representation Memory: Decoupling Structural Topology and Semantics for Lifelong Agents

Abstract

Equipping agents with long-term memory raises the challenge of representation entanglement. Storing encoder-specific vectors ties retrieval to a particular indexing model. We propose Intrinsic Representation Memory (IRM), a framework that decouples structural memory from semantic representation. Instead of generating text descriptions or storing frozen embeddings, IRM constructs a vector-free symbolic graph from a model's intrinsic attention salience during a standard forward pass. This zero-generation write retains entity links as inspectable symbolic memory. At read time, the current model reuses this graph and dynamically regenerates value representations within its own semantic space. IRM extends to multimodal settings, processing both text and raw image pixels without a separate captioner or embedder. Across four models, cross-built graphs retain at least 94.8% of each reader's own-graph text recall. With a common frozen 9B answer reader, IRM improves NarrativeQA F1 by 5.56 points over HippoRAG-2, with lower F1 on MuSiQue and mixed results on LoCoMo. In a two-memory study using the same Qwen model, construction is more than an order of magnitude faster than HippoRAG-2 under the measured implementations, while per-query latency is higher. By eliminating symbolic graph reconstruction during reader changes and accelerating memory writes, IRM offers a reader-portable memory foundation for continually evolving multimodal agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent Memory, Long-Term Memory, Retrieval-Augmented Generation, Graph-Based Retrieval, Model-Agnostic Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qtMvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose Intrinsic Representation Memory (IRM), a framework that decouples structural memory from semantic representation. Instead of generating text descriptions or storing frozen embeddings, IRM constructs a vector-free symbolic graph from a model’s intrinsic attention salience during a standard forward pass. IRM improves NarrativeQA F1 by 5.56 points over HippoRAG-2, with lower F1 on MuSiQue and mixed results on LoCoMo.

Critical strengths

S1: This paper has proposed an interesting question, and use a novel method as a solution. Using a model's intrinsic attention salience is a meaningful line of thought. S2: The paper conducted multiple experiments and honestly presented the results, including its weaker aspects. S3: The papaer has practical real-world value, substantially reducing the write cost. Also, the symbolic storage consists purely of integers and strings, supporting exact deletion and restoration (13.1 MB), which a clean engineering property.

Critical weaknesses

W1: Core innovation contribution is in doubt. The ablation of α∥v∥ edges vs. co-occurrence edges yields only +0.85 R@10 and +1.44 F1, and the paper itself concedes that "this comparison does not establish the superiority of salience-based connectivity." This is one of the paper's core selling points, yet the experimental evidence is weak. W2: Every reader switch requires re-selecting the (layer, KV head) pair on HotpotQA. The logistic probe must be re-fit on 600 external passages and the actual deployment cost contradicts the "zero-generation" promise. W3:Head selection uses retrieval AUC rather than relation extraction AUC, the correlation between the two is not demonstrated.

Questions for authors

Q1: Can you explain why IRM trails in some benchmark? MuSiQue: IRM trails by 7 points, LoCoMo IRM trails by 2.02 points. Q2: See in Weakness.

1 comment

The main concern has been addressed in detail in the authors’ response. I will maintain the score.

Reviewer_q5JaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper proposes IRM, a lifelong memory design that stores a vector-free symbolic graph at write time and regenerates semantics at read time. The core idea is to separate topology from reader-specific embeddings, so a new reader can reuse the same store. Experiments on MuSiQue, NarrativeQA, LoCoMo, and MemGallery show mixed accuracy, strong recall retention across builders and readers, and large write-time speedups under their measured stack.

Critical strengths
  1. The paper sets up an interesting cost trade: move part of the work to query/read time, and make memory construction much faster.
  2. The graph still works after a model swap. Across builder×reader pairs, text recall stays high.
Critical weaknesses
  1. The cost trade cuts both ways. Write is cheaper, but each query is slower. Similarity also needs a full value re-encode of the corpus into a session cache. So 'no stored vectors' does not mean 'no vector cost', it means the cost is moved to read/prep time. The symbolic graph is mostly a thin router: entity strings and links, while meaning stays in the raw text. The abstract presents attention-salience write as a main idea, but edge weights are dropped and the walk is undirected co-occurrence, and ablations show salience adds little over co-occurrence.
  2. A model swap is not free. The symbolic graph can be kept, but the new reader still needs new comparison heads, a new entity probe, and a full corpus value re-prep. The paper does not report the cost, timing tables only measure queries on an already-ready memory. Overall answer quality is also not clearly strong: on some benchmarks the result gets worse (e.g., same-reader MuSiQue vs HippoRAG-2; LoCoMo overall vs DRAGON).
Questions for authors
  1. For a full model swap, what is the wall-clock cost of head selection, entity-probe fitting, and corpus value re-prep, relative to rebuilding a HippoRAG-2 memory?
  2. After excluding query-likelihood, how do per-query latency and answer quality compare with HippoRAG-2 under the same reader?
  3. On LoCoMo temporal and adversarial abstention, where do most failures come from: seed miss, graph expansion, or final reader errors?
  4. When the reader changes, is the old session value cache discarded entirely, and how often must the corpus be re-prepped in a long-running agent?
  5. Across tasks of different lengths (short dialogues vs long corpora), what is the end-to-end wall-clock comparison of IRM and HippoRAG-2? That is to say, write + reader prep + per-query retrieve/answer, not only write time or ready-memory query time?
Reviewer_HpsPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a new RAG method that constructing a vector-free symbolic graph from a model's intrinsic attention salience during a standard forward pass. Using ordinary RAG method with embedding similarity to match key words and PPR on the graph, it reaches good performance on HippoRAG-2, MuSiQue, and LoCoMo.

Critical strengths
  1. Using the attention for building Graph is a new method.
Critical weaknesses
  1. The method is still largely follows the graphRAG method, with only the graph construction process differ.
  2. Using attention to build graph can be expensive than previous building method.
  3. The performance gain is not significant.
Questions for authors
  1. How to set the chunk size when writing? Large can be expensive due to the quadratic complexity of attention, while small chunk size can lose many semantic connection.
1 comment
AuthorsOpenReview ↗

Response to Reviewer HpsP, follow-up

This comment briefly recaps our response of 9 Nov to your concerns, together with what we explained to the other reviewers.

  • Difference from GraphRAG. GraphRAG generates text descriptions and searches with embeddings. IRM stores only selected spans, links and raw text, and finds candidates with graph search and BM25, without embedding similarity. With re-ranking and answering fixed, the IRM graph gives 37.34 MuSiQue F1 against 34.41 for BM25 candidates. (Sec. 4.5, Table 5.)
  • Construction cost. With the same model and engine, IRM builds the memory of one LoCoMo conversation in 68.6 s, and HippoRAG-2 in 2,931 s. (Supplement Sec. D.1, Table S9.)
  • Size of the gains. With the same answer model, IRM scores higher on NarrativeQA (26.91 against 23.32), LoCoMo (50.70 against 44.72) and MemGallery (66.82 against 59.77). HippoRAG-2 scores higher on MuSiQue. (Sec. 4.2, Table 2.)
  • Chunk size. IRM uses each benchmark's own input units, with median lengths of 45 to 132 tokens. Attention is computed only within each unit, so its cost per unit does not grow with the corpus, and units that contain the same stored node are connected through it. (Supplement Sec. A.1, Table S1.)

The full list of changes is in our reply to Reviewer q5Ja. If any point remains unclear, please tell us and we will answer right away.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.