Intrinsic Representation Memory: Decoupling Structural Topology and Semantics for Lifelong Agents

Equipping agents with long-term memory raises the challenge of representation entanglement. Storing encoder-specific vectors ties retrieval to a particular indexing model. We propose Intrinsic Representation Memory (IRM), a framework that decouples structural memory from semantic representation. Instead of generating text descriptions or storing frozen embeddings, IRM constructs a vector-free symbolic graph from a model's intrinsic attention salience during a standard forward pass. This zero-generation write retains entity links as inspectable symbolic memory. At read time, the current model reuses this graph and dynamically regenerates value representations within its own semantic space. IRM extends to multimodal settings, processing both text and raw image pixels without a separate captioner or embedder. Across four models, cross-built graphs retain at least 94.8% of each reader's own-graph text recall. With a common frozen 9B answer reader, IRM improves NarrativeQA F1 by 5.56 points over HippoRAG-2, with lower F1 on MuSiQue and mixed results on LoCoMo. In a two-memory study using the same Qwen model, construction is more than an order of magnitude faster than HippoRAG-2 under the measured implementations, while per-query latency is higher. By eliminating symbolic graph reconstruction during reader changes and accelerating memory writes, IRM offers a reader-portable memory foundation for continually evolving multimodal agents.