MemCanvas: Visual Long-Term Memory for MLLM-based Agents

Despite rapid advances in multimodal large language models (MLLMs), agents built upon them remain fundamentally stateless, processing each interaction in isolation with no capacity to retain or reuse experiential knowledge. Existing lifelong memory systems, whether parametric or retrieval-based, operate predominantly in text, discarding the rich visual structure of images, tables, and diagrams or reducing them to lossy textual descriptions. We present MemCanvas, a training-free, model-agnostic visual memory framework that stores heterogeneous multimodal experiences as structured canvas images, allowing agents to accumulate and retrieve knowledge through their native visual perception channel. MemCanvas introduces three coordinated mechanisms: an adaptive layout algorithm that compresses and assembles multimodal elements into structured visual canvases, a hybrid retrieval system that fuses visual and textual encoder embeddings for cross-modal memory search, and a frequency-adaptive consolidation strategy that progressively degrades rarely-accessed memories to curb storage growth. Extensive experiments across five benchmarks with two backbones show that MemCanvas improves direct inference by +17.5 EM on HotpotQA, outperforms text-based memory baselines (Mem0, MemVerse, A-Mem, Text-RAG) without task-specific training—except on purely text-only questions, where text stores remain preferable—and matches the best classical eviction strategy per retained memory at matched retention while degrading memories progressively instead of deleting them. Hybrid retrieval and progressive visual forgetting further improved the practicality of visual memory, showing that reducing the language tax need not require retraining or unbounded storage. Together, these results suggest that the dominant text-first memory stack is not merely inefficient but mismatched to multimodal reasoning, and that preserving experience in visual form can improve long-horizon agent performance.