- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MemCanvas, a training-free visual long-term memory framework for multimodal language model agents. Instead of representing stored experiences primarily as text, MemCanvas organizes heterogeneous information into structured image canvases that can be processed by the visual encoder of an MLLM. The system consists of an adaptive layout mechanism, hybrid text-image retrieval, and frequency-adaptive memory consolidation through progressive resolution reduction. Experiments on several question-answering benchmarks show improvements over direct inference and a number of text-based memory and retrieval approaches. The paper also investigates retrieval efficiency, layout choices, storage costs, and memory consolidation.
Critical strengths
-
The paper presents a practical and intuitive alternative to text-centric agent memory. Storing multimodal experiences as structured visual canvases can preserve information that may be lost when images, tables, and diagrams are converted into short textual summaries. The framework is training-free with respect to the main MLLM and can therefore be applied without modifying the model architecture.
-
The system is evaluated along several practically relevant dimensions. Beyond downstream task accuracy, the authors examine retrieval configurations, layout choices, computational cost, and memory consolidation. The progressive degradation mechanism is also an interesting alternative to binary memory eviction, especially when partial visual information remains useful under constrained storage.
Critical weaknesses
-
The evaluation protocol does not fully establish the claimed long-term agent-memory capability. In the main benchmark protocol, the memory bank is built from labeled training examples and includes questions, gold answers, and associated metadata. The system then retrieves these stored examples to answer held-out questions. This is a legitimate retrieval-augmented inference setting, but it is different from long-term memory over evolving agent experiences. Importantly, the answer-removal ablation in Appendix G shows that removing gold-answer fields reduces HotpotQA EM from 50.70 to 44.04, suggesting that a substantial part of the reported improvement depends on access to labeled prior examples. The paper should more clearly distinguish case-based retrieval gains from benefits attributable to persistent visual memory.
-
The novelty and comparative advantage over closely related multimodal memory systems remain insufficiently established. Visual context rendering, hybrid embedding retrieval, and frequency-based memory management are individually well-established techniques. Their combination is practically appealing, but the paper does not provide a direct head-to-head comparison with some closely related visual or multimodal memory systems, such as V-Mem or VizoMem. The current evaluation also places substantial emphasis on conventional QA benchmarks rather than long-horizon multimodal interactions involving memory updates, temporal validity, and fine-grained visual recall. These limitations make it difficult to determine whether MemCanvas constitutes a sufficiently significant advance in lifelong multimodal agent memory.
Questions for authors
-
Can the authors report the full set of memory-system comparisons under the answer-removed protocol, using exactly the same stored information for MemCanvas and the text or multimodal retrieval baselines? The existing answer-removal ablation is useful, but a complete matched comparison would help isolate the benefit of canvas representation.
-
How does MemCanvas compare with strong visual or multimodal memory systems such as V-Mem and VizoMem under a matched backbone, retrieval budget, storage budget, and evaluation protocol?
-
Can the authors demonstrate the benefit of the proposed framework on a long-term multimodal interaction benchmark where relevant evidence comes from evolving historical observations rather than labeled training examples? In particular, how well does the system handle memory updates, conflicting observations, and temporally stale images?
-
For progressive memory degradation, how does performance change as a function of the actual retained storage budget and the amount of visual detail required by future queries? What evidence shows that gradual downsampling is preferable to strong eviction policies when measured under identical memory capacity and query distributions?
We thank the reviewer for constructive, concrete questions. Each has a measured answer below.
W1 (protocol vs. long-term memory claim). Agreed. The paper is repositioned: the abstract and introduction now state that the main protocol measures retrieval-augmented inference with stored demonstrations (isolating representation and retrieval), and the case-based vs. representation distinction is made explicit via the matched answer-removal table (Global Table 2): ScienceQA Text-RAG 87.53 → 86.72, canvas 89.74 → 88.42, so on ScienceQA the gain is largely case-based for both formats (the answer-free canvas retains +0.52 over no-memory, within noise). HotpotQA Text-RAG 44.36 → 17.57 vs. canvas 50.70 → 44.04 EM. The answer-free canvas leads the answer-free text baseline by +26.47 EM (CI ±1.4), so the visual-representation benefit persists exactly where the retrieval signal is weak. Both statements now appear in the main text.
W2 (novelty vs. V-Mem/VizoMem, QA emphasis). Every mechanism has an isolation row under one protocol: layout (format isolation, Table 13), retrieval (α sweep, Table 6, plus the CLIP/SigLIP2 encoder ablation, Table 12), consolidation (T×S grid, Table 7, plus the eviction-strategy comparison, Table 11). Head-to-head: a V-Mem-style baseline (raw-unit storage + per-query modality routing, matched backbone/K/protocol) scores 87.50 (ScienceQA) and 31.13 (OK-VQA) vs. canvas 89.74 / 56.92. VizoMem has no released ColGlyph checkpoint, which makes a head-to-head comparison infeasible, and we state this. We compare its closest controllable counterparts (OCR-of-canvas 87.39 and Text-RAG) and match its token-efficiency claim (−35% vision tokens vs. fixed-width rendering, Table 13). Long-horizon results are added (Q3).
Q1. Done, in Global Table 2 and summarized in W1: identical stored information and retrieval for every cell. On ScienceQA both formats' gains are largely answer-dependent (stated openly), while on HotpotQA the answer-free canvas beats the answer-free text baseline by +26.47 EM, so the format advantage there is not answer lookup.
Q2. Backbone, retrieval budget (K=2, hybrid CLIP), and protocol are matched by construction for every new row. Storage: the consolidated bank occupies 286.2 MB including embeddings (Table 10), versus 470.9 MB for Image Fewshot and 89.0 MB for Mem0, and Table 11 compares eviction policies at identical capacity (3,374 entries, same query stream): frequency-adaptive degradation 89.74 vs. LRU 89.8, FIFO 88.27, Ebbinghaus 87.03, Random 85.39. The distinct claim is the continuous fidelity spectrum (degraded entries stay retrievable instead of being deleted).
Q3 (long-horizon: updates, conflicts, stale images). Three additions. (i) LoCoMo hybrid (the text of the retrieved episodes injected alongside their canvases, identical retrieval and judge): 75.06 → 87.53 (+12.47, with multi-hop 78.82 → 92.21 and open-domain 70.99 → 85.85). Visual memory composes with textual recall on continuous dialogues, and the GPT-4o-mini-protocol version is queued. (ii) Update probe: for 5% of the bank we re-rendered canvases with a flipped answer and an "UPDATED" header. On the 190 queries whose top-1 memory was updated, the model follows the updated record in 38.9% of cases, retains its parametric answer in 57.9%, and flips elsewhere in 3.2%. Updates are read and can steer answers, without blind copying. (iii) Conflict and stale-image probes use the same template (queued), and a dedicated long-horizon multimodal-memory benchmark is the next evaluation target, with the protocol detailed in the revision. We will report these in the discussion thread.
Q4 (degradation vs. budget, visual-detail demand). The retention–accuracy curve (Table 7): 100% retention → 89.92, 26.5% → 89.74 (−0.18), 17.1% → 89.20, 10.2% → 88.95, 4.8% → 88.62, down to 0.8% → 87.96, all above no-memory (87.90), with the storage trajectory of every (T,S) configuration in Fig. 4 and the consolidated bank at 286.2 MB (Table 10). At identical capacity and query distribution (3,374 entries), progressive degradation scores 89.74 vs. LRU 89.8 / FIFO 88.27 / Ebbinghaus 87.03 / Random 85.39, which is parity with the best binary policy and clearly above the rest, while degraded entries remain retrievable instead of deleted. On visual-detail demand, the rendering advantage concentrates on image-bearing queries (IMG +5.11 over raw-image injection), so degradation costs concentrate there. A per-modality breakdown across degradation levels is queued and will be posted in this thread.