acceptodds
Under review as a conference paper at ICLR 2027

MemCanvas: Visual Long-Term Memory for MLLM-based Agents

Abstract

Despite rapid advances in multimodal large language models (MLLMs), agents built upon them remain fundamentally stateless, processing each interaction in isolation with no capacity to retain or reuse experiential knowledge. Existing lifelong memory systems, whether parametric or retrieval-based, operate predominantly in text, discarding the rich visual structure of images, tables, and diagrams or reducing them to lossy textual descriptions. We present MemCanvas, a training-free, model-agnostic visual memory framework that stores heterogeneous multimodal experiences as structured canvas images, allowing agents to accumulate and retrieve knowledge through their native visual perception channel. MemCanvas introduces three coordinated mechanisms: an adaptive layout algorithm that compresses and assembles multimodal elements into structured visual canvases, a hybrid retrieval system that fuses visual and textual encoder embeddings for cross-modal memory search, and a frequency-adaptive consolidation strategy that progressively degrades rarely-accessed memories to curb storage growth. Extensive experiments across five benchmarks with two backbones show that MemCanvas improves direct inference by +17.5 EM on HotpotQA, outperforms text-based memory baselines (Mem0, MemVerse, A-Mem, Text-RAG) without task-specific training—except on purely text-only questions, where text stores remain preferable—and matches the best classical eviction strategy per retained memory at matched retention while degrading memories progressively instead of deleting them. Hybrid retrieval and progressive visual forgetting further improved the practicality of visual memory, showing that reducing the language tax need not require retraining or unbounded storage. Together, these results suggest that the dominant text-first memory stack is not merely inefficient but mismatched to multimodal reasoning, and that preserving experience in visual form can improve long-horizon agent performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multimodal artificial intelligence, long-term memory, visual memory, multimodal agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oFS8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemCanvas, a training-free visual long-term memory framework for multimodal language model agents. Instead of representing stored experiences primarily as text, MemCanvas organizes heterogeneous information into structured image canvases that can be processed by the visual encoder of an MLLM. The system consists of an adaptive layout mechanism, hybrid text-image retrieval, and frequency-adaptive memory consolidation through progressive resolution reduction. Experiments on several question-answering benchmarks show improvements over direct inference and a number of text-based memory and retrieval approaches. The paper also investigates retrieval efficiency, layout choices, storage costs, and memory consolidation.

Critical strengths
  1. The paper presents a practical and intuitive alternative to text-centric agent memory. Storing multimodal experiences as structured visual canvases can preserve information that may be lost when images, tables, and diagrams are converted into short textual summaries. The framework is training-free with respect to the main MLLM and can therefore be applied without modifying the model architecture.

  2. The system is evaluated along several practically relevant dimensions. Beyond downstream task accuracy, the authors examine retrieval configurations, layout choices, computational cost, and memory consolidation. The progressive degradation mechanism is also an interesting alternative to binary memory eviction, especially when partial visual information remains useful under constrained storage.

Critical weaknesses
  1. The evaluation protocol does not fully establish the claimed long-term agent-memory capability. In the main benchmark protocol, the memory bank is built from labeled training examples and includes questions, gold answers, and associated metadata. The system then retrieves these stored examples to answer held-out questions. This is a legitimate retrieval-augmented inference setting, but it is different from long-term memory over evolving agent experiences. Importantly, the answer-removal ablation in Appendix G shows that removing gold-answer fields reduces HotpotQA EM from 50.70 to 44.04, suggesting that a substantial part of the reported improvement depends on access to labeled prior examples. The paper should more clearly distinguish case-based retrieval gains from benefits attributable to persistent visual memory.

  2. The novelty and comparative advantage over closely related multimodal memory systems remain insufficiently established. Visual context rendering, hybrid embedding retrieval, and frequency-based memory management are individually well-established techniques. Their combination is practically appealing, but the paper does not provide a direct head-to-head comparison with some closely related visual or multimodal memory systems, such as V-Mem or VizoMem. The current evaluation also places substantial emphasis on conventional QA benchmarks rather than long-horizon multimodal interactions involving memory updates, temporal validity, and fine-grained visual recall. These limitations make it difficult to determine whether MemCanvas constitutes a sufficiently significant advance in lifelong multimodal agent memory.

Questions for authors
  1. Can the authors report the full set of memory-system comparisons under the answer-removed protocol, using exactly the same stored information for MemCanvas and the text or multimodal retrieval baselines? The existing answer-removal ablation is useful, but a complete matched comparison would help isolate the benefit of canvas representation.

  2. How does MemCanvas compare with strong visual or multimodal memory systems such as V-Mem and VizoMem under a matched backbone, retrieval budget, storage budget, and evaluation protocol?

  3. Can the authors demonstrate the benefit of the proposed framework on a long-term multimodal interaction benchmark where relevant evidence comes from evolving historical observations rather than labeled training examples? In particular, how well does the system handle memory updates, conflicting observations, and temporally stale images?

  4. For progressive memory degradation, how does performance change as a function of the actual retained storage budget and the amount of visual detail required by future queries? What evidence shows that gradual downsampling is preferable to strong eviction policies when measured under identical memory capacity and query distributions?

1 comment

We thank the reviewer for constructive, concrete questions. Each has a measured answer below.

W1 (protocol vs. long-term memory claim). Agreed. The paper is repositioned: the abstract and introduction now state that the main protocol measures retrieval-augmented inference with stored demonstrations (isolating representation and retrieval), and the case-based vs. representation distinction is made explicit via the matched answer-removal table (Global Table 2): ScienceQA Text-RAG 87.53 → 86.72, canvas 89.74 → 88.42, so on ScienceQA the gain is largely case-based for both formats (the answer-free canvas retains +0.52 over no-memory, within noise). HotpotQA Text-RAG 44.36 → 17.57 vs. canvas 50.70 → 44.04 EM. The answer-free canvas leads the answer-free text baseline by +26.47 EM (CI ±1.4), so the visual-representation benefit persists exactly where the retrieval signal is weak. Both statements now appear in the main text.

W2 (novelty vs. V-Mem/VizoMem, QA emphasis). Every mechanism has an isolation row under one protocol: layout (format isolation, Table 13), retrieval (α sweep, Table 6, plus the CLIP/SigLIP2 encoder ablation, Table 12), consolidation (T×S grid, Table 7, plus the eviction-strategy comparison, Table 11). Head-to-head: a V-Mem-style baseline (raw-unit storage + per-query modality routing, matched backbone/K/protocol) scores 87.50 (ScienceQA) and 31.13 (OK-VQA) vs. canvas 89.74 / 56.92. VizoMem has no released ColGlyph checkpoint, which makes a head-to-head comparison infeasible, and we state this. We compare its closest controllable counterparts (OCR-of-canvas 87.39 and Text-RAG) and match its token-efficiency claim (−35% vision tokens vs. fixed-width rendering, Table 13). Long-horizon results are added (Q3).

Q1. Done, in Global Table 2 and summarized in W1: identical stored information and retrieval for every cell. On ScienceQA both formats' gains are largely answer-dependent (stated openly), while on HotpotQA the answer-free canvas beats the answer-free text baseline by +26.47 EM, so the format advantage there is not answer lookup.

Q2. Backbone, retrieval budget (K=2, hybrid CLIP), and protocol are matched by construction for every new row. Storage: the consolidated bank occupies 286.2 MB including embeddings (Table 10), versus 470.9 MB for Image Fewshot and 89.0 MB for Mem0, and Table 11 compares eviction policies at identical capacity (3,374 entries, same query stream): frequency-adaptive degradation 89.74 vs. LRU 89.8, FIFO 88.27, Ebbinghaus 87.03, Random 85.39. The distinct claim is the continuous fidelity spectrum (degraded entries stay retrievable instead of being deleted).

Q3 (long-horizon: updates, conflicts, stale images). Three additions. (i) LoCoMo hybrid (the text of the retrieved episodes injected alongside their canvases, identical retrieval and judge): 75.06 → 87.53 (+12.47, with multi-hop 78.82 → 92.21 and open-domain 70.99 → 85.85). Visual memory composes with textual recall on continuous dialogues, and the GPT-4o-mini-protocol version is queued. (ii) Update probe: for 5% of the bank we re-rendered canvases with a flipped answer and an "UPDATED" header. On the 190 queries whose top-1 memory was updated, the model follows the updated record in 38.9% of cases, retains its parametric answer in 57.9%, and flips elsewhere in 3.2%. Updates are read and can steer answers, without blind copying. (iii) Conflict and stale-image probes use the same template (queued), and a dedicated long-horizon multimodal-memory benchmark is the next evaluation target, with the protocol detailed in the revision. We will report these in the discussion thread.

Q4 (degradation vs. budget, visual-detail demand). The retention–accuracy curve (Table 7): 100% retention → 89.92, 26.5% → 89.74 (−0.18), 17.1% → 89.20, 10.2% → 88.95, 4.8% → 88.62, down to 0.8% → 87.96, all above no-memory (87.90), with the storage trajectory of every (T,S) configuration in Fig. 4 and the consolidated bank at 286.2 MB (Table 10). At identical capacity and query distribution (3,374 entries), progressive degradation scores 89.74 vs. LRU 89.8 / FIFO 88.27 / Ebbinghaus 87.03 / Random 85.39, which is parity with the best binary policy and clearly above the rest, while degraded entries remain retrievable instead of deleted. On visual-detail demand, the rendering advantage concentrates on image-bearing queries (IMG +5.11 over raw-image injection), so degradation costs concentrate there. A per-modality breakdown across degradation levels is queued and will be posted in this thread.

Reviewer_d6SxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MemCanvas, a training-free memory layer for MLLM agents that renders each stored episode (question, answer, metadata, images, tables) into a single PNG canvas via an adaptive layout scorer, indexes it with a fixed-α fusion of CLIP image and text embeddings, and consolidates the bank by stepping rarely-retrieved canvases down a resolution ladder (1.0→0.75→0.5→0.25→delete). On five QA benchmarks, the training split is converted into a canvas bank and the test split is answered with top-2 retrieved canvases as visual context. Gains over direct inference are positive on OK-VQA, ChartQA, and HotpotQA, small on ScienceQA, and negative on text-only questions and on LoCoMo.

Critical strengths
  • S1. Simple, training-free system; no benchmark-specific tuning; consistent within-model comparisons across two backbones.
  • S2. Gains over direct inference are positive on OK-VQA, ChartQA, and HotpotQA.
  • S3. Progressive resolution degradation is a clean alternative to binary eviction and is shown to match LRU efficiency at 26.5% retention.
  • S4. Limitations section is honest (same-stream effect, no deduplication, LoCoMo failure).
Critical weaknesses
  • W1. Questionable choice of main baseline. The main competing method is Mem0. However, Mem0 was not designed for multimodal memory, so comparing it on equal footing is somewhat unfair. The proposed method should be compared against multimodal memory methods to establish its effectiveness.
  • W2. QA benchmarks instead of memory benchmarks. The experiments are mostly on QA datasets rather than agent memory evaluation datasets. On LoCoMo, which is designed for memory evaluation, the method actually falls behind Mem0.
  • W3. Memory vs. retrieval-augmented few-shot. The main protocol stores a labeled training split and retrieves nearest demonstrations; stored samples have no temporal or session continuity, so update, contradiction resolution, and cross-episode reference — the defining problems of long-term memory — are not exercised. The only continuous-experience benchmark (LoCoMo) is in the appendix, where MemCanvas (64.2) trails Mem0 (66.9), MemGPT (74.0), and even Full-Context with no memory (72.9).
  • W4. Overstated motivation. “Agents … remain fundamentally stateless … with no mechanism to accumulate … knowledge” sits uneasily with the paper's own related-work section, which lists several such mechanisms.
  • W5. HotpotQA contains no visual structure, yet yields the headline gain. The paper’s motivation is preserving images/tables/charts, and it also reports that text stores are preferable on text-only questions (MMQA text subset, LoCoMo). HotpotQA appears to contradict both.
  • W6. Retrieval fusion appears to contribute little; the gain seems to come mainly from the injection format. α=1.0 gives 88.36 and α=0 gives 88.82 vs. 87.90 with no memory, so hybrid retrieval adds at most ~0.9 of the 1.84 gain, while Table 13 shows format alone moves OK-VQA from 23.5 to 56.9 with retrieval fixed.
  • W7. Missing natural baseline. Table 13 lacks the most obvious few-shot format: retrieved question text + answer + original image fed as separate inputs. Without it, whether the canvas beats “just feed both modalities” is unknown.
  • W8. Heuristic forgetting with unclear benefit. The forgetting mechanism is somewhat heuristic. Were T, S, and α selected on a held-out validation split, or on the evaluation split reported in Tables 6–7? It is also unclear whether the forgetting mechanism is actually effective: without forgetting, accuracy is in fact higher (89.92). That forgetting reduces storage is largely expected. The contribution would be more convincing with a new observation — for example, that forgetting prunes unnecessary search space and thereby improves accuracy.
  • W9. Missing prompt templates. The prompt templates are not provided in the paper.
Questions for authors

Please see Weaknesses.

2 comments

We thank the reviewer for the most thorough reading and respond point by point.

W1 (main baseline). Agreed. Mem0/A-Mem are re-characterized as text-memory upper bounds, not the main competitor. We add the closest train-free multimodal-memory baseline, a V-Mem-style system (raw-unit storage, original images preserved, per-query modality routing by the top-1/top-2 margin, matched K/backbone/protocol): ScienceQA 87.50 and OK-VQA 31.13 vs. canvas 89.74 / 56.92, so the canvas wins by 2.24 and 25.79. MemVerse (fine-tuned KG memory) is already compared (38.35 on OK-VQA). VizoMem cannot be run head-to-head (ColGlyph is purpose-trained with no released checkpoint). We compare its closest controllable counterparts, OCR-of-canvas-as-text (87.39) and Text-RAG, and match its token-efficiency claim (−35% vision tokens, Table 13).

W2/W3 (QA vs. memory benchmarks, protocol is few-shot retrieval). Accepted. We reposition rather than defend: the abstract/introduction now carry the §4.1 statement (the main protocol measures retrieval-augmented inference with stored demonstrations, isolating representation and retrieval), and update/conflict/cross-episode are explicitly out of scope, first in Limitations. On LoCoMo we accept the canvas-only result (64.2 < Mem0 66.9 < Full-Context 72.9, GPT-4o-mini protocol) and diagnose it as the dense-text reading bottleneck. New: a hybrid that injects, alongside each retrieved canvas, the text of the same stored episode (identical retrieval and judge, local Qwen backbone) lifts 75.06 → 87.53 (+12.47, multi-hop 78.82 → 92.21, open-domain 70.99 → 85.85), showing the channels compose. The GPT-4o-mini-protocol version is queued and will be posted in this thread. Full-Context's 72.9 benefits from LoCoMo's ~300-turn sessions fitting in the context window, whereas canvas banks target the beyond-window regime (now stated explicitly).

W4 (motivation). Corrected as suggested: "Agents … remain fundamentally stateless …" → "existing memory mechanisms for MLLM agents are predominantly text-centric and discard the visual structure of images, tables, and diagrams."

W5 (HotpotQA vs. motivation). Headline reordered to the visual benchmarks (OK-VQA +38.2, ChartQA +23.95 over direct). HotpotQA is retained as a secondary finding with its mechanism: structured layout and rendering-time denoising aid multi-hop text reading (consistent with the optical-compression literature in §2.2), and the paper keeps its explicit report that text stores are preferable on text-only questions (MMQA text subset 61.72 vs. 63.50 direct, and LoCoMo).

W6 (fusion contributes little). We agree, on the paper's own numbers: α=0 yields 88.82 (+0.92 over no-memory) while α=0.75 yields 89.74 (+1.84), so fusion contributes at most half of the ScienceQA gain, and Table 13 shows presentation format alone moves OK-VQA from 23.5 to 56.9 with retrieval fixed. Layout/rendering is therefore the primary mechanism, and fusion's role is a single cross-modal index. Where queries are themselves multimodal, retrieval quality does matter: on MMQA image questions MemCanvas leads the best baseline by +4.04 EM (53.64 vs. 49.60, Table 4), and the encoder ablation (CLIP 89.26 vs. SigLIP2 87.83, Table 12) is independent evidence. Contributions and §4.4.1 are rewritten to this attribution.

W7 (natural baseline). Added in Global Table 1: interleaved 86.96 / 30.80 vs. canvas 89.74 / 56.92 (SciQA/OK-VQA). Separate inputs fall below no memory on ScienceQA.

W8 (forgetting, and where were T, S, α selected?). (i) Selection: α=0.75, K=2, T=1000, S=0 were fixed once, via the sweep analyses now in Tables 6–7 (ScienceQA, the first benchmark), then frozen and applied unchanged to the other four benchmarks, which were never used for selection (no benchmark-specific tuning, §4.1). The sweeps bound any selection effect: every α in the sweep (88.36–89.74) and every (T,S) cell (≥ 87.96) stays above the no-memory baseline (87.90), so no conclusion depends on the selection point. The revision states this provenance. (ii) Benefit: we agree forgetting is a storage mechanism, not an accuracy mechanism (no-forgetting 89.92 vs. 89.74, Table 7), and the revision claims exactly that. The measurements: at T=1000, S=0 the bank retains 26.5% of entries at −0.18 accuracy and 286.2 MB (Table 10), and even at 0.8% survival accuracy stays above no-memory (87.96 vs. 87.90). At identical capacity and query distribution (3,374 entries), progressive degradation matches LRU (89.74 vs. 89.8) and exceeds FIFO, Ebbinghaus, and Random (88.27 / 87.03 / 85.39, Table 11), while degraded entries remain retrievable instead of deleted. We claim neither an accuracy gain nor a retrieval-speed gain from forgetting.

W9 (prompt templates). All templates (query assembly, every injection variant, judge) are in a new appendix.

We believe these controls address the soundness concerns, and we will gladly run further matched comparisons.

Thank you for the response.

However, most of the responses to the weaknesses rest on rewriting. This rewriting does not create a new contribution. In addition, an updated PDF has not been uploaded.

I therefore keep my score.

Reviewer_LX3COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MemCanvas, a training-free memory that renders each stored example into a laid-out canvas image. Retrieval fuses CLIP image and text embeddings, and rarely retrieved canvases are downscaled, then deleted. The bank uses labeled training splits of five QA benchmarks, with Qwen2.5-VL-7B and GPT-4o-mini.

Critical strengths
  1. The controls are more thorough than usual: answer removal, format isolation, random query orders, and eviction at matched retention. It also reports where text memory wins.

  2. Layout matters among rendered formats. On OK-VQA, adaptive layout scores 56.92 versus 42.95 for naive concatenation, with 1,892 versus 3,513 vision tokens.

Critical weaknesses
  1. The evidence that rendering itself helps is weak. The isolation study lists no format that interleaves the original image with its text record. On ScienceQA image-context questions, MemCanvas scores 88.05, below Mem0 at 88.25 and A-Mem at 91.20.

  2. Much of the gain looks like lookup of answer-bearing training exemplars. Without answer fields, ScienceQA drops from 89.74 to 88.42 (87.90 with no memory), and HotpotQA EM from 50.70 to 44.04, below Text-RAG at 44.36. The 17.5 EM HotpotQA headline is measured against no retrieval.

Questions for authors
  1. Can you add an interleaved baseline, original image plus compressed text, with the same retrieval and K? Report ScienceQA image questions, OK-VQA, and HotpotQA with token counts. A match would show that rendering adds little.

  2. Can you deduplicate the bank, strip answers for all methods, and score forgetting on a held-out stream with confidence intervals? Why is direct accuracy only 8.91 on ChartQA and 18.77 on OK-VQA? Format failures would mean exemplars mainly fix the answer format.

1 comment

We thank the reviewer for a precise reading. Both weaknesses named the exact missing controls, and we ran them.

W1 (evidence that rendering helps, IMG category). We ran the interleaved baseline exactly as specified: for each retrieved memory (identical α=0.75 CLIP retrieval, K=2, same backbone and prompts), the original image and the same compressed text record are injected as separate inputs, so only the presentation varies (Global Table 1):

SciQA (IMG subset)OK-VQAHotpotQA EM
Interleaved82.9430.8044.36*
MemCanvas88.0556.9250.70
memory vision tokens2046950*

*HotpotQA has no images, so interleaved degenerates to Text-RAG. Overall ScienceQA: interleaved 86.96 vs. canvas 89.74 (no-memory 87.90). Separate inputs fall 0.94 below no memory and 0.57 below text-only injection, and canvas tokens are fixed at 1,892 by canvas resolution. This is not a match: at matched retrieval and matched content, rendering contributes +2.78 on ScienceQA and +26.12 on OK-VQA (roughly 19× the CI). The advantage concentrates exactly on image-bearing questions (IMG subset +5.11).

On the IMG-category comparison (88.05 < Mem0 88.25 < A-Mem 91.20, submitted Table 3): IMG queries already contain the query image, so injecting any retrieved visual content competes for the visual encoder, while text stores complement the query. Consistent with this, the interleaved baseline, which also injects an image, falls to 82.94 on IMG, and the canvas beats raw-image injection by +5.11 there. The IMG gap versus text methods is caused by injecting retrieved images at all, not by rendering. A modality-aware variant that skips visual injection for image-bearing queries is a straightforward extension (future work).

W2 (answer-bearing exemplars, headline). We agree, and the revision quantifies it openly. Matched answer removal (Global Table 2): ScienceQA Text-RAG 87.53 → 86.72, canvas 89.74 → 88.42. On ScienceQA the gain is largely answer-dependent for both formats (the answer-free canvas retains +0.52 over no-memory 87.90, within noise). HotpotQA EM: Text-RAG 44.36 → 17.57, canvas 50.70 → 44.04. The answer-free canvas beats the answer-free text baseline by +26.47 EM (CI ±1.4) and sits within 0.32 EM of the answer-bearing Text-RAG, so the HotpotQA advantage is not answer lookup. The text record loses far more from answer removal than the canvas, because a text record without its answer field reduces to a bare question, whereas the canvas preserves the structured evidence that supports multi-hop reading. The +17.5 headline was indeed measured against no retrieval. The abstract now leads with OK-VQA (+38.2 over direct) and ChartQA (+23.95), and HotpotQA is reported against matched text retrieval (+6.34 EM) with its answer-removal decomposition.

Q1. Done. See W1 and Global Table 1: ScienceQA image questions (IMG 82.94 vs. 88.05), OK-VQA (30.80 vs. 56.92), HotpotQA EM (44.36 vs. 50.70), all with token counts and all sharing one retrieval map, K, backbone, and prompts.

Q2 (dedup, answer stripping for all methods, held-out stream with CIs, and why direct accuracy is only 8.91/18.77). Dedup: 8.4% of the 12,726 ScienceQA entries are exact duplicates (1,069 entries, 280 groups, largest 56), and near-duplicates are rarer than they appear (449,623 of 81 million text-embedding pairs exceed cosine 0.98, i.e. 0.55%). Re-evaluating on the deduplicated bank (11,657 entries, identical protocol) gives 88.73, still above the no-memory baseline (87.90) and comparable to the answer-removal control (88.42), so the gain is not inflated by duplicates. Strip answers for all methods: done (Global Table 2). Held-out stream with CIs: prequential protocol (retrieval counters updated on the first half only, second half fully held out): 88.21, binomial 95% CI ±1.0. Five random query orders give a mean of 89.35. The 1.53-point same-stream effect is now stated in the main text with these bands. Why is direct accuracy so low? Error-type analysis (format failure = empty or >15-word answer, otherwise knowledge error). The reviewer's hypothesis is confirmed on OK-VQA, and we now say so: 72.4% of the 4,099 direct errors are format failures, and canvas memory cuts format errors 2,966 → 42 (98.6%). Of the direct-inference errors, 1,417 format failures and 645 knowledge failures are repaired, at the cost of 161 previously correct answers, and the 645 knowledge repairs cover 56.9% of the 1,133 knowledge errors. On ChartQA the hypothesis does not hold: 68.1% of direct errors are knowledge errors, and canvases fix both types (format 558 → 224, knowledge 1,191 → 1,065). On ScienceQA (MCQ) format errors are zero, and the gain is retrieval-side. The full analysis is in the appendix.

All of the above enters the revision, and we are happy to add any further control during the discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Matched-protocol evidence and repositioning.

The reviews converge on three demands: (1) a matched interleaved baseline, (2) answer removal for all retrieval methods under identical stored information, and (3) positioning the paper as what it measures. We ran all of it. Every new run below follows the paper's protocol (hybrid CLIP retrieval, α=0.75, K=2, Qwen2.5-VL-7B, greedy decoding, identical prompt templates), with only the tested factor varying. Binomial 95% CIs: ±1.0 ScienceQA (N=4,241), ±1.4 OK-VQA (N=5,046), ±1.1 HotpotQA EM (N=7,405).

Table 1. Rendering contribution at identical retrieval (only the presentation of the same retrieved records varies)

PresentationSciQAOK-VQAHotpotQA EMvision tokens
No memory87.9018.7733.240
Interleaved: original image + text record, separate inputs86.9630.8044.36*204 / 695
Text-RAG: record as text87.5323.5444.36*0
V-Mem-style: raw units + per-query modality routing87.5031.13n/an/a
MemCanvas: adaptive-layout canvas89.7456.9250.701,892

*HotpotQA memories contain no images, so interleaved degenerates to Text-RAG. Interleaved and V-Mem-style rows are new runs under the paper's retrieval protocol. On ScienceQA, separate inputs fall 0.94 below no memory (87.90) and 0.57 below text-only injection, while the canvas leads by +2.78. On the image-context subset, interleaved scores 82.94 vs. 88.05 for the canvas (submitted Table 3), so the advantage concentrates exactly on image-bearing questions. The decisive deltas (OK-VQA +26.12, roughly 19× the CI) are far beyond noise. ScienceQA deltas are small and treated as such below.

Table 2. Matched answer removal (same index and retrieval, answer fields stripped from both formats)

with answerswithout answersΔ
Text-RAG (SciQA)87.5386.72−0.81
MemCanvas (SciQA)89.7488.42−1.32
Text-RAG (HotpotQA, EM)44.3617.57−26.79
MemCanvas (HotpotQA, EM)50.7044.04−6.66

Canvas answer-free cells are the submitted Appendix-G values, and text answer-free cells are new runs under the identical retrieval. Verdicts, stated in the paper: on ScienceQA, gains are largely answer-dependent for both formats (the answer-free canvas retains +0.52 over no-memory, within noise). On HotpotQA the format advantage is not answer lookup: the answer-free canvas leads the answer-free text baseline by +26.47 EM (CI ±1.4), sits within 0.32 EM of the answer-bearing Text-RAG (44.36), and retains 62% of its gain over no retrieval (§4.2). The text baseline loses far more from answer removal than the canvas (−26.8 vs. −6.7 EM), because a text record without its answer field reduces to a bare question, whereas the canvas preserves the structured evidence and layout that support multi-hop reading.

Consolidation under storage budgets. The T×S grid (Table 7) gives the retention–accuracy curve: 100% retention → 89.92, 26.5% → 89.74 (−0.18), 17.1% → 89.20, 10.2% → 88.95, down to 0.8% → 87.96, all above the no-memory baseline (87.90). The consolidated bank occupies 286.2 MB including embeddings (Table 10), versus 470.9 MB for Image Fewshot and 89.0 MB for Mem0. At identical capacity and query distribution (3,374 entries), progressive degradation matches LRU (89.74 vs. 89.8) and exceeds FIFO (88.27), Ebbinghaus (87.03), and Random (85.39) (Table 11), while degraded entries remain retrievable instead of deleted. We claim storage efficiency and graceful degradation for forgetting, not an accuracy gain (no-forgetting is 89.92) and not a retrieval-speed gain.

Positioning (revised). MemCanvas is a visual memory representation evaluated under retrieval-augmented inference with stored demonstrations: the §4.1 protocol statement now leads the abstract/introduction, "fundamentally stateless" is removed, and update/conflict/cross-episode are explicitly out of scope. Long-horizon results are added compositionally (details per reviewer).

Revision. Everything above enters the revision, plus the T/S/α selection statement with sensitivity bounds and new appendices (prompt templates, error-type analysis, dedup statistics, update probe, LoCoMo hybrid).

Area_Chair_oEF1OpenReview ↗

Please review the rebuttal and provide further comments as appropriate

Dear Reviewers,

The authors have submitted their rebuttal. Please review it and provide further comments as appropriate.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.