acceptodds
Under review as a conference paper at ICLR 2027

MEMLENS: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

Abstract

Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: longcontext LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K–256K tokens) under a cross-modal token-counting scheme. On 634 image-essential and image-supportive questions, controlled image ablation drops two frontier LVLMs below 2% accuracy. Across 27 LVLMs and 7 memoryaugmented agents, the best 32K accuracies are 58.68% for LVLMs and 33.46% for memory agents. LVLMs degrade as conversations grow, whereas memory agents are comparatively length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. Abstention results suggest memory-oriented post-training risks weakening correct refusal. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval.

Primary area:
datasets and benchmarks
Keywords:
memory, VLM, evaluatioin

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rrHeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the MemLens benchmark, the first to evaluate multimodal long-term conversational memory on both Large Vision Language Models and memory-augmented agents, spanning input lengths of up to 256k tokens on 789 different questions. Each sample consists of multiple conversations or sessions, and the goal for the model is to answer a question based on the knowledge contained in the full history of conversations. The conversation history is populated with filler conversations, adding nothing important; multimodal or haystack conversations, making retrieval harder; and evidence conversations, containing important information to answer the final query. The benchmark tests five different memory abilities, i.e., Information Extraction, Multi-Session Reasoning, Temporal Reasoning, Knowledge Update, and Answer Refusal, for which a different task is formulated. The authors evaluated 27 LVLMs and 7 memory-augmented agents, showing that while LVLMs obtain the best performance, LVLMs performance degrades much faster than memory agents as conversations grow.

Critical strengths
  1. The paper is complete in all its parts. Although benchmark creation is complex, everything is well justified while generated data is checked via automated controls or human judgment.
  2. The benchmark size is sufficiently large, and the evaluated model set is comprehensive.
  3. Results and analysis are informative.
  4. The paper fills a gap in the current literature, i.e., it is the first benchmark to measure multi-session performance with controlled context length in multimodal sessions.
Critical weaknesses
  1. The paper borrows plenty from LongMemEval. As the title mentions, lots of the design choices are borrowed from the LongMemEval benchmark. Specifically, (i) how concatenated time-stamped conversation histories are directly put into LVLMs' context; (ii) models' ability taxonomy; (iii) session construction. On top of these, MemLens explains how to integrate visual queries. Thus, the technical novelty is limited, in my opinion.
  2. Comparison with memory-augmented agents is disappointing. I liked the idea, though not much the execution. What strikes me is that most memory-augmented agents do not support images, and among the two that do, one is using an old model compared to the tested LVLMs. Then, to make these models support images, the authors convert images into BLIP-generated captions, which makes LVLMs vs. memory-augmented agents an apples-to-pears comparison. A more interesting approach would've been investigating the best design choices for memory-augmented agents using tested LVLMs as backbones, which is similar to what LongMemEval does.
  3. Memory handling for LVLMs is unrealistic. This is strictly related to the previous point. For LVLMs, this paper fits all conversations into the model's context. This is unrealistic, as existing closed-source models are likely not handling it that way, since they must deal with hundreds of conversations and documents per user [1]. As a result, this benchmark evaluates models' ability to deal with long context, and not how existing models deal with memory from lots of conversations.

[1] https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Questions for authors
  1. Why did the authors compare LVLMs with memory agents instead of testing different LVLMs and LVLMs coupled with existing memory-handling techniques?
  2. Most of the drop for memory-augmented agents comes from the fixed memory representation at storage time (L.407). Now, since for LVLMs memory is not compressed, retrieving raw images instead of compressing them would largely address this problem. What do the authors think about this? If what I am saying makes sense, then would their observation still stand?
  3. Would it be technically feasible, in the authors' opinion, to integrate agentic workflows in the benchmark? (I am not expecting authors to do so during the rebuttal; rather, I am checking whether it is feasible)
3 comments
AuthorsOpenReview ↗

Comment 1/2 — Same LVLM, different memory access

We thank the reviewer for recognizing the rigor of our data construction, the breadth of model coverage, and the value of evaluating multimodal multi-session memory at standardized context lengths. We are especially grateful for the suggestion to give the same LVLM retrieved access to original images. Our response below compares full-history access, retrieved original images, and retrieved captions using the same LVLM.

Q1: Why compare LVLMs with memory agents? We evaluate each memory agent as released because most are post-trained around a specific backbone: MemAgent-7B uses the RL-trained RL-MemoryAgent-7B, and Memory-T1 uses an RL-trained Qwen2.5-3B. Swapping in a different LVLM would discard the memory behavior these agents were trained for. For comparability, configurable pipelines such as MemOS use a frozen Qwen3-VL-8B of comparable scale rather than a frontier LVLM. The paper already measures backbone sensitivity within a fixed pipeline (Table 21): replacing Mem0's default backbone with gpt-4.1-mini raises Overall accuracy by 10.65 percentage points. The reviewer's proposal isolates a complementary factor, memory handling with the LVLM held fixed, which we now report.

Q2: Same LVLM, different access to the conversation history. We use GPT-5.4-mini as the LVLM on the canonical 195-question subset at the 32K context length. BM25 searches the conversation text of each session with the question and selects five sessions, kept in their original order. The settings with BM25 top-5 retrieved sessions and original images, GLM-4.6V captions, or BLIP-2 captions use the same retrieval specification and differ only at image positions, which hold either the original image or a caption from a question-blind captioner that sees neither the question nor the reference answer.

History access and visual representationCorrect / questionsAccuracyMean LVLM input tokens
Full conversation history with original images98/19550.26%25,551
BM25 top-5 retrieved sessions with original images90/19546.15%11,915
The same BM25 top-5 retrieved sessions with GLM-4.6V captions77/19539.49%11,304
The same BM25 top-5 retrieved sessions with BLIP-2 captions51/19526.15%10,792

In the table above, retrieving original images is 4.10 points below the full conversation history while sending 53.37% fewer input tokens to the LVLM. Replacing the same images with BLIP-2 captions costs 20.00 points; GLM-4.6V captions remain 6.67 points below original images and recover 13.33 points over BLIP-2 captions.

Does the observation that memory pipelines can lose faithfulness to original visual evidence still hold? The table shows a 20.00-point accuracy difference between original images and BLIP-2 captions, with the same LVLM and the same retrieved sessions. GLM-4.6V captions recover about two-thirds of the aggregate gap between BM25-retrieved sessions with original images and the same sessions with BLIP-2 captions. Accuracy is higher with GLM-4.6V captions than with BLIP-2 captions and remains below that with original images.

BM25 retrieves 70.4% of evidence sessions, and all evidence sessions for 99 of 173 answerable questions. The BM25 comparison isolates visual representation by holding the retrieved sessions and answer LVLM fixed across the conditions with original images, GLM-4.6V captions, and BLIP-2 captions. The token counts report LVLM inputs and exclude indexing and captioning costs. We are also annotating 60 incorrect answers from these three retrieval settings by hand, recording whether the required visual fact appears in the image or caption the LVLM received; we will report the results once the annotation is complete.

AuthorsOpenReview ↗

Comment 2/2 — Agentic workflows and benchmark contributions

Q3: Agentic workflows. Yes, integration is technically feasible, and the benchmark already provides what such an evaluation needs. Sessions can be ingested chronologically with the final question withheld; images can be exposed through persistent identifiers that carry no answer labels; and at query time an agent can call tools to search sessions, read their conversation text, and retrieve original images. The existing judge rubric scores the final answer, while tool calls, evidence access, and token use can be logged. The setting with BM25 top-5 retrieved sessions and original images in our Q2 response already exercises retrieval and image delivery. A learned multi-step agent is a feasible extension; we will document this interface in the appendix.

C1: Relation to LongMemEval. We thank the reviewer for this important point. We deliberately build on LongMemEval's five memory abilities and timestamp-ordered multi-session histories, which keeps MEMLENS comparable with text-only memory evaluation, and we will credit this more explicitly. What a text-only benchmark cannot test is memory for visual evidence. On this foundation, MEMLENS contributes three things:

  1. Questions whose evidence lives in images. Entity abstraction replaces the entity in the conversation text with a higher-level concept, and is designed so that the answer must be recalled from the original image.
  2. Standardized context lengths for multimodal histories. The same questions are evaluated from 32K to 256K tokens, with image tokens counted toward the context length.
  3. Direct LVLMs and memory agents on the same multimodal questions. LongMemEval compares long-context LLMs with memory designs over text. With images in the history, the same comparison raises a question that text cannot pose: whether visual memory should be stored as captions or as original images.

We will add an explicit comparison of inherited and new components to the related work.

C3: Memory handling for LVLMs. We agree that deployed assistants do not keep every past conversation in context. The article the reviewer cites [1] describes a trade-off, and MEMLENS measures both sides of it. Keeping everything in context suffers from context rot: recall degrades as context grows, at any window size. Our length-controlled histories (about 14→93 sessions and 20→138 images from 32K to 256K) quantify this for direct LVLMs: from 32K to 128K, Qwen3.5-122B-A10B falls from 58.68% to 44.03%, and even Gemini-3.1-Pro and Kimi-K2.5 lose 3.45 and 3.93 points. Compressing history instead risks what [1] warns about: overly aggressive compaction discards details whose importance only becomes apparent later. The evaluated memory agents implement the techniques [1] describes. MemAgent-7B reads 5K-token chunks and overwrites a fixed-length memory (compaction), while Mem0, MemOS, and M3-Agent write memories to external stores and retrieve them (structured note-taking). Their Overall accuracy changes by at most 4.81 points from 32K to 256K yet stays low, consistent with visual details being discarded at storage time, before the question is known. The answer backbone alone does not explain this shortfall: M2A trails direct Qwen3-VL-8B-Instruct, its reported answer backbone, by 35.39 points on the same 195 items. In the Q2 comparison, replacing original images with BLIP-2 captions corresponds to a 20.00-point drop, while loading original images just in time is 4.10 points below the full conversation history and uses 53.37% fewer input tokens (Q2). MEMLENS thus measures how well each memory strategy preserves visual evidence relative to full access. We will state this framing in §1 and §3.

[1] Anthropic. Effective context engineering for AI agents. September 29, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

I thank the authors for the detailed response and for running the additional experiments.

W2. The BM25 experiment is informative, and I thank the authors for running it. In my reading, it supports the concern I raised. With the same LVLM, retrieving the original images is only 4.10 points below the full history and clearly above every memory agent evaluated on the same 195 items. The largest drop (20 points) comes from replacing images with BLIP-2 captions, which is the representation the authors chose for four of the agents. The weak agent results therefore reflect caption-based pipelines more than memory augmentation in general. I would encourage the authors to move this baseline into the main results, scope the conclusions in §4.3 accordingly, and, if possible, report it at 128K and 256K, since this is where the comparison with direct LVLMs matters most.

Overall, the rebuttal addresses several of my concerns, but some still hold. Therefore, I will keep my score.

Reviewer_kFBFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

MEMLENS tests memory over long conversations containing both text and images, with 789 questions covering five abilities and histories built at 32K–256K token budgets. Many questions require information that is present in an image but deliberately omitted from the text. The paper evaluates 27 LVLMs and seven memory systems, with different coverage, and includes image-removal tests, human review, and checks of the grading models.

Critical strengths
  1. Removing the evidence images drops two frontier models below 2% accuracy on 634 questions, giving direct evidence that these questions require visual information (Table 3).
  2. The five ability categories, controlled history lengths, and extensive human and grading checks make this a useful benchmark for studying different memory failures (Section 3; Appendices E and G).
Critical weaknesses
  1. Some systems are tested on 789 questions and others on 195, with different grading models, so the main figure does not support precise comparisons between all systems despite the agreement checks (Appendix B).
  2. Poor agent performance is attributed to losing visual information during storage, but differences in answering models, image access, and retrieval also affect the results, so storage alone has not been isolated (Section 4.3; Appendices G.6–G.7).
Questions for authors
  1. Do the model–agent gaps remain when all systems are compared on the same questions using the same grading model?
  2. Could you distinguish the evidence for visual storage loss from failures caused by retrieval or the answering model?
  3. What license, permission, or other legal basis allows redistribution of the 4,695 web images described in Appendix C.7?
4 comments
AuthorsOpenReview ↗

Matched questions and judges

We thank the reviewer for the careful reading and for recognizing the paper's cross-modality ablation (Table 3), as well as the coverage of five memory abilities at standardized context lengths and our human and judge quality checks. All three questions go to the heart of how our results should be read. We answer the evaluation questions with new analyses below.

Q1: Do the model–agent gaps remain on the same questions under the same judge? Yes. This check matters because the paper's headline contrast, 58.68% for the best direct LVLM versus 33.46% for the best memory agent, combines the full 789-question benchmark with two judges: Qwen3-VL-235B-A22B-Instruct for direct LVLMs and GPT-5.4-mini for the five full-benchmark agents (Appendix B.2). Appendix H acknowledges that this judge difference is not shown to be negligible. We re-judged the paper's original 32K outputs on the canonical 195-question subset with a single judge, GPT-5.4, and re-judged five systems with Gemini-3.1-Pro as a second judge.

SystemPaper: 789 items, original judgesSame 195 items: original judgesSame 195 items: GPT-5.4Same 195 items: Gemini-3.1-Pro
Qwen3.5-122B-A10B58.68%63.59%64.10% (125/195)64.62%
GPT-5.452.72%51.28%49.74% (97/195)50.77%
MemOS33.46%35.38%34.87% (68/195)35.38%
Mem027.12%30.77%30.26% (59/195)31.28%

Matching questions and judge does not shrink the gap between the strongest direct LVLM and the strongest memory agent. Qwen3.5-122B-A10B − MemOS is 25.22 percentage points in the paper, 28.21 points on the same items under the original judges, and 29.23 points under GPT-5.4 (95% CI [21.54, 36.92]). The other prespecified gaps also hold under GPT-5.4: GPT-5.4 − Mem0 is +19.49 points [12.31, 26.15], and Qwen3.5-122B-A10B − Mem0 is +33.85 points [26.67, 40.51]. All three remain significant at Holm-adjusted p<.001 within our prespecified four-comparison family. Gemini-3.1-Pro reproduces all three gaps (+19.49, +33.33, and +29.23 points), and the two judges disagree on 12 of 975 judgments across the five systems they both scored (98.77% agreement). We are also hand-checking 60 of these outputs, stratified by question type and by whether the two judges agree, with 18 labeled independently by a second annotator; we will report the results once the annotation is complete.

The subset and the judge also leave the direct-LVLM ranking largely unchanged. Extending the six-model check in Appendix G.2 (ρ=1.00) to 26 direct LVLMs, we find that rankings on the 195 items correlate with full-benchmark rankings at Spearman ρ=0.95 under the original judge, and re-judging with GPT-5.4 preserves the 195-item ranking at ρ=0.99. In total, the re-judged panel covers 31 configurations and 6,240 judgments; Gemini-3.1-Pro, Memory-T1, and M2A were not re-judged in this round.

Each system keeps its released answer backbone. Most memory agents are post-trained around a specific backbone (MemAgent-7B uses the RL-trained RL-MemoryAgent-7B, and Memory-T1 an RL-trained Qwen2.5-3B), and configurable pipelines such as MemOS use a frozen Qwen3-VL-8B of comparable scale. These are therefore system-level gaps; our Q2 response isolates the memory component with the LVLM held fixed. We will add the matched table beside Figure 2 and label every panel with its items and judge.

AuthorsOpenReview ↗

Visual storage, retrieval, and answering

Q2: Separating visual storage loss from retrieval and answering failures. We agree this distinction is essential, because the cross-agent evidence in the paper varies storage, retrieval, and the answer backbone together. We therefore provide a comparison of original images and captions using the same LVLM and the same BM25 retrieval method, then interpret it alongside the paper's separate diagnostics of retrieval and answering.

Original images versus captions. We use GPT-5.4-mini as the common LVLM on the canonical 195-question subset at 32K. BM25 searches the conversation text of each session with the question and selects five sessions. The three retrieval settings use the same retrieval specification and differ only at image positions, which contain the original image, a GLM-4.6V caption, or a BLIP-2 caption from a question-blind captioner.

History access and visual representationCorrect / 195Accuracy
Full conversation history with original images9850.26%
BM25 top-5 retrieved sessions with original images9046.15%
The same BM25 top-5 retrieved sessions with GLM-4.6V captions7739.49%
The same BM25 top-5 retrieved sessions with BLIP-2 captions5126.15%

Storage. In the table above, replacing the original images in the BM25-retrieved sessions with BLIP-2 captions corresponds to a 20.00-point drop. Replacing BLIP-2 captions with GLM-4.6V captions recovers 13.33 points, while GLM-4.6V captions remain 6.67 points below original images. Appendix G.6 separately shows the same direction inside a released agent: changing only MemAgent-7B's captioner from BLIP-2 to GLM-4.6V raises IE accuracy by 9.67 points (Overall +2.25 points; two runs per setting).

Retrieval. BM25 retrieves 70.4% of evidence sessions, and all evidence sessions for 99 of 173 answerable questions, so retrieval is imperfect. The setting with BM25-retrieved sessions and original images is 4.10 points below the setting with the full conversation history and original images. All three BM25 settings use the same retrieved sessions, holding session selection fixed in the representation comparison. Inside the agents, the retrieval logs in Appendix G.6 (Figure 19) show that Mem0 and Memory-T1 surface 82–89% of evidence sessions on average and make 87–95% of their errors with at least half of the evidence sessions retrieved, whereas 78.1% of M3C's errors fall below that threshold.

Answering. M2A trails direct Qwen3-VL-8B-Instruct, its reported answer backbone, by 35.39 points on the same 195 items (Appendix G.6): a capable answer backbone does not guarantee high accuracy after memory construction and retrieval. The four settings in the representation comparison use one LVLM configuration, so the 20.00-point difference between retrieved original images and retrieved BLIP-2 captions is not produced by switching the answer model.

These results support the paper's finding that visual compression loses answer-relevant evidence: replacing original images with BLIP-2 captions reduces accuracy by 20.00 points with the LVLM and retrieved sessions held fixed. The separate retrieval diagnostics identify an additional failure pattern: most of M3C's errors occur with fewer than half of the evidence sessions retrieved. We will revise §4.3 to distinguish the measured visual-representation loss from retrieval and answering failures. We are annotating archived incorrect answers from the retrieved-original-image, retrieved-GLM-caption, and retrieved-BLIP-caption settings by ability and retrieval coverage, and will report the audit once annotation is complete.

AuthorsOpenReview ↗

Image release and planned revisions

Q3: Image sources and redistribution. Thank you for identifying this issue. We agree that the provenance and redistribution of the images are real release concerns, and we take them seriously. Our current description in Appendix C.7 is not sufficiently precise about what may be redistributed, and we will correct both the paper and the release procedure. Specifically, we will distinguish two sources in the camera-ready release:

  1. Images collected from the web: following the approach used by LoCoMo (Maharana et al., 2024), we will release the original source URLs and verification metadata, together with a download script, rather than redistribute copies of the image files.
  2. Images from existing datasets: we will continue to include the original images only where their source dataset license permits redistribution, accompanied by attribution and the applicable license text. In MEMLENS, this category consists of the 48 synthetic analog-clock images from Analog-Clocks, released under GPL-2.0.

Our author-produced annotations will remain under CC-BY-4.0 and the evaluation code under MIT. We will align Appendix C.7, the datasheet, the dataset card, and the release documentation with this policy. We sincerely thank the reviewer for bringing this serious issue to our attention; the revised release plan above is intended to address it directly and responsibly.

Summary of revisions.

  1. A matched table with the same 195 items and a single judge beside Figure 2, with every panel labeled by its items and judge.
  2. The comparison of original images and captions in Appendix G.6, with §4.3 distinguishing visual-representation loss from retrieval and answering failures.
  3. The corrected image release and documentation described above.
  4. The human audits of judge decisions and failure stages, added once the annotation is complete.

We thank the reviewer again; these questions make both the comparison and the release more rigorous.

AuthorsOpenReview ↗

Follow-up human annotation results

Thank you for your careful review and for highlighting the cross-modality ablation, the coverage of five memory abilities at standardized context lengths, and our benchmark quality checks. We have completed the human audit described in our response to Q1 and would like to share the results.

Q1. Matched evaluation and human audit of judge decisions. On the same 195 questions scored by GPT-5.4, Qwen3.5-122B-A10B leads MemOS by 29.23 percentage points (95% CI [21.54, 36.92]). All three prespecified gaps remain significant after Holm correction (adjusted p < .001), and Gemini-3.1-Pro reproduces all three gaps.

Annotator A served as the primary annotator and labeled all 60 outputs. Annotator B independently labeled 18 prespecified outputs with system names, judge verdicts, and Annotator A’s labels withheld. Annotator A had access to the judge verdicts. The two annotators agreed on 83% of the 18 overlapping outputs (Cohen’s κ = 0.74). Annotator B then reviewed each annotation disagreement against the evidence sessions and recorded the rationale for the final verdict.

We stratified the sample by question type and judge agreement. It includes all 12 disagreements and 48 sampled agreements from the 969 outputs that both judges had scored when the sample was drawn.

Judge verdictsAuditedDecidable from the evidenceHuman verdict matches GPT-5.4Human verdict matches Gemini-3.1-Pro
Agree484242/4242/42
Disagree1272/75/7

Where the judges agreed, the human verdict matched both judges on all 42 decidable outputs. Both judges marked the other six outputs incorrect. Those cases remained unresolved under the annotation protocol, most often because the verdict depended on treating a session timestamp as the date of an event reported in that session.

Among the seven decidable disagreements, GPT-5.4 rejected five correct answers that paraphrased the reference or gave a more specific response to the question, such as “vanilla” when the question asked for the flavor and the reference was “vanilla gelato.” Gemini-3.1-Pro accepted one answer that hedged between two positions and rejected one correct paraphrase. The evidence was insufficient to resolve the other five disagreements.

After weighting to the sampling frame, the human verdicts match GPT-5.4 on 99.4% and Gemini-3.1-Pro on 99.8% of decidable outputs. Annotator B’s blinded verdicts match GPT-5.4 on all 15 outputs that B could decide.

Using the human verdicts and allowing either outcome for the undecidable cases, the three gaps remain 29.23–30.26 percentage points for Qwen3.5-122B-A10B over MemOS, 18.97–20.00 for GPT-5.4 over Mem0, and 33.85–34.36 for Qwen3.5-122B-A10B over Mem0. Even if each judge disagreement were resolved in the direction least favorable to the corresponding comparison, the gaps would remain 28.21, 18.46, and 32.31 points, respectively. We will add the annotation protocol and per-output labels to the revision.

Q2. Storage, retrieval, and answering. With the same LVLM and BM25-retrieved sessions, replacing the original images with BLIP-2 captions reduces accuracy by 20.00 percentage points. GLM-4.6V captions recover 13.33 of those points. We will revise §4.3 to distinguish the effect of changing the visual representation from failures in retrieval and answering.

Q3. Image release. We will provide source URLs, verification metadata, and a download script for web-collected images. We will redistribute image files only where the source license permits it.

We would welcome any remaining questions. If these results address your concerns, we would appreciate your reconsideration of the score.

Reviewer_tyPpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MemLens, a benchmark for evaluating multimodal long-term memory in multi-session conversations. It contains 789 questions across five memory abilities and four standardized context lengths (32K–256K). The authors evaluate 27 LVLMs and seven memory-augmented agents, finding that long-context models generally achieve higher accuracy but degrade with increasing context length, while memory agents are more length-stable but struggle to preserve fine-grained visual information.

Critical strengths

There are four strengths after carefully reading this paper:

  • A relevant and meaningful evaluation gap. The paper addresses an important limitation of existing conversational memory benchmarks: many include images but do not establish whether visual evidence is actually necessary. Evaluating long-context LVLMs and memory agents under a shared multimodal benchmark is valuable.

  • Strong validation of visual dependence. The image-ablation experiment is a particularly convincing part of the paper. The dramatic performance drop after removing evidence images shows that the benchmark genuinely tests visual information access, rather than merely adding images to otherwise text-answerable questions.

  • Broad evaluation with informative findings. Evaluating 27 LVLMs and seven memory agents across multiple memory abilities and context lengths is a substantial effort. The per-ability analysis reveals meaningful differences that would be hidden by overall accuracy alone, particularly for multi-session reasoning and knowledge updating.

  • Careful benchmark construction and quality control. The combination of automated filtering, multiple rounds of human review, and detailed evaluation diagnostics strengthens the credibility of the dataset. I also appreciate the extensive appendix and reproducibility information.

Critical weaknesses

The main concerns during the whole review processing are:

  • The claimed joint multimodal dependency is not fully established. The image-ablation experiment demonstrates that visual information is necessary, but does not show that both images and surrounding conversation text are necessary. In fact, Appendix F.5 describes a visual judge checking whether the image and question alone are sufficient. An image-only ablation would help clarify how much of the benchmark truly requires cross-modal integration rather than visual retrieval.

  • The comparison between LVLMs and memory agents is substantially confounded. Four memory agents receive BLIP-2 captions, and even the native multimodal agents do not retain access to original image pixels at answer time. Their weaker performance on image-dependent questions is therefore somewhat expected. Differences in backbone strength and retrieval configurations further complicate the comparison. A matched memory baseline that retrieves original images would make the architectural conclusions much more convincing.

  • Not all reported comparisons use the same evaluation conditions. Memory-T1 and M2A are evaluated on only 195 questions, and the full-benchmark agents are scored by a different LLM judge from the direct LVLMs. The deterministic audit also shows nontrivial scoring differences. Furthermore, direct LVLMs are not evaluated at 256K, despite this being one of the benchmark's four target lengths. These issues weaken the precision of the cross-system rankings and length-scaling conclusions.

  • The synthetic setting limits how broadly the findings can be generalized. The benchmark relies heavily on generated conversations, deliberate entity abstraction, and artificially inserted evidence and distractors. This is useful for controlled evaluation, but it may not fully represent how multimodal information accumulates in realistic user-agent interactions. The relationship to [1] and especially [2] also deserves closer discussion.

  • Some explanations of the failure mechanisms go beyond what the experiments isolate. For example, the visual-error categories partly follow from the question's annotated modality requirements rather than directly observing which evidence a model retrieved or perceived. The oracle experiment supplies both selected facts and images, so it cannot cleanly separate retrieval from extraction failures. Similarly, the proposed relationship between memory-oriented post-training and weaker abstention is suggestive, but not established by a controlled pre/post-training comparison.

Refs:

  1. Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents.
  2. M^3Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions.
Questions for authors

I have some minor questions about this paper:

  • Have the authors evaluated an image-only condition, where the model receives the question and relevant evidence images but not the surrounding textual facts? I am particularly interested in how many questions genuinely require information from both modalities, given the visual sufficiency check described in Appendix F.5.

  • Could the main LVLM-agent comparisons be reported using the same question subset and evaluation judge, ideally with uncertainty intervals? It would also be helpful to evaluate at least the frontier LVLMs at 256K, so that the longest-context comparisons are genuinely matched.

7 comments
AuthorsOpenReview ↗

Text–image dependence

We thank the reviewer for recognizing the image-removal ablation, the breadth of the evaluation, our quality control, and the reproducibility materials. The two questions go to what the benchmark must demonstrate: whether conversation text still matters once the evidence images are supplied, and whether the system comparisons hold on matched questions under a single judge. We ran new analyses for both and answer them first, then address the weaknesses.

Q1: Accuracy with only the question and the evidence images, and how many questions need both modalities. Conversation text contributes substantially even when the evidence images are supplied: across 634 image-essential and image-supportive questions, removing the conversation text lowers accuracy from 69.24% to 17.19%. We use GPT-5.4-mini as the single LVLM and compare inputs from the correct evidence sessions, without distractor sessions. These 634 questions cover IE, MSR, TR, and KU; answer-refusal questions are excluded.

The images-only condition supplies the question, its query date, and exactly the same images in their original order. It removes conversation text, session dates, and image file names. Apart from a sentence identifying the images as previously shared, the only added instruction is "Answer the question concisely. Give only the answer, without explanation." There is no instruction to respond with "Insufficient information." The text + images condition retains the complete evidence-session transcripts, including user and assistant turns, session dates, and images at their original positions. The text-only condition retains these transcripts and dates but replaces images with neutral placeholders; it runs on the shared 200-question subset. All conditions use low reasoning effort, a 4096-token decoding budget, and GPT-5.4 as the judge.

Accuracy on correct evidence sessions; all conditions retain the same question and query date. The final row restricts all three conditions to the same 200 questions.

QuestionsText + imagesImages onlyText only
All 634439/634 (69.24%)109/634 (17.19%)not run
Information extraction (IE)190/246 (77.24%)36/246 (14.63%)not run
Multi-session reasoning (MSR)36/78 (46.15%)10/78 (12.82%)not run
Temporal reasoning (TR)149/194 (76.80%)34/194 (17.53%)not run
Knowledge update (KU)64/116 (55.17%)29/116 (25.00%)not run
Shared 200-question subset143/200 (71.50%)19/200 (9.50%)11/200 (5.50%)

Removing conversation text costs 52.05 percentage points overall, with losses in all four abilities. The largest losses are on IE (62.60 points) and TR (59.28 points); supplying the images preserves visual content but removes the conversational associations and time references used to answer. The IE definition in §3.1 distinguishes identifying an entity in an image and retrieving associated information from the surrounding text (Entity) from recalling a visual detail (PrevInfo). The shared 200-question subset gives a complementary comparison: removing text costs 62.00 points, while removing images costs 66.00 points. Both modalities contribute to accuracy on this shared question set. These comparisons measure model behavior; the fraction of questions whose answers logically require both modalities needs a separate evidence audit.

This ablation complements Table 3 by measuring the contribution of conversation text when the evidence images are already supplied. It uses full evidence-session transcripts, while Table 3 supplies selected evidence facts. Removing conversation text also removes session dates, so the TR and KU losses include the loss of temporal anchors. The shared 200-question results are unweighted comparisons.

To count the questions that need both modalities at the level of information rather than model behavior, we are conducting a human evidence audit of 120 image-essential and image-supportive questions, sampled at random within ability × modality strata. For each question, the annotator records the textual facts and visual content the answer requires and whether either modality alone determines it; a second annotator independently labels 36 of the questions to measure agreement. We will report the weighted proportions and agreement, and add them to the revision, once the annotation is complete.

AuthorsOpenReview ↗

Matched evaluation and 256K results

Appendix F.5. The reviewer is right to distinguish the construction filter from the contribution of conversation text. Accepting an image + question input establishes sufficiency under the visual judge; that check alone establishes neither image necessity nor text necessity. The ablation above measures the contribution of conversation text to answering, and the human audit will assess information requirements. F.5 also describes the text-only leakage judge as a question-only screen while its printed template includes the conversation text and the reference answer, and §3.3 repeats the question-only description. We will correct §3.3 and F.5 to state the inputs each filter receives for each question type, state the evidence requirement of each ability in §3.4, and add this ablation next to Table 3.

Q2: Matched comparison under a single judge, with intervals, and the 256K setting. The system-level gaps hold on matched questions under a single judge. The paper's headline contrast, 58.68% for the best direct LVLM versus 33.46% for the best memory agent, combines the full 789-question benchmark with two judges (Appendix B.2), and Appendix H acknowledges that this judge difference is not shown to be negligible. We therefore re-judged the paper's original 32K outputs on the canonical 195-question subset with GPT-5.4, and re-judged five systems with Gemini-3.1-Pro as a second judge.

SystemPaper: 789 items, original judgesSame 195 items: original judgesSame 195 items: GPT-5.4Same 195 items: Gemini-3.1-Pro
Qwen3.5-122B-A10B58.68%63.59%64.10% (125/195)64.62%
GPT-5.452.72%51.28%49.74% (97/195)50.77%
MemOS33.46%35.38%34.87% (68/195)35.38%
Mem027.12%30.77%30.26% (59/195)31.28%

The direct–agent gaps remain substantial on matched questions under a single judge. Qwen3.5-122B-A10B − MemOS is 25.22 points in the paper, 28.21 points on the same items under the original judges, and 29.23 points under GPT-5.4 (95% CI [21.54, 36.92]). GPT-5.4 − Mem0 is +19.49 points [12.31, 26.15], and Qwen3.5-122B-A10B − Mem0 is +33.85 points [26.67, 40.51]. All three remain significant at Holm-adjusted p<.001 within our prespecified four-comparison family; the fourth comparison, Qwen3-VL-8B-Instruct − M2A, was not re-judged in this round and enters the adjustment as untested (p=1). Gemini-3.1-Pro reproduces all three gaps (+19.49, +33.33, and +29.23 points). Across 26 direct LVLMs, rankings on the 195 items correlate with full-benchmark rankings at Spearman ρ=0.95, and re-judging with GPT-5.4 preserves the 195-item ranking at ρ=0.99. The re-judged panel covers 31 configurations and 6,240 judgments; Gemini-3.1-Pro, Memory-T1, and M2A were not re-judged in this round.

The two judges largely agree on the new panel: they disagree on 12 of 975 judgments across the five systems they both scored (98.77% agreement). Separately, the deterministic audit in Appendix E.2 finds that the judge credits incorrect answers on a weighted 5.4% of closed-form items and preserves top-five membership under rule-based scoring within its diagnostic panel. The paper's 484-item human verification reports κ=0.86 against consensus labels. For the new single-judge panel, we are hand-checking 60 system outputs, stratified by question type and by whether the two judges agree, with 18 labeled independently by a second annotator; we will report the results once the annotation is complete.

256K. As stated in Appendix B.1, limited compute and context windows below 256K for several LVLMs (for example, 200K for Claude Sonnet 4.5 and 128K for the Gemma3 family; Table 4) led us to evaluate all direct LVLMs at 32K–128K, so that the panel is compared at the same lengths. Gemini-3.1-Pro has a 1M-token window and can process the complete 256K multimodal histories. For this rebuttal, we re-judged its original 128K outputs from the submitted paper and evaluated a new 256K run, using GPT-5.4 as the judge for both on the canonical 195-question subset. Under this common judge, accuracy decreases from 109/195 (55.90%) at 128K to 99/195 (50.77%) at 256K, corresponding to 10 fewer correct answers and a 5.13-percentage-point decline, with the denominator fixed at 195. At 256K, the paper's original MemOS and Mem0 outputs, re-judged by GPT-5.4 on the same items, score 59/195 (30.26%) and 55/195 (28.21%). We will add the matched table beside Figure 2, label every panel with its items and judge, and report per-system length coverage next to the length analysis.

AuthorsOpenReview ↗

Comparison design and benchmark scope

We address the remaining weaknesses in the order raised.

System comparisons vary captions, pixel access, backbone, and retrieval together. We agree, and the comparison is system-level by design. Most memory agents are post-trained around a specific backbone (MemAgent-7B uses the RL-trained RL-MemoryAgent-7B, and Memory-T1 an RL-trained Qwen2.5-3B), so swapping in another LVLM would discard the memory behavior they were trained for; configurable pipelines such as MemOS use a frozen Qwen3-VL-8B of comparable scale. Table 21 already measures backbone sensitivity within a fixed pipeline: replacing Mem0's backbone with gpt-4.1-mini raises Overall accuracy by 10.65 points. To compare original images with captions, we use the same LVLM and the same retrieved sessions on the canonical 195-question subset at 32K. BM25 selects five sessions per question from their conversation text. The settings with BM25 top-5 retrieved sessions and original images, GLM-4.6V captions, or BLIP-2 captions use the same retrieval specification, differing only at image positions, which contain the original image or a caption from a question-blind captioner.

History access and visual representationCorrect / 195Accuracy
Full conversation history with original images9850.26%
BM25 top-5 retrieved sessions with original images9046.15%
The same BM25 top-5 retrieved sessions with GLM-4.6V captions7739.49%
The same BM25 top-5 retrieved sessions with BLIP-2 captions5126.15%

In the table above, replacing the original images in the BM25 top-5 retrieved sessions with BLIP-2 captions costs 20.00 points, while GLM-4.6V captions recover 13.33 of those points and remain 6.67 points below original images. The setting with BM25-retrieved sessions and original images is 4.10 points below the setting with the full conversation history and original images.

The comparison above is consistent with the observation in §4.3 that current memory pipelines can lose faithfulness to original visual evidence at storage time: accuracy differs by 20.00 points between original images and BLIP-2 captions, with the same LVLM and the same retrieved sessions.

Synthetic construction and ecological validity. We agree that a synthetic benchmark does not replace real long-term interactions, and Appendix H states this scope. The construction provides what natural logs cannot: each question and its evidence stay fixed while the surrounding history grows from 32K to 256K, so differences across lengths reflect history length rather than changes in the questions or evidence. Entity abstraction serves the same diagnostic purpose: it replaces named entities in the conversation text with higher-level concepts to make visual identification part of answering. Three controls limit construction artifacts: three rounds of human review, including a conversational-naturalness audit of every evidence session (Appendix E.1); filler sessions, which form the majority of each context, drawn from real ShareGPT and UltraChat conversations; and a text-classification check comparing evidence sessions with haystack sessions. Both classes in that check are produced by the same generation pipeline; the classifiers achieve 56.71% (TF-IDF) and 57.92% (DeBERTa) accuracy (Appendix G.8).

Relation to Mem-Gallery and M³Exam. We discuss Mem-Gallery in our related work (§2). M³Exam is concurrent work, first posted on arXiv in June 2026, and we will add it to the related-work discussion. MEMLENS's distinct emphasis is evaluating the same questions at four standardized multimodal context lengths from 32K to 256K, verifying visual dependence through image removal (two frontier LVLMs fall below 2%), and comparing direct LVLMs with memory agents on identical questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.