- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the MemLens benchmark, the first to evaluate multimodal long-term conversational memory on both Large Vision Language Models and memory-augmented agents, spanning input lengths of up to 256k tokens on 789 different questions. Each sample consists of multiple conversations or sessions, and the goal for the model is to answer a question based on the knowledge contained in the full history of conversations. The conversation history is populated with filler conversations, adding nothing important; multimodal or haystack conversations, making retrieval harder; and evidence conversations, containing important information to answer the final query. The benchmark tests five different memory abilities, i.e., Information Extraction, Multi-Session Reasoning, Temporal Reasoning, Knowledge Update, and Answer Refusal, for which a different task is formulated. The authors evaluated 27 LVLMs and 7 memory-augmented agents, showing that while LVLMs obtain the best performance, LVLMs performance degrades much faster than memory agents as conversations grow.
Critical strengths
- The paper is complete in all its parts. Although benchmark creation is complex, everything is well justified while generated data is checked via automated controls or human judgment.
- The benchmark size is sufficiently large, and the evaluated model set is comprehensive.
- Results and analysis are informative.
- The paper fills a gap in the current literature, i.e., it is the first benchmark to measure multi-session performance with controlled context length in multimodal sessions.
Critical weaknesses
- The paper borrows plenty from LongMemEval. As the title mentions, lots of the design choices are borrowed from the LongMemEval benchmark. Specifically, (i) how concatenated time-stamped conversation histories are directly put into LVLMs' context; (ii) models' ability taxonomy; (iii) session construction. On top of these, MemLens explains how to integrate visual queries. Thus, the technical novelty is limited, in my opinion.
- Comparison with memory-augmented agents is disappointing. I liked the idea, though not much the execution. What strikes me is that most memory-augmented agents do not support images, and among the two that do, one is using an old model compared to the tested LVLMs. Then, to make these models support images, the authors convert images into BLIP-generated captions, which makes LVLMs vs. memory-augmented agents an apples-to-pears comparison. A more interesting approach would've been investigating the best design choices for memory-augmented agents using tested LVLMs as backbones, which is similar to what LongMemEval does.
- Memory handling for LVLMs is unrealistic. This is strictly related to the previous point. For LVLMs, this paper fits all conversations into the model's context. This is unrealistic, as existing closed-source models are likely not handling it that way, since they must deal with hundreds of conversations and documents per user [1]. As a result, this benchmark evaluates models' ability to deal with long context, and not how existing models deal with memory from lots of conversations.
[1] https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
Questions for authors
- Why did the authors compare LVLMs with memory agents instead of testing different LVLMs and LVLMs coupled with existing memory-handling techniques?
- Most of the drop for memory-augmented agents comes from the fixed memory representation at storage time (L.407). Now, since for LVLMs memory is not compressed, retrieving raw images instead of compressing them would largely address this problem. What do the authors think about this? If what I am saying makes sense, then would their observation still stand?
- Would it be technically feasible, in the authors' opinion, to integrate agentic workflows in the benchmark? (I am not expecting authors to do so during the rebuttal; rather, I am checking whether it is feasible)
3 comments
Comment 2/2 — Agentic workflows and benchmark contributions
Q3: Agentic workflows. Yes, integration is technically feasible, and the benchmark already provides what such an evaluation needs. Sessions can be ingested chronologically with the final question withheld; images can be exposed through persistent identifiers that carry no answer labels; and at query time an agent can call tools to search sessions, read their conversation text, and retrieve original images. The existing judge rubric scores the final answer, while tool calls, evidence access, and token use can be logged. The setting with BM25 top-5 retrieved sessions and original images in our Q2 response already exercises retrieval and image delivery. A learned multi-step agent is a feasible extension; we will document this interface in the appendix.
C1: Relation to LongMemEval. We thank the reviewer for this important point. We deliberately build on LongMemEval's five memory abilities and timestamp-ordered multi-session histories, which keeps MEMLENS comparable with text-only memory evaluation, and we will credit this more explicitly. What a text-only benchmark cannot test is memory for visual evidence. On this foundation, MEMLENS contributes three things:
- Questions whose evidence lives in images. Entity abstraction replaces the entity in the conversation text with a higher-level concept, and is designed so that the answer must be recalled from the original image.
- Standardized context lengths for multimodal histories. The same questions are evaluated from 32K to 256K tokens, with image tokens counted toward the context length.
- Direct LVLMs and memory agents on the same multimodal questions. LongMemEval compares long-context LLMs with memory designs over text. With images in the history, the same comparison raises a question that text cannot pose: whether visual memory should be stored as captions or as original images.
We will add an explicit comparison of inherited and new components to the related work.
C3: Memory handling for LVLMs. We agree that deployed assistants do not keep every past conversation in context. The article the reviewer cites [1] describes a trade-off, and MEMLENS measures both sides of it. Keeping everything in context suffers from context rot: recall degrades as context grows, at any window size. Our length-controlled histories (about 14→93 sessions and 20→138 images from 32K to 256K) quantify this for direct LVLMs: from 32K to 128K, Qwen3.5-122B-A10B falls from 58.68% to 44.03%, and even Gemini-3.1-Pro and Kimi-K2.5 lose 3.45 and 3.93 points. Compressing history instead risks what [1] warns about: overly aggressive compaction discards details whose importance only becomes apparent later. The evaluated memory agents implement the techniques [1] describes. MemAgent-7B reads 5K-token chunks and overwrites a fixed-length memory (compaction), while Mem0, MemOS, and M3-Agent write memories to external stores and retrieve them (structured note-taking). Their Overall accuracy changes by at most 4.81 points from 32K to 256K yet stays low, consistent with visual details being discarded at storage time, before the question is known. The answer backbone alone does not explain this shortfall: M2A trails direct Qwen3-VL-8B-Instruct, its reported answer backbone, by 35.39 points on the same 195 items. In the Q2 comparison, replacing original images with BLIP-2 captions corresponds to a 20.00-point drop, while loading original images just in time is 4.10 points below the full conversation history and uses 53.37% fewer input tokens (Q2). MEMLENS thus measures how well each memory strategy preserves visual evidence relative to full access. We will state this framing in §1 and §3.
[1] Anthropic. Effective context engineering for AI agents. September 29, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
I thank the authors for the detailed response and for running the additional experiments.
W2. The BM25 experiment is informative, and I thank the authors for running it. In my reading, it supports the concern I raised. With the same LVLM, retrieving the original images is only 4.10 points below the full history and clearly above every memory agent evaluated on the same 195 items. The largest drop (20 points) comes from replacing images with BLIP-2 captions, which is the representation the authors chose for four of the agents. The weak agent results therefore reflect caption-based pipelines more than memory augmentation in general. I would encourage the authors to move this baseline into the main results, scope the conclusions in §4.3 accordingly, and, if possible, report it at 128K and 256K, since this is where the comparison with direct LVLMs matters most.
Overall, the rebuttal addresses several of my concerns, but some still hold. Therefore, I will keep my score.
Comment 1/2 — Same LVLM, different memory access
We thank the reviewer for recognizing the rigor of our data construction, the breadth of model coverage, and the value of evaluating multimodal multi-session memory at standardized context lengths. We are especially grateful for the suggestion to give the same LVLM retrieved access to original images. Our response below compares full-history access, retrieved original images, and retrieved captions using the same LVLM.
Q1: Why compare LVLMs with memory agents? We evaluate each memory agent as released because most are post-trained around a specific backbone: MemAgent-7B uses the RL-trained RL-MemoryAgent-7B, and Memory-T1 uses an RL-trained Qwen2.5-3B. Swapping in a different LVLM would discard the memory behavior these agents were trained for. For comparability, configurable pipelines such as MemOS use a frozen Qwen3-VL-8B of comparable scale rather than a frontier LVLM. The paper already measures backbone sensitivity within a fixed pipeline (Table 21): replacing Mem0's default backbone with gpt-4.1-mini raises Overall accuracy by 10.65 percentage points. The reviewer's proposal isolates a complementary factor, memory handling with the LVLM held fixed, which we now report.
Q2: Same LVLM, different access to the conversation history. We use GPT-5.4-mini as the LVLM on the canonical 195-question subset at the 32K context length. BM25 searches the conversation text of each session with the question and selects five sessions, kept in their original order. The settings with BM25 top-5 retrieved sessions and original images, GLM-4.6V captions, or BLIP-2 captions use the same retrieval specification and differ only at image positions, which hold either the original image or a caption from a question-blind captioner that sees neither the question nor the reference answer.
In the table above, retrieving original images is 4.10 points below the full conversation history while sending 53.37% fewer input tokens to the LVLM. Replacing the same images with BLIP-2 captions costs 20.00 points; GLM-4.6V captions remain 6.67 points below original images and recover 13.33 points over BLIP-2 captions.
Does the observation that memory pipelines can lose faithfulness to original visual evidence still hold? The table shows a 20.00-point accuracy difference between original images and BLIP-2 captions, with the same LVLM and the same retrieved sessions. GLM-4.6V captions recover about two-thirds of the aggregate gap between BM25-retrieved sessions with original images and the same sessions with BLIP-2 captions. Accuracy is higher with GLM-4.6V captions than with BLIP-2 captions and remains below that with original images.
BM25 retrieves 70.4% of evidence sessions, and all evidence sessions for 99 of 173 answerable questions. The BM25 comparison isolates visual representation by holding the retrieved sessions and answer LVLM fixed across the conditions with original images, GLM-4.6V captions, and BLIP-2 captions. The token counts report LVLM inputs and exclude indexing and captioning costs. We are also annotating 60 incorrect answers from these three retrieval settings by hand, recording whether the required visual fact appears in the image or caption the LVLM received; we will report the results once the annotation is complete.