acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Multimodal Self-Evolving Memory for Personalized Vision-Language Agents

Abstract

Personalized AI assistants need long-term memory that evolves as users interact with them over time and across domains. Yet evaluations focused on text and static memory snapshots reveal little about how multimodal memory supports lifelong personalization. We introduce MemoryVCD, a benchmark that separates three memory design dimensions: representation, updating, and growth. Built on real users’ time-stamped multimodal histories, it combines five input formats with two growth axes: within-domain progression with advancing prediction targets, and cross-domain expansion with fixed targets. We evaluate six personalization tasks using up to twelve vision-language backbones from three providers and four main memory-update methods. Visual information generally improves personalization, but no update method performs best across all tasks. Accumulated history helps ranking and choice, while compression and recency can better preserve quantitative accuracy and local detail. More history does not always improve performance, and cross-domain memory can cause persistent negative transfer in visual prediction. These findings point to memory systems that selectively retain and transfer evidence as users evolve. MemoryVCD provides a shared testbed for studying multimodal lifelong personalization.

Primary area:
datasets and benchmarks
Keywords:
Personalization, User Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5qoDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MemoryVCD is a proposed benchmark that tests multimodal memory for personalized vision-language agents using long-term Amazon review histories. Evaluating across multiple model families, the authors claim that visual memory helps; no update method dominates on their task; current LLMs struggle with full user histories in context; and cross-domain memory may cause negative transfer. However, the contribution is limited by substantial overlap with prior work, and several experimental claims are not properly controlled.

Critical strengths
  1. The paper proposes a large-scale multimodal personalization benchmark with real user histories.
  2. The self-evolving protocol is well-scoped.
  3. The experiments cover significant breadth across multiple frontier models and families. Especially, Tables 3-4 and Figures 4-5 are easy to follow.
Critical weaknesses

Missing baselines and comparisons

  1. The paper does not test or mention several closely related recent works, most important of which are M2A [1] and MIRIX [2]. These papers already implement online updates and personalization with multimodal memory. Discussing how MemoryVCD differs from these papers is crucial to understanding its contribution.
  2. Further, within- and cross-domain growth have also been studied with Amazon movie and book ratings in MAP [3], while MemoryVCD notes it as a novel contribution (lines 107-109, 125-126). The authors must adjust their contribution statements by incorporating this highly relevant literature.

Data collection and methodology

  1. The proposed dataset is not a new collection. It incrementally extends MemoryCD. Some of the experimental methods also substantially overlap with MemoryCD, weakening the paper’s ability to be an independent contribution. The authors should quantify exact record reuse. As I see it, the multimodal task formulation and evaluations across time and domain carry the novelty burden, but they lack comparisons to existing benchmarks in the literature.
  2. All reported results are point estimates without any reported error margins. For instance, this makes small differences such as 0.367 and 0.360 in Choice accuracy between GPT-5.5 and GPT-5 (Table 3) not discernible.
  3. GPT-5 is used as the only judge for image predictions. No inter-judge tests for the task; no calibration. GPT-5 family models are also dominant in the experiments, so using the same model for evaluation is (a) somewhat circular and (b) biases the results due to self-preference. Neither is tested.
  4. Experiments in the main paper are mostly illustrative, with Tables 3 and 4 locking themselves to one domain (Home and Kitchen or Books), setting (text_image), and time budget (T_4). While this makes sense from a budget or available page space perspective, the paper does not justify selecting either of these options, which makes the results less rigorous. My major concern is assuming the resulting format comparisons generalize across update rules and memory stages without testing those interactions.
  5. Further, Table 4’s Time and Domain columns switch between H&K and Books, which further disables any comparison.

Experimental findings

  1. Finding 1 under 3.1, "visual signals broadly improve personalization," is slightly definitional. Providing the LLM with more information, especially in a different modality, is supposed to help if the improvement is driven by better recognition of the current product rather than better use of personal memory. The paper does not isolate this effect.
  2. Rendered text, similarly, changes how the same semantic information is encoded. Considering it as evidence for visual signal requires some qualification that I don’t see in the paper. I suggest the authors consider an experiment where historical images and target images are independently varied and compared with shuffled user histories. This would help understand if gains come from mere multimodal inputs or multimodal personal memory.
  3. Finding 2 "model scaling benefits personalization..": Model "scaling" cannot be confirmed through closed models -- that needs to be tested with OSS LLMs for which parameter counts are well known. A "capability" finding makes sense here; a "scaling" finding does not. The paper clarifies this in the write-up, yet the headline finding and the abstract use the “scaling” terminology.
  4. Finding 3, lines 375-376: "...that require a global view of a user's preferences" -- Such tasks, or what makes them "require a global view," were never defined
  5. Lines 398-408 and Finding 4: The paper mostly observes task-specific rankings, but the explanations and conclusions are far more generalized than what the evidence reliably supports. For instance, a ranking improvement by itself does not demonstrate retention of “global preferences,” especially as the setting was never tested. Similarly, just because RecentMem is the best generation method does not automatically mean that preserving recent wording caused the improvement. The paper needs to conduct experiments to isolate these causes before the claims can be reliably made.
  6. “Full history is not always best” is also not experimentally strong, as the main experiment on the time axis changes both memory and prediction targets. While the paper points to Appendix M, that experiment covers only two methods on one model, domain, and format.

Writing

  1. The writing in the paper is quite dense, as several sentences pack multiple ideas, with some presenting strong claims from limited, illustrative data. This makes it quite hard to follow and understand.
  2. Several critical details about the experimental setup are relegated to the appendix. For instance, the task definitions (Lines 668-688) must be presented in the main text for a better reader experience.

[1] Feng et al., 2026, M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions [2] Wang and Chen, 2025, MIRIX: Multi-Agent Memory System for LLM-Based Agents [3] Jiarui Chen, 2025, Memory Assisted LLM for Personalized Recommendation System

Questions for authors
  1. Can the authors quantify how the selected records from the Amazon Reviews 2023 dataset differ from MemoryCD, if at all?
  2. How does MemoryVCD differ from M2A, MIRIX, MAP, and XSMoE?
  3. Are all reported results in Tables 3-4 from one run of the corpus?
  4. Was GPT-5 the judge for image predictions calibrated against human judgment? Did you test a second judge to measure inter-rater reliability?
Reviewer_LWSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemoryVCD, a benchmark for evaluating multimodal memory in personalized vision-language agents. It uses longitudinal Amazon interaction histories from 323 users across twelve domains and evaluates six personalization tasks with five input formats. Experiments examine memory growth over time and across domains, compare memory-update methods, and evaluate up to twelve vision-language backbones.

Critical strengths
  • The histories span years of real user activity. Explicit time and domain cutoffs allow personalization to be examined at different stages of available history.
  • The six tasks and five input formats expose differences that a single aggregate score would hide. The comparisons across twelve backbones and several memory policies make the benchmark broadly usable.
  • The D4 matched-donor comparison shows that a user's own history helps ranking in the tested settings. The fixed-target experiment also shows gains from additional history when the prediction targets are held constant.
Critical weaknesses
  • The empty-memory reference behind the negative-transfer result is unclear. Section 2 defines D0 as empty memory, and Table 4/Figure 5 compare Books image scores near 1.0 against D0=1.717. However, Appendix K says earlier plots reused an empty-memory result with different test items, while Appendix A places matched D0 controls outside the main trajectories. The paper does not identify whether 1.717 comes from a run on the same final-three targets. The source-only results show poor prediction, but establishing a drop below empty memory requires an empty-memory run on the same final-three targets.
  • The controls do not isolate the contribution of the user's own source-domain visual history. The donor experiment includes target-domain history (Appendix C), and the format sweep changes both memory and task inputs (Appendix A). Without a source-only comparison holding task inputs fixed, the own-history and format effects cannot be separated in that setting.
  • Image Prediction mixes three different information states. Appendix A reveals each preceding target interaction before the next prediction, including at D0-D3. Only the first prediction has no revealed target-domain interactions. Averaging all three obscures whether the D1-D3 visual deficit precedes this feedback or persists after it, which affects the persistence and recovery interpretation in Section 3.3.
  • RecentMem's cross-domain selection rule is unspecified. Appendix A sorts interactions within each domain and concatenates the domains, adding target-domain history last at D4. Section 2 and Appendix D do not say whether recency is defined by global timestamps or by position in this list. These rules can retain different evidence, so the D4 recovery cannot be interpreted without knowing the retained domain composition.
Questions for authors
  1. Was Books D0=1.717 in Table 4/Figure 5 evaluated on the same final-three targets as D1-D3? Please reconcile this with Appendix K and report matched empty-memory and source-only scores. Both conditions should use the same reveal policy, judge and scored examples.
  2. What are the D0-D3 Image Prediction scores for the first held-out interaction and for the later two, separately, using common scored targets?
  3. Does RecentMem use global timestamps or positions in the domain-concatenated list? Please give the retained domain proportions at D3, D4 and D5, including the setting behind the Books decline in Table 25.
  4. As a secondary clarification, what are the paired score differences and user-level uncertainty estimates for the intermediate-cutoff peaks and close format comparisons? Please use the same runs and users; Appendix G's mixed-run sample-size comparison does not provide these estimates.
Reviewer_2i3MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MEMORYVCD, a benchmark for multimodal long-term memory in personalized vision-language agents. Built on Amazon Reviews 2023, it selects 323 long-history, cross-domain users with 314,116 interactions and compares five input formats, six personalization tasks, and multiple memory-management strategies. Evaluation follows two growth axes: a within-domain time axis that advances both history and prediction targets, and a cross-domain axis that fixes the targets while progressively adding histories from different domains. The results suggest substantial task dependence: retrieval over accumulated history is particularly effective for item ranking, while compression and recent interactions perform better on some rating and generation tasks.

Critical strengths
  1. The research problem is important, and the evaluation perspective is valuable. Rather than evaluating only a final memory snapshot, the paper examines personalization as histories grow, while considering both visual information and cross-domain histories.
  2. The data and experimental coverage are broad. Real users’ longitudinal histories provide a useful complement to simulated conversations. Comparisons across formats, models, and memory methods offer substantial material for studying different design choices.
Critical weaknesses
  1. Inconsistent cross-domain empty-memory baseline. Main-text Table 4 and Figure 5 compare with subsequent stages to support claims about cross-domain benefits and negative transfer. However, Appendix K states that earlier plots reused a time-axis empty-memory result evaluated on different test items and therefore excludes that reference. The manuscript does not clearly reconcile these descriptions. If the main-text baseline uses unmatched targets, the observed differences may reflect target difficulty rather than memory effects. The authors should identify the affected results and provide target-matched empty-memory controls.
  2. Confounded interpretation of time-axis trajectories. The time-axis protocol advances both memory and prediction targets, making it difficult to attribute performance changes to memory growth. For example, ReadAgent’s rating MAE increases from 0.453 at to 0.580 at , but these stages evaluate different items. The fixed-target experiments in Appendix M partially address this concern, yet cover only a limited set of configurations. Broader controls are needed to distinguish memory effects from changes in user behavior and target difficulty.
  3. Insufficient validation that the benchmark measures user-specific preference memory. In the H&K ranking control with GPT-5-mini, matched-donor history achieves 0.755 NDCG@5, compared with 0.367 for empty memory and 0.855 for the user’s own history. Thus, substantial gains can arise without the individual’s own history, leaving the contribution of user-specific preferences insufficiently isolated. Moreover, lexical-overlap metrics do not directly measure personal writing style, while open-ended image prediction conflates category prediction with visual-attribute matching. Additional controls and validation are needed to establish the benchmark’s construct validity.
Questions for authors
  1. Do the results in Table 4 and Figure 5 use exactly the same test items as –? Please reconcile these results with Appendix K and clarify whether the cross-domain gains and visual negative-transfer findings hold under target-matched empty-memory controls.
  2. The format sweep changes both memory inputs and target/candidate inputs. Could you vary these two factors independently to determine whether the gains come from visual information in historical memory or from better understanding the current candidates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.