- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MemoryVCD is a proposed benchmark that tests multimodal memory for personalized vision-language agents using long-term Amazon review histories. Evaluating across multiple model families, the authors claim that visual memory helps; no update method dominates on their task; current LLMs struggle with full user histories in context; and cross-domain memory may cause negative transfer. However, the contribution is limited by substantial overlap with prior work, and several experimental claims are not properly controlled.
Critical strengths
- The paper proposes a large-scale multimodal personalization benchmark with real user histories.
- The self-evolving protocol is well-scoped.
- The experiments cover significant breadth across multiple frontier models and families. Especially, Tables 3-4 and Figures 4-5 are easy to follow.
Critical weaknesses
Missing baselines and comparisons
- The paper does not test or mention several closely related recent works, most important of which are M2A [1] and MIRIX [2]. These papers already implement online updates and personalization with multimodal memory. Discussing how MemoryVCD differs from these papers is crucial to understanding its contribution.
- Further, within- and cross-domain growth have also been studied with Amazon movie and book ratings in MAP [3], while MemoryVCD notes it as a novel contribution (lines 107-109, 125-126). The authors must adjust their contribution statements by incorporating this highly relevant literature.
Data collection and methodology
- The proposed dataset is not a new collection. It incrementally extends MemoryCD. Some of the experimental methods also substantially overlap with MemoryCD, weakening the paper’s ability to be an independent contribution. The authors should quantify exact record reuse. As I see it, the multimodal task formulation and evaluations across time and domain carry the novelty burden, but they lack comparisons to existing benchmarks in the literature.
- All reported results are point estimates without any reported error margins. For instance, this makes small differences such as 0.367 and 0.360 in Choice accuracy between GPT-5.5 and GPT-5 (Table 3) not discernible.
- GPT-5 is used as the only judge for image predictions. No inter-judge tests for the task; no calibration. GPT-5 family models are also dominant in the experiments, so using the same model for evaluation is (a) somewhat circular and (b) biases the results due to self-preference. Neither is tested.
- Experiments in the main paper are mostly illustrative, with Tables 3 and 4 locking themselves to one domain (Home and Kitchen or Books), setting (text_image), and time budget (T_4). While this makes sense from a budget or available page space perspective, the paper does not justify selecting either of these options, which makes the results less rigorous. My major concern is assuming the resulting format comparisons generalize across update rules and memory stages without testing those interactions.
- Further, Table 4’s Time and Domain columns switch between H&K and Books, which further disables any comparison.
Experimental findings
- Finding 1 under 3.1, "visual signals broadly improve personalization," is slightly definitional. Providing the LLM with more information, especially in a different modality, is supposed to help if the improvement is driven by better recognition of the current product rather than better use of personal memory. The paper does not isolate this effect.
- Rendered text, similarly, changes how the same semantic information is encoded. Considering it as evidence for visual signal requires some qualification that I don’t see in the paper. I suggest the authors consider an experiment where historical images and target images are independently varied and compared with shuffled user histories. This would help understand if gains come from mere multimodal inputs or multimodal personal memory.
- Finding 2 "model scaling benefits personalization..": Model "scaling" cannot be confirmed through closed models -- that needs to be tested with OSS LLMs for which parameter counts are well known. A "capability" finding makes sense here; a "scaling" finding does not. The paper clarifies this in the write-up, yet the headline finding and the abstract use the “scaling” terminology.
- Finding 3, lines 375-376: "...that require a global view of a user's preferences" -- Such tasks, or what makes them "require a global view," were never defined
- Lines 398-408 and Finding 4: The paper mostly observes task-specific rankings, but the explanations and conclusions are far more generalized than what the evidence reliably supports. For instance, a ranking improvement by itself does not demonstrate retention of “global preferences,” especially as the setting was never tested. Similarly, just because RecentMem is the best generation method does not automatically mean that preserving recent wording caused the improvement. The paper needs to conduct experiments to isolate these causes before the claims can be reliably made.
- “Full history is not always best” is also not experimentally strong, as the main experiment on the time axis changes both memory and prediction targets. While the paper points to Appendix M, that experiment covers only two methods on one model, domain, and format.
Writing
- The writing in the paper is quite dense, as several sentences pack multiple ideas, with some presenting strong claims from limited, illustrative data. This makes it quite hard to follow and understand.
- Several critical details about the experimental setup are relegated to the appendix. For instance, the task definitions (Lines 668-688) must be presented in the main text for a better reader experience.
[1] Feng et al., 2026, M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions [2] Wang and Chen, 2025, MIRIX: Multi-Agent Memory System for LLM-Based Agents [3] Jiarui Chen, 2025, Memory Assisted LLM for Personalized Recommendation System
Questions for authors
- Can the authors quantify how the selected records from the Amazon Reviews 2023 dataset differ from MemoryCD, if at all?
- How does MemoryVCD differ from M2A, MIRIX, MAP, and XSMoE?
- Are all reported results in Tables 3-4 from one run of the corpus?
- Was GPT-5 the judge for image predictions calibrated against human judgment? Did you test a second judge to measure inter-rater reliability?