- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PREF2ACT, a benchmark for evaluating continual preference memory in multimodal agents. The benchmark contains six personas and long-horizon multimodal interaction histories that include dialogue, image generation and editing. It evaluates agents using cumulative memory questions and repeated anchor questions that track preference acquisition and retention over time. Experiments show that existing methods struggle to integrate distributed feedback, do not consistently improve as interaction histories grow, and often fail to retain scenario-specific preferences. The results also reveal a knowing–doing gap, which means agents may correctly identify relevant preferences but still fail to generate or edit corresponding images.
Critical strengths
-
This work innovatively integrates multimodal memory, personalized understanding, and personalized generation into the PREF2ACT benchmark.
-
The knowing–doing gap revealed by the experiments is insightful, as it highlights the difficulty current multimodal agents face in translating identified user preferences into preference-consistent visual outputs.
Critical weaknesses
The paper considers a variety of memory methods, but their implementation details are not sufficiently clear. Please refer to the “Questions for the Authors” for further details.
Questions for authors
-
The paper compares a broad range of memory methods, but their downstream performance can be substantially affected by the quality of the retrieved information. It would be helpful if the authors could report retrieval-level metrics such as supporting-evidence precision. Specifically, the findings for RQ2 may arise from different factors, including inaccurate retrieval or failure to integrate multiple pieces of evidence. A more fine-grained analysis would help identify the main causes of the observed acquisition and retention failures.
-
The paper evaluates seven textual memory methods and five multimodal memory methods, but it remains unclear what visual information (such as image captions) is available to the textual memory methods. Clarifying how historical images are organized for textual memory methods would make the comparison between textual and multimodal memory configurations easier to interpret.
-
Several evaluated memory methods, such as Mem0 and MemoryOS, rely heavily on LLM calls. However, the auxiliary LLMs used for these operations are not clearly specified. Moreover, processing all the sections in a single batch may produce different results from incrementally updating memory section by section. It is also unclear whether later checkpoints continue from the previous memory state or rebuild memory from the complete history. In addition, different memory methods have memory items with different levels of details. Therefore, does not necessarily provide an equal amount of information. It would be better if the authors could explain more implementation details.
-
Preference changes and unrelated-topic interruptions are common in realistic interactions. Although the paper briefly gives a corresponding statement, the frequencies of these events are not reported. Providing additional statistics would improve the clarity and realism assessment of the benchmark.
-
It would also be useful if the authors could provide evaluation results for more recent unified multimodal models, such as SenseNova_U1_8B_MoT_Infographic-V3.
5 comments
[2/2] Author response
Q4: Details of Preference Changes and Topic Interruptions
Regarding unrelated-topic interruptions, each section contains 16–20 groups, including 3–4 groups on other scenarios as distractions. These unrelated-topic interruptions account for approximately 20% of the groups in each section. These comprise 2 image generation groups and 1–2 understanding groups, each also containing 3–10 turns. The remaining groups focus on the section’s main scenario.
Regarding preference changes, we clarify that these mainly concern differences across subscenarios. Most subscenarios within a scenario share a common preference. When an exception is introduced, it is limited to only one subscenario. For example, within Home & Daily Life, the preferred interior style is modern minimalist for Living Room, Bathroom, and most other subscenarios, but neo-Chinese for Bedroom subscenarios. This design assesses both shared preferences and distinctions across subscenarios.
Q5: More Unified Multimodal Models
Thank you for suggesting this model. SenseNova_U1_8B_MoT_Infographic-V3 is a valuable addition to our unified multimodal baselines. Given the limited time, we have so far evaluated it on one persona for Full (Text), NaiveRAG, Mem0, Full (MM), MuRAG, UniversalRAG. We report these preliminary results below.
| Memory | Q1 | Q2 | Q3-Inf. | Q3-Gen. | Anchor Avg. |
|---|---|---|---|---|---|
| Full (Text) | 26.42 | 26.19 | 35.38 | 70.33 | 30.12 |
| NaiveRAG | 51.73 | 24.61 | 38.24 | 74.58 | 47.85 |
| Mem0 | 57.94 | 29.36 | 33.80 | 76.19 | 58.41 |
| Full (MM) | 34.83 | 26.77 | 29.50 | 78.39 | 38.65 |
| MuRAG | 61.86 | 30.53 | 44.74 | 85.66 | 52.72 |
| UniversalRAG | 60.39 | 25.20 | 46.14 | 86.88 | 55.23 |
These results suggest that memory methods remain a larger bottleneck than differences between backbone models. We therefore see improving retrieval and preference summarization as a priority for future memory methods.
We have not added them to the PDF because the evaluation is still incomplete. We will continue the evaluations during the rebuttal period and share as many additional results as possible in this response.
Additional Experimental Results
We have now completed the evaluation of the remaining memory methods on Persona 1 and report the additional results below.
| Memory | Q1 | Q2 | Q3-Inf. | Q3-Gen. | Anchor Avg. |
|---|---|---|---|---|---|
| Generative Agents | 38.6 | 20.4 | 32.7 | 71.8 | 40.5 |
| Reflexion | 25.8 | 18.3 | 26.9 | 69.4 | 29.7 |
| A-Mem | 54.7 | 15.8 | 31.6 | 75.2 | 55.8 |
| MemoryOS | 56.3 | 12.5 | 24.1 | 70.6 | 52.4 |
| NGM | 53.5 | 24.8 | 34.2 | 79.7 | 50.9 |
| AUGUSTUS | 52.1 | 22.7 | 29.8 | 81.3 | 51.6 |
We continue to find that memory methods appear to be a larger bottleneck than the choice of backbone model. Improving retrieval and preference summarization therefore remains an important direction for future work.
Please let us know if you have any further concerns or suggestions.
[1/2] Author response
Q1: Retrieval Quality and Evidence Integration
Thank you for this valuable suggestion. It motivated us to include this experiment and its discussion in the main text, in Section 4.3 (Figure 6 and Obs. 6)
We compare retrieval recall and task scores across sections using Qwen3-VL-8B with UniversalRAG.
On preference induction questions, scores remain low despite high recall. Recall exceeds 65% across all sections and exceeds 85% for global and personal preference induction, while scores remain below 40. This suggests that retrieving relevant evidence does not ensure successful preference induction. Agents still struggle to integrate multiple pieces of evidence into explicit preferences.
On fine-grained recognition questions, scores generally follow changes in recall, pointing to retrieval as a key limitation. These results highlight the need to strengthen both retrieval and evidence integration in current memory methods for different tasks.
Q2: Visual Information for Textual Memory Methods
Thank you for pointing out the missing details in our description. We have added these details in Appendix C.3.1.
We used Qwen3-VL-8B to caption all generated images. This model has shown competitive captioning performance in prior research (Liu et al.[1]). Textual memory methods receive these captions alongside the dialogue text. We acknowledge that the captions may omit or misrepresent details relevant to the six global preference dimensions and various scenario preferences. However, overly detailed prompts could introduce human bias by emphasizing selected visual attributes. We therefore consider a simple, general prompt a more neutral choice for comparison. The prompt used for image captioning is as follows:
You are an image captioning assistant. Generate a detailed, objective caption for the provided image. Describe the main subjects, their visible attributes and actions, the setting, and key spatial relationships. Base the caption only on visible evidence. Do not speculate, make subjective judgments, or invent details. Return only the caption in a single coherent paragraph.
[1] Liu Z, Wang H, Zhang Z. CAPEval: A Decoupled Caption Evaluation across Understanding and Generation[J]. arXiv preprint arXiv:2608.02589, 2026.
Q3: Memory Implementation Details
We will clarify the relevant implementation details. We have added these details in Appendix C.3.1.
[2] Bei Y, Wei T, Ning X, et al. Mem-gallery: Benchmarking multimodal long-term conversational memory for mllm agents[C]. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. 2026: 40750-40784.