- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SMILE, a small-large collaboration framework for long-term multimodal personalization. A small MLLM is trained to distill query-conditioned personalized context from raw interaction histories, and a larger frozen MLLM reasons over the compact handoff. The method uses P2C-CoT supervised fine-tuning and Context Handoff Policy Optimization. Experiments on Persona-MME, LCMP, PersonaMem-v2, and LongPersona-MME report improved performance while passing only about 2.6% of the history to the large model.
Critical strengths
The problem is important: long-term multimodal personalization requires both efficient access to history and strong reasoning over selected evidence. The small-large division of labor is sensible, and the reported efficiency gains are appealing. The paper evaluates across several benchmarks and shows consistent improvements over Full, RAG, and dataset-specific baselines. The plug-in use of stronger large MLLMs is also practically attractive.
Critical weaknesses
The main issue is generalization. The small MLLM is specialized per benchmark in most experiments, and the training pipeline relies on powerful MLLM annotation and synthetic data, so the results do not yet establish transfer to new users, domains, or deployment histories. Comparisons are also somewhat asymmetric and do not comprehensively cover stronger learned retrieval or compression methods under matched compute. In addition, the efficiency analysis focuses on context passed to the large model but does not adequately account for annotation, training, and query-time distillation costs. LongPersona-MME is central to the long-history claims but is not available during review, limiting reproducibility.
Questions for authors
- How well does one small MLLM trained on one benchmark transfer to other benchmarks without benchmark-specific specialization?
- What is the end-to-end annotation and training cost, and how many queries are needed to amortize it?
- Can the authors compare against stronger learned retrieval or compression baselines under a matched inference budget?
- Can LongPersona-MME or a representative subset be released during review?
7 comments
Response to Reviewer EvXu (2/3)
(Part 2/3)
[Weaknesses/Q2] (1) How about the training and annotation overhead? (2) How about the query-time distillation costs.
We sincerely thank you for raising this question regarding the training cost of SMILE. We would first like to clarify that SMILE is deliberately designed to minimize training overhead: we keep the expensive L-MLLM completely frozen and only optimize lightweight LoRA adapters on the S-MLLM. Moreover, we use a single frozen Qwen3.5-9B as the CHPO interpreter, allowing the resulting S-MLLM checkpoint to be directly paired with substantially stronger L-MLLMs (e.g., Qwen3.5-27B and GPT-5.6-Sol) without any additional training. Furthermore, the entire annotation pipeline is automated, requiring no manual annotation effort. We clarify the costs from two complementary perspectives below.
(1) End-to-end Annotation and Training Cost. SMILE incurs limited offline costs by design. First, the entire training data annotation pipeline is fully automated, requiring no manual annotation effort. Specifically, a powerful MLLM (GPT-5.6-Sol) automatically performs oracle interaction identification and deletion-only context pruning. Second, we only optimize lightweight LoRA adapters (rank 16) on the language-model component of the 4B S-MLLM, while keeping its vision encoder and projector frozen. The L-MLLM remains completely frozen throughout training. In practice, a full specialization on Persona-MME takes only approximately 16 hours on two NVIDIA RTX 6000D GPUs, including 12 hours for SFT and 4 hours for CHPO. More importantly, CHPO consistently uses the same frozen Qwen3.5-9B as its interpreter, regardless of the L-MLLM used at inference time. Consequently, a single trained S-MLLM checkpoint can be directly paired with different and substantially stronger L-MLLMs (e.g., Qwen3.5-27B and GPT-5.6-Sol) without any additional training. This design substantially reduces the offline training investment while enabling flexible test-time scaling.
(2) Complete Per-query Inference Cost. We would also like to clarify that our reported inference FLOPs already account for the complete computational cost of both S-MLLM-based context distillation and L-MLLM-based personalized reasoning, rather than only the context passed to the L-MLLM. Specifically, as formalized in Appendix C, the total inference cost of SMILE consists of two components: processing the complete interaction history with the lightweight S-MLLM and reasoning over the compact context with the L-MLLM:
= O(N_S T_S)+O(N_L T_L).$$ Therefore, the query-time distillation cost is explicitly included in our efficiency evaluation. Even after accounting for both models, SMILE requires only **65.6% of the inference FLOPs of Full on Persona-MME (9B)** and further reduces this to **55.5% on LongPersona-MME (9B)**, demonstrating that the efficiency gains are not merely achieved by shifting computation to the small model.Response to Reviewer EvXu (3/3)
(Part 3/3)
[Weaknesses/Q3] Comparison with stronger learned retrieval or compression baselines under a matched inference budget.
We sincerely thank the reviewer for this valuable suggestion. Our original baseline selection follows the established evaluation protocol of PersonaVLM, and PersonaVLM (CVPR 2026) itself was a state-of-the-art method for long-term MLLM personalization. SMILE already substantially outperforms it, demonstrating the effectiveness of our approach over existing learned memory-based methods.
Nevertheless, to further address your concerns and provide a more comprehensive comparison, we additionally introduce two stronger learned baselines, PPRO (2026 arxiv) and LLMLingua-2 (2024 ACL Findings), covering learned retrieval and learned context compression, respectively. PPRO constructs episodic and semantic memory banks and optimizes user query rewriting via GRPO with retrieval and downstream answer quality as feedback. LLMLingua-2 formulates context compression as token-level classification, using a data-distilled bidirectional Transformer encoder to identify and preserve essential tokens while removing redundant ones.
We evaluate all methods under a matched inference budget to ensure a fair comparison. As shown in the table below, SMILE consistently achieves substantially better performance. The reasons are: (1) PPRO rewrites queries without accessing the underlying memory banks, limiting its ability to resolve highly personalized and context-dependent queries beyond conventional semantic retrieval, while its pre-constructed and compressed memories may further discard fine-grained evidence before future queries arise. (2) LLMLingua-2 adopts a task-agnostic compression objective without explicitly optimizing for personalized evidence relevance, making it less effective at preserving personalized-query-specific multimodal clues.
| Method | Memory | Intent | Preference | Behavior | Relationship | Growth | Alignment | Overall |
|---|---|---|---|---|---|---|---|---|
| Qwen3.5-9B (Backbone) | 37.76 | 57.69 | 46.40 | 43.96 | 50.72 | 64.41 | 66.08 | 54.29 |
| + PPRO | 74.18 | 81.43 | 60.73 | 58.43 | 76.08 | 77.61 | 86.31 | 75.19 |
| + LLMLingua-2 | 61.34 | 74.80 | 53.48 | 52.07 | 68.93 | 72.39 | 80.16 | 67.94 |
| + SMILE (Ours) | 86.92 | 86.12 | 68.16 | 65.69 | 78.22 | 85.77 | 94.91 | 82.82 |
We sincerely hope that our detailed clarifications and additional empirical evidence have adequately addressed your concerns and resolved the remaining ambiguities. If so, we would greatly appreciate it if you could reconsider your assessment to our work. We would also be more than happy to further discuss any remaining questions or concerns.
Response to Reviewer EvXu (1/3)
We sincerely thank you for the thoughtful questions and positive assessment of our work. We are particularly encouraged by the your recognition of the importance of long-term multimodal personalization, the soundness of our small–large division of labor, the appealing efficiency gains, the consistent improvements across diverse benchmarks, and the practical value of plug-and-play scaling with stronger L-MLLMs. Below, we address the remaining questions and provide further clarifications.
We sincerely thank the reviewer for raising this important question regarding the generalization of SMILE. We would like to clarify that our paper has already included cross-distribution and cross-benchmark transfer evaluations without any additional S-MLLM specialization, which provide direct empirical evidence of its generalization ability.
(1) Why benchmark-specific specialization is adopted in some experiments. The different training configurations are motivated by the intrinsic differences in benchmark designs and task formulations, rather than a reliance on benchmark-specific architectural designs or heuristics. Specifically, Persona-MME focuses on user-centric personalization across seven dimensions, with MCQ-based evaluation. PersonaMem-v2 targets implicit user preferences and involves both MCQ and open-ended response generation, with its training supervision constructed from explicitly provided evidence snippets. In contrast, LCMP is concept-centric and emphasizes reasoning over evolving attributes, temporal conflicts, and short-term events jointly with long-term knowledge. Its training supervision is constructed around concept-level histories and annotated key points, with different evaluation protocols. Consequently, their supervision structures and task-specific objectives are inherently different. We therefore construct benchmark-aligned training collections to provide appropriate supervision, rather than forcing incompatible task formulations into a single training configuration. Importantly, this does not imply that SMILE relies on benchmark-specific model architectures or manually designed retrieval rules.
(2) Direct transfer to unseen users and substantially different history distributions. More importantly, we have already demonstrated transfer without additional specialization in two settings:
Persona-MME → LongPersona-MME: We directly apply the S-MLLM trained on Persona-MME to our newly constructed LongPersona-MME, without any additional training or adaptation. Notably, LongPersona-MME contains 50 entirely new personas, with no overlap in personas, dialogues, or images with Persona-MME or the S-MLLM's training collection. Despite this distribution shift and substantially longer interaction histories (up to 256K tokens), SMILE achieves 73.76% accuracy, substantially outperforming Full (64.67%) with the same Qwen3.5-9B L-MLLM, an absolute gain of 9.09 percentage points. This directly demonstrates generalization to unseen users and independently constructed multimodal interaction histories.
PersonaMem-v2-32K → PersonaMem-v2-128K: We also evaluate the same specialized S-MLLM on the 128K setting after training on PersonaMem-v2, without additional specialization. With Qwen3.5-9B as the L-MLLM, SMILE achieves 37.16% MCQ accuracy and 35.39% open-ended accuracy, compared with 31.38% and 25.54% for Full, respectively. This demonstrates robust transfer to substantially longer interaction histories within the same benchmark.
We sincerely thank you for this suggestion. Due to the double-blind review requirements and the substantial size of LongPersona-MME, hosting the full dataset or a representative subset on a publicly accessible platform during the review period may compromise author anonymity. To provide further insight into the dataset, we have additionally included some representative data examples in the appendix, to eliminate your concerns. We are fully committed to releasing LongPersona-MME upon acceptance, as stated in the paper, to facilitate future research and reproducibility.