acceptodds
Under review as a conference paper at ICLR 2027

SMILE: Small-Large Collaboration for Efficient Long-Term MLLM Personalization

Abstract

Enabling general-purpose MLLMs to adapt to each user's unique traits, habits, and evolving preferences is essential for building next-generation personalized multimodal assistants. Existing methods for personalizing MLLMs into lifelong human companions often adopt an ahead-of-time, query-agnostic context management strategy. They pre-compress and condense interaction histories into memories after each interaction, risking the loss of personalized multimodal evidence required in subsequent user queries. To overcome this limitation, we posit that an MLLM should browse the complete raw interaction history just-in-time conditioned on each incoming personalized query, and introduce SMILE, a novel SMall-large collaboration framework for effIcient Long-tErm MLLM personalization. SMILE breaks "one-model-fits-all" paradigm, specializing a Small MLLM (S-MLLM) as a lightweight yet intelligent personalized context distiller and employing a Large MLLM (L-MLLM) as a powerful but expensive personalized context reasoner. Specifically, given a contextualized and idiosyncratic user query, the S-MLLM learns to distill compact personalized context from the long raw interaction history, and hand it off to the frozen L-MLLM for reasoning and response generation, delivering efficient and effective customized assistance through collaboration. Experiments on four diverse benchmarks show that SMILE achieves superior performance while handing off averagely about 2.6% of the interaction history to the L-MLLM. Moreover, since SMILE keeps L-MLLM untouched during training, it enables nearly "free-lunch" scaling of personalized reasoning at inference time by seamlessly plugging in larger and stronger L-MLLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Term MLLM Personalization, Small-Large Model Collaboration, Efficient MLLM Personalization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_EvXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMILE, a small-large collaboration framework for long-term multimodal personalization. A small MLLM is trained to distill query-conditioned personalized context from raw interaction histories, and a larger frozen MLLM reasons over the compact handoff. The method uses P2C-CoT supervised fine-tuning and Context Handoff Policy Optimization. Experiments on Persona-MME, LCMP, PersonaMem-v2, and LongPersona-MME report improved performance while passing only about 2.6% of the history to the large model.

Critical strengths

The problem is important: long-term multimodal personalization requires both efficient access to history and strong reasoning over selected evidence. The small-large division of labor is sensible, and the reported efficiency gains are appealing. The paper evaluates across several benchmarks and shows consistent improvements over Full, RAG, and dataset-specific baselines. The plug-in use of stronger large MLLMs is also practically attractive.

Critical weaknesses

The main issue is generalization. The small MLLM is specialized per benchmark in most experiments, and the training pipeline relies on powerful MLLM annotation and synthetic data, so the results do not yet establish transfer to new users, domains, or deployment histories. Comparisons are also somewhat asymmetric and do not comprehensively cover stronger learned retrieval or compression methods under matched compute. In addition, the efficiency analysis focuses on context passed to the large model but does not adequately account for annotation, training, and query-time distillation costs. LongPersona-MME is central to the long-history claims but is not available during review, limiting reproducibility.

Questions for authors
  1. How well does one small MLLM trained on one benchmark transfer to other benchmarks without benchmark-specific specialization?
  2. What is the end-to-end annotation and training cost, and how many queries are needed to amortize it?
  3. Can the authors compare against stronger learned retrieval or compression baselines under a matched inference budget?
  4. Can LongPersona-MME or a representative subset be released during review?
7 comments
AuthorsOpenReview ↗

Response to Reviewer EvXu (1/3)

We sincerely thank you for the thoughtful questions and positive assessment of our work. We are particularly encouraged by the your recognition of the importance of long-term multimodal personalization, the soundness of our small–large division of labor, the appealing efficiency gains, the consistent improvements across diverse benchmarks, and the practical value of plug-and-play scaling with stronger L-MLLMs. Below, we address the remaining questions and provide further clarifications.

[Weaknesses/Q1] Benchmark-specific specialization and generalization of the S-MLLM, particularly whether training on one benchmark can transfer to another.

We sincerely thank the reviewer for raising this important question regarding the generalization of SMILE. We would like to clarify that our paper has already included cross-distribution and cross-benchmark transfer evaluations without any additional S-MLLM specialization, which provide direct empirical evidence of its generalization ability.

(1) Why benchmark-specific specialization is adopted in some experiments. The different training configurations are motivated by the intrinsic differences in benchmark designs and task formulations, rather than a reliance on benchmark-specific architectural designs or heuristics. Specifically, Persona-MME focuses on user-centric personalization across seven dimensions, with MCQ-based evaluation. PersonaMem-v2 targets implicit user preferences and involves both MCQ and open-ended response generation, with its training supervision constructed from explicitly provided evidence snippets. In contrast, LCMP is concept-centric and emphasizes reasoning over evolving attributes, temporal conflicts, and short-term events jointly with long-term knowledge. Its training supervision is constructed around concept-level histories and annotated key points, with different evaluation protocols. Consequently, their supervision structures and task-specific objectives are inherently different. We therefore construct benchmark-aligned training collections to provide appropriate supervision, rather than forcing incompatible task formulations into a single training configuration. Importantly, this does not imply that SMILE relies on benchmark-specific model architectures or manually designed retrieval rules.

(2) Direct transfer to unseen users and substantially different history distributions. More importantly, we have already demonstrated transfer without additional specialization in two settings:

  • Persona-MME →\to LongPersona-MME: We directly apply the S-MLLM trained on Persona-MME to our newly constructed LongPersona-MME, without any additional training or adaptation. Notably, LongPersona-MME contains 50 entirely new personas, with no overlap in personas, dialogues, or images with Persona-MME or the S-MLLM's training collection. Despite this distribution shift and substantially longer interaction histories (up to 256K tokens), SMILE achieves 73.76% accuracy, substantially outperforming Full (64.67%) with the same Qwen3.5-9B L-MLLM, an absolute gain of 9.09 percentage points. This directly demonstrates generalization to unseen users and independently constructed multimodal interaction histories.

  • PersonaMem-v2-32K →\to PersonaMem-v2-128K: We also evaluate the same specialized S-MLLM on the 128K setting after training on PersonaMem-v2, without additional specialization. With Qwen3.5-9B as the L-MLLM, SMILE achieves 37.16% MCQ accuracy and 35.39% open-ended accuracy, compared with 31.38% and 25.54% for Full, respectively. This demonstrates robust transfer to substantially longer interaction histories within the same benchmark.

[Weaknesses/Q4] Release of LongPersonaMME during review process.

We sincerely thank you for this suggestion. Due to the double-blind review requirements and the substantial size of LongPersona-MME, hosting the full dataset or a representative subset on a publicly accessible platform during the review period may compromise author anonymity. To provide further insight into the dataset, we have additionally included some representative data examples in the appendix, to eliminate your concerns. We are fully committed to releasing LongPersona-MME upon acceptance, as stated in the paper, to facilitate future research and reproducibility.

AuthorsOpenReview ↗

Response to Reviewer EvXu (2/3)

(Part 2/3)

[Weaknesses/Q2] (1) How about the training and annotation overhead? (2) How about the query-time distillation costs.

We sincerely thank you for raising this question regarding the training cost of SMILE. We would first like to clarify that SMILE is deliberately designed to minimize training overhead: we keep the expensive L-MLLM completely frozen and only optimize lightweight LoRA adapters on the S-MLLM. Moreover, we use a single frozen Qwen3.5-9B as the CHPO interpreter, allowing the resulting S-MLLM checkpoint to be directly paired with substantially stronger L-MLLMs (e.g., Qwen3.5-27B and GPT-5.6-Sol) without any additional training. Furthermore, the entire annotation pipeline is automated, requiring no manual annotation effort. We clarify the costs from two complementary perspectives below.

(1) End-to-end Annotation and Training Cost. SMILE incurs limited offline costs by design. First, the entire training data annotation pipeline is fully automated, requiring no manual annotation effort. Specifically, a powerful MLLM (GPT-5.6-Sol) automatically performs oracle interaction identification and deletion-only context pruning. Second, we only optimize lightweight LoRA adapters (rank 16) on the language-model component of the 4B S-MLLM, while keeping its vision encoder and projector frozen. The L-MLLM remains completely frozen throughout training. In practice, a full specialization on Persona-MME takes only approximately 16 hours on two NVIDIA RTX 6000D GPUs, including 12 hours for SFT and 4 hours for CHPO. More importantly, CHPO consistently uses the same frozen Qwen3.5-9B as its interpreter, regardless of the L-MLLM used at inference time. Consequently, a single trained S-MLLM checkpoint can be directly paired with different and substantially stronger L-MLLMs (e.g., Qwen3.5-27B and GPT-5.6-Sol) without any additional training. This design substantially reduces the offline training investment while enabling flexible test-time scaling.

(2) Complete Per-query Inference Cost. We would also like to clarify that our reported inference FLOPs already account for the complete computational cost of both S-MLLM-based context distillation and L-MLLM-based personalized reasoning, rather than only the context passed to the L-MLLM. Specifically, as formalized in Appendix C, the total inference cost of SMILE consists of two components: processing the complete interaction history with the lightweight S-MLLM and reasoning over the compact context with the L-MLLM:

= O(N_S T_S)+O(N_L T_L).$$ Therefore, the query-time distillation cost is explicitly included in our efficiency evaluation. Even after accounting for both models, SMILE requires only **65.6% of the inference FLOPs of Full on Persona-MME (9B)** and further reduces this to **55.5% on LongPersona-MME (9B)**, demonstrating that the efficiency gains are not merely achieved by shifting computation to the small model.
AuthorsOpenReview ↗

Response to Reviewer EvXu (3/3)

(Part 3/3)

[Weaknesses/Q3] Comparison with stronger learned retrieval or compression baselines under a matched inference budget.

We sincerely thank the reviewer for this valuable suggestion. Our original baseline selection follows the established evaluation protocol of PersonaVLM, and PersonaVLM (CVPR 2026) itself was a state-of-the-art method for long-term MLLM personalization. SMILE already substantially outperforms it, demonstrating the effectiveness of our approach over existing learned memory-based methods.

Nevertheless, to further address your concerns and provide a more comprehensive comparison, we additionally introduce two stronger learned baselines, PPRO (2026 arxiv) and LLMLingua-2 (2024 ACL Findings), covering learned retrieval and learned context compression, respectively. PPRO constructs episodic and semantic memory banks and optimizes user query rewriting via GRPO with retrieval and downstream answer quality as feedback. LLMLingua-2 formulates context compression as token-level classification, using a data-distilled bidirectional Transformer encoder to identify and preserve essential tokens while removing redundant ones.

We evaluate all methods under a matched inference budget to ensure a fair comparison. As shown in the table below, SMILE consistently achieves substantially better performance. The reasons are: (1) PPRO rewrites queries without accessing the underlying memory banks, limiting its ability to resolve highly personalized and context-dependent queries beyond conventional semantic retrieval, while its pre-constructed and compressed memories may further discard fine-grained evidence before future queries arise. (2) LLMLingua-2 adopts a task-agnostic compression objective without explicitly optimizing for personalized evidence relevance, making it less effective at preserving personalized-query-specific multimodal clues.

MethodMemoryIntentPreferenceBehaviorRelationshipGrowthAlignmentOverall
Qwen3.5-9B (Backbone)37.7657.6946.4043.9650.7264.4166.0854.29
+ PPRO74.1881.4360.7358.4376.0877.6186.3175.19
+ LLMLingua-261.3474.8053.4852.0768.9372.3980.1667.94
+ SMILE (Ours)86.9286.1268.1665.6978.2285.7794.9182.82

We sincerely hope that our detailed clarifications and additional empirical evidence have adequately addressed your concerns and resolved the remaining ambiguities. If so, we would greatly appreciate it if you could reconsider your assessment to our work. We would also be more than happy to further discuss any remaining questions or concerns.

Reviewer_qDNFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMILE, a small–large multimodal model framework for long-term personalization. A small model selects and prunes relevant interactions from a user's history, and a frozen larger model uses the resulting context to answer queries. The small model is trained using supervised initialization and Context Handoff Policy Optimization (CHPO). Experiments cover Persona-MME, PersonaMem-v2, LCMP-H and the newly constructed LongPersona-MME, and evaluate prediction performance, inference cost and transfer to longer histories and different readers.

Critical strengths
  • With the 9B reader, Persona-MME accuracy improves from 76.95 to 82.82 at 65.6% of Full's reported inference FLOPs (Table 1; Figure 3). The accounting includes both models. Appendix E.4 also reports measured runtime gains with the 27B reader and longer histories.
  • Table 3 separates supervised training from the two CHPO rewards. Combining the rewards improves over SFT on the reported benchmarks, including 80.89 to 82.82 on Persona-MME and 67.20 to 73.76 on LongPersona-MME.
  • The trained distiller works with stronger readers. The Persona-MME checkpoint also transfers to LongPersona-MME without further training; its 50 new users have personas, histories and images stated to be disjoint from Persona-MME and the training collection.
Critical weaknesses
  • Table 3 does not isolate the second pruning pass. The localizer already selects relevant turns and images, but there is no comparison that sends those same interactions to the reader verbatim. The extra pass may reduce context, but its accuracy and cost benefit over learned selection alone is unmeasured.
  • LongPersona-MME is designed to require historical images and evidence from multiple interactions, but neither dependency is quantified (Appendix D.4). The profile-only answerability filter does not measure them. LCMP-H tests integration across records; it does not establish how often the newly constructed LongPersona-MME questions require it.
  • The selection of β, τ and the final checkpoint is not described. The main Persona-MME score of 82.82 is also the best result in both Table 6 sweeps. Across β settings, accuracy ranges from 78.57 to 82.82, compared with 80.89 for SFT. The additional CHPO gain therefore needs a stated validation split or prior selection rule.
  • The LongPersona-MME Overall score cannot be reproduced from the reported definitions. Appendix D.2 defines primary-task accuracy and the optional Personality Alignment probe separately, while Table 4 lists 621 MCQs and 124 alignment pairs. The paper does not specify how these enter Overall or whether averaging is over items, users or runs. This affects the 27B comparison, with Full at 73.23 and SMILE at 75.95, and the GPT comparison, with Full at 71.79 and SMILE at 76.64.
  • Section 4.2 attributes PersonaVLM's limitations partly to evidence loss during memory management and lack of direct raw-history access for query expansion. Yet the original method retains raw dialogue turns and full images in episodic memory and performs multi-round retrieval [1]. Appendix D.1 says only the backbone is changed. The score gap does not show whether needed evidence was lost during storage, missed by retrieval, or not used in reasoning.

[1] Nie et al. PersonaVLM: Long-Term Personalized Multimodal LLMs, 2026.

Questions for authors
  1. What does pruning add over passing the selected interactions verbatim, in accuracy, context size and total two-model cost? Please keep the trained localizer, selected turn/image IDs and reader fixed.
  2. How many LongPersona-MME questions require historical images, and how many require more than one supporting interaction? Please give evidence-grounded examples and, if available, matched image-removal or single-interaction results.
  3. Were β, τ and the checkpoint fixed before inspecting the final Persona-MME results? What validation split or prior rule selected them, and was Table 6 produced afterward?
  4. Please give the evaluated counts and averaging rule behind the LongPersona-MME Overall scores, including whether and how the 621 MCQs and 124 alignment pairs contribute to Overall. Do the Full-27B/GPT baselines and their SMILE variants use the same evaluation items?
  5. Please describe what PersonaVLM's episodic retrieval returns in Appendix D.1, including any summaries, linked raw turns and full images. How does that retrieval path differ from the direct raw-history access for query expansion discussed in Section 4.2, and what evidence separates storage loss from retrieval or reasoning failure?
5 comments
AuthorsOpenReview ↗

Response to Reviewer qDNF (1/3)

We sincerely thank you for the thoughtful questions and constructive comments, as well as the highly positive assessment of our work, particularly regarding: (1) the substantial personalization gains with reduced inference FLOPs; (2) the effectiveness of combining the two CHPO rewards; and (3) the strong transferability of the trained distiller to stronger readers and unseen long-history benchmarks. Below, we make our best efforts to address the remaining concerns.

[W1/Q1] The motivation and effectiveness of the pruning stage.

We sincerely thank you for raising this question about the motivation and effectiveness of the pruning stage. In essence, as stated in our method section, our pruning serves as a conservative safeguard against potentially excessive context overhead, rather than an aggressive compression mechanism. Consider a scenario where an interaction contains corrupted or noisy spans (e.g., a large number of garbled characters) surrounding otherwise useful information. Passing such content to the L-MLLM not only incurs unnecessary token costs but may also introduce noise into downstream reasoning. To address this, we deliberately constrain the S-MLLM to deletion-only pruning, preventing it from rewriting or hallucinating historical content. This design is particularly suitable for the limited capacity of the S-MLLM, as simple deletion of non-essential content is much more reliable for a small model than abstracting or regenerating personalized information from scratch. Consequently, pruning primarily removes clearly non-informative content while having minimal impact on normal, information-dense interactions.

To further address your concern, we additionally compare SMILE with a variant that removes the pruning stage while keeping the localizer and selected interactions unchanged (w/o pruning). As shown in the table below, pruning reduces the handed-off context length and thus the L-MLLM's inference overhead, while causing only small changes in performance, further validating its role as a conservative safeguard rather than an aggressive compression mechanism. Although the reduction may be modest on clean benchmark histories, this conservative safeguard becomes particularly valuable in practical and real-world scenarios involving corrupted or excessively noisy interactions.

MethodAverage ACC (%)Context Ratio (%)
Qwen3.5-9B (Backbone)54.29-
SMILE w/o Pruning82.418.6
SMILE (Ours)82.826.7

[W2/Q2] The multimodal and multi-interaction dependencies of LongPersona-MME. Specifically, what proportion of questions require historical images to answer, and what proportion require integrating multiple historical interactions?

We sincerely thank you for this valuable question. To quantify both dependencies, we annotate all LongPersona-MME MCQs with GPT-6.1-Terra. Specifically, the annotator reads the complete interaction history together with all its images, and marks every history turn that supports the gold option over the three distractors. Based on these annotations, 29.0% of the questions appear to involve historical images that are relevant to answering them. Moreover, 40.9% of the questions require integrating multiple historical interactions, i.e., their supporting turns span two or more separate interactions. Therefore, a substantial portion of LongPersona-MME questions require multi-turn interactions to resolve.

[W3/Q3] The selection of the two hyperparameters, β\beta and τ\tau. How were their values determined?

Yes, they were fixed before the final Persona-MME results were inspected. β\beta and τ\tau were not tuned: they were kept at the default values of our implementation. The CHPO checkpoint (step 200 of 400) was selected on a held-out validation split of 60 Persona-MME training users (404 questions), which are excluded from CHPO training and disjoint from the 200 evaluation users. Table 6 was produced afterwards on the test set, as a sensitivity analysis only. We will state this protocol in Appendix D.5.

[W4/Q4] The evaluation protocol and score aggregation of LongPersona-MME. How are the reported Overall scores computed?

We apologize for the lack of clarity. The Overall score is computed as the item-weighted accuracy over all evaluated items, including the primary MCQs across the six dimensions and the Personality Alignment pairs. Each alignment pair contributes two binary judgments, one for each candidate response. Specifically, we first aggregate the correct predictions across all evaluated items and divide by the total number of scored items, rather than taking an unweighted average of the individual dimension accuracies. We have further clarified this aggregation procedure in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer qDNF (2/3)

(Part 2/3)

[W5/Q5] Positioning and comparison of PersonaVLM. (1) Add a description of what PersonaVLM's episodic retrieval returns. (2) Clarify the limitations of its retrieval paradigm compared with directly accessing historical interactions for query expansion. (3) Clarify the limitations of its ahead-of-time memory compression. (4) Is its main limitation retrieval failure or evidence loss?

We sincerely thank you for the insightful questions and suggestions regarding our PersonaVLM baseline.

(1) Description of Episodic Retrieval. First, following your suggestion, we have revised Appendix D.1 to provide a more detailed description of PersonaVLM's episodic retrieval mechanism, including summaries, linked raw turns and full images. We appreciate the reviewer for pointing out that this aspect was insufficiently described in the original manuscript.

(2) Limitations of its retrieval paradigm. To clarify this distinction, we first briefly revisit how PersonaVLM performs memory retrieval:PersonaVLM first uses an MLLM to determine whether additional historical information is needed. If retrieval is triggered, the MLLM generates retrieval keywords and a time range based on the current query, only recent dialogue context, and user profile, followed by parallel retrieval from the pre-constructed semantic, episodic, and procedural memory banks. Therefore, a fundamental limitation is that the initial retrieval-condition generation does not directly access the complete long-term raw interaction history. Long-term personalized queries are often highly contextual and implicit, and their intended historical references may not be sufficiently expressed by the current query and recent dialogue alone. Consequently, even with multi-round MLLM-guided retrieval, the effectiveness of retrieval remains fundamentally constrained by the information available when formulating the initial retrieval conditions, making it difficult to fully resolve the limitations of conventional semantic retrieval. In contrast, the direct access to raw interaction history for query-conditioned evidence localization advocated in our work can mitigate this limitation, and SMILE is built upon precisely this design philosophy.

(3) Limitations of pre-constructed memory representations. Furthermore, although PersonaVLM retains the complete raw interaction history in its episodic memory, not all of its memory types preserve information at the original level of granularity. In particular, its semantic memory proactively extracts, abstracts, and organizes historical information into higher-level semantic knowledge and user preferences. Some visual concepts may even be represented solely by textual descriptions rather than their original images. Our concern regarding ahead-of-time compression specifically refers to this semantic memory: since it is also retrieved and used for answering queries, such pre-constructed abstractions may fail to preserve fine-grained evidence required by future queries. Nevertheless, as discussed above, the more fundamental limitation of PersonaVLM lies in its memory-mediated retrieval process, which may fail to retrieve the evidence required by highly contextualized queries.

AuthorsOpenReview ↗

Response to Reviewer qDNF (3/3)

(Part 3/3)

(4) As discussed in our responses to (2)–(3), PersonaVLM's primary limitation lies in retrieving the correct evidence, while this limitation is intertwined with the potential information loss introduced by its ahead-of-time memory abstraction. Specifically, since PersonaVLM retrieves from multiple memory types, while not all memories adopt a pre-compression mechanism, it is difficult to completely disentangle retrieval failures from information loss during memory construction.

Nevertheless, to further eliminate your concerns, we inspected PersonaVLM's reasoning chains on LongPersona-MME: on all evaluation prompts, it issued exactly one retrieval round and then answered, never refining its query after seeing the retrieved memories. Its multi-round retrieval therefore cannot compensate for retrieval conditions formed without access to the raw history, contributing to its inferior downstream performance. Furthermore, maintaining multiple heterogeneous memory types and increasingly complex retrieval pipelines over long-term interactions introduces substantial maintenance overhead and accumulated noise, potentially making memory consistency and reliable evidence retrieval increasingly difficult. In contrast, SMILE deliberately avoids such elaborate memory structures and multi-round retrieval, adopting a simple and direct query-conditioned localization paradigm that is more robust and ultimately achieves substantially stronger performance.

We hope that our responses and additional experiments have helped clarify the key concerns and further demonstrate the merits of our approach. We would be grateful if these clarifications could be taken into consideration in your overall evaluation of the paper. We sincerely appreciate your time and constructive feedback, and would be happy to further discuss any remaining concerns.

Reviewer_4hqbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses long-term multimodal personalization. It argues that memory-based methods compress history before the query arrives and may discard details needed later, while semantic retrieval struggles with anaphoric, context-dependent personalized queries. The proposed SMILE uses a 4B small MLLM to read the full raw history, select query-relevant interactions, and prune them via deletion-only edits into a compact context, which is handed to a frozen large MLLM for answering. The small model is trained with SFT on GPT-annotated trajectories, followed by CHPO, a GRPO-based stage whose reward combines recall-biased localization with the large model’s answer confidence. Across four benchmarks, including a newly built 256K-token LongPersona-MME, SMILE outperforms Full-context, RAG, and memory-based baselines while passing only about 2.6% of the history to the large model, and it can be paired with stronger large models without retraining.

Critical strengths
  1. Exceptionally clear writing and presentation. The paper moves coherently from the ahead-of-time vs. just-in-time contrast, through the asymmetry analysis, to the small-large collaboration design. Figures 1 and 2 are intuitive, and the appendix provides pseudocode, full prompts, and training samples that aid understanding and reproduction. Insightful problem decomposition. The authors assign token-intensive context localization and reasoning-intensive inference to models of different scales. This perspective also exposes the fundamental limitations of memory-based methods and semantic retrieval in personalization.
  2. Modular, plug-and-play design. The L-MLLM stays frozen, and the S-MLLM is trained once and paired directly with 9B, 27B, or proprietary models, transferring to the 256K-token LongPersona-MME without retraining. This is valuable for real-world systems where the large model cannot be fine-tuned.
  3. Strong long-context gains at low cost. For example, on LongPersona-MME (9B), SMILE improves over Full from 64.67 to 73.76 while handing off only about 1–7% of the history. The gains hold consistently across three distinct types of personalization benchmarks. Rigorous training design. The over-selection penalty, deletion-only pruning with a validator, hiding options from the S-MLLM, and filtering unanswerable samples effectively guard against reward hacking and information leakage.
  4. A new evaluation resource. LongPersona-MME extends histories to 256K tokens with no overlap with existing data, filling a gap in evaluating personalization over ultra-long histories.
Critical weaknesses
  1. Missing comparison with related context-compression methods. Using a small model to select or compress context for a larger reader has been explored in RAG and long-context work (e.g., RECOMP, CompAct, LongLLMLingua). However, the experiments compare only against a fixed top-5 RAG and memory-based methods. I suggest adding comparisons with such methods, as well as a RAG baseline that rewrites the query before retrieval.

  2. The overall compute savings are smaller than the abstract suggests. The paper emphasizes that only about 2.6% of the history is passed to the large model, but for every query the 4B small model still reads the entire history from scratch, and this cost grows as the history grows. By the paper’s own numbers, total compute is only about 35–45% lower than Full and clearly higher than RAG. Moreover, in multi-turn use Full can cache the history it has already processed, which may lower its actual cost further; this is not discussed.

Questions for authors
  1. Effect of interpreter scale: CHPO uses a 9B model as the interpreter during training. Switching to 27B or GPT-5.6-Sol yields only small gains on Persona-MME (84.20 vs. 84.28). Are the distilled contexts biased toward what suits the 9B model? Would training with a stronger interpreter further improve performance on the large-model side?

  2. Non-multiple-choice settings: The handoff reward relies on the L-MLLM’s probability margin over answer options. For free-form or open-ended tasks, the paper proposes an MLLM-as-Judge alternative but reports no experiments. Could you provide results for this variant, or describe how it differs from the multiple-choice version in training stability and effectiveness?

  3. Inconsistent CR values: Persona-MME CR is 2.58% in the abstract and Table 3 but 6.7% in Table 7 and Fig. 3a. Could you clarify how these are computed differently?

  4. Stability of results: Why does using either reward alone underperform SFT on Persona-MME (80.39 and 78.46 vs. 80.89)? Could you report means and variances over multiple random seeds?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4hqb (1/4)

We sincerely thank the reviewer for the thoughtful questions and highly positive assessment of our work. We are particularly encouraged by the reviewer’s recognition of our clear presentation and insightful problem decomposition, the soundness of our small–large collaboration design, the modular and plug-and-play nature of SMILE, its substantial personalization gains with low computational overhead, the rigor of our training design, and the value of LongPersona-MME as a new evaluation resource for ultra-long multimodal personalization. Below, we address the remaining questions and provide further clarifications.

[W1] Comparison with context-compression methods, and a RAG method that rewrites the query before retrieval.

We sincerely thank you for this valuable suggestion. Our original baseline suite already includes strong state-of-the-art methods for long-term MLLM personalization at the time of submission, namely PersonaVLM (CVPR 2026) and TAME (KDD 2026). Our baseline selection also follows the established evaluation settings of prior work, particularly PersonaVLM. SMILE already demonstrates substantial improvements over these strong baselines.

Nevertheless, to further address your concern, we follow the suggestion and introduce two additional baselines: (1) LLMLingua-2 (ACL 2024 Findings), a learned, task-agnostic prompt compression method, and (2) PPRO (2026 arXiv), a personalized retrieval framework that optimizes query rewriting before memory retrieval. The results are presented in the table below.

As shown, SMILE substantially outperforms both methods. We attribute this performance gap to their different context management objectives: (1) LLMLingua-2 adopts a task-agnostic compression objective without explicitly optimizing for personalized evidence relevance, making it less effective at preserving multimodal clues specific to personalized queries. (2) PPRO rewrites queries without directly accessing the underlying long-term memory during query reformulation, limiting its ability to resolve highly personalized and context-dependent queries beyond conventional semantic retrieval. Moreover, its pre-constructed and compressed memory representations may discard fine-grained evidence before future queries arise.

MethodMemoryIntentPreferenceBehaviorRelationshipGrowthAlignmentOverall
Qwen3.5-9B (Backbone)37.7657.6946.4043.9650.7264.4166.0854.29
+ PPRO74.1881.4360.7358.4376.0877.6186.3175.19
+ LLMLingua-261.3474.8053.4852.0768.9372.3980.1667.94
+ SMILE (Ours)86.9286.1268.1665.6978.2285.7794.9182.82
AuthorsOpenReview ↗

Response to Reviewer 4hqb (2/4)

(Part 2/4)

[W2] (1) How should we understand SMILE's computational cost as interaction histories grow? (2) What are the computational costs and limitations of RAG in this setting? (3) How does KV caching affect the computational cost of Full, and can SMILE benefit from the same optimization?

We sincerely thank you for this valuable question. We would like to clarify three key points. First, the computational cost of the L-MLLM is substantially higher than that of the lightweight S-MLLM (as also evidenced by our reported FLOPs, where the FLOPs of SMILE already explicitly account for both models' computational costs). Thus, passing only 2.6% of the history to the expensive L-MLLM is precisely how SMILE achieves substantial overall savings, despite the S-MLLM processing the full history. Second, as discussed in our paper, conventional RAG is inherently inadequate for highly personalized and context-dependent queries, while a history-aware RAG capable of resolving such queries would incur computational costs comparable to or even higher than SMILE. Third, history caching (e.g, KV cache) is equally applicable to SMILE and can further reduce its inference cost, rather than being an exclusive advantage of Full. We elaborate on these points below.

(1) Computational scaling with increasingly long histories. We would like to clarify that the absolute computation of all methods increases as interaction histories grow. However, the key difference lies in which model bears this cost. Full exposes the expensive L-MLLM to the entire history, whereas SMILE delegates history reading and query-conditioned evidence localization to the substantially smaller S-MLLM, reserving the L-MLLM for always reasoning over compact, relevant contexts. This is precisely the motivation behind our small–large division of labor. Empirically, SMILE requires only 65.6% of Full's FLOPs on Persona-MME (9B), and this further decreases to 55.5% on LongPersona-MME (9B), where histories extend to 256K tokens. This demonstrates that SMILE's relative computational advantage becomes more pronounced as histories grow.

(2) Computational cost of RAG. As discussed in the paper, conventional embedding-based RAG relies on surface-level semantic similarity and is inadequate for highly personalized, implicit, and context-dependent queries. Simply increasing the retrieval budget does not fundamentally resolve this evidence-localization problem. A more capable history-aware RAG would need to first access and understand the broader interaction history to expand or disambiguate the query before retrieval. However, this additional history-reading and query-expansion process itself incurs substantial computation, making its overall cost comparable to or even higher than SMILE. SMILE addresses this challenge more directly by integrating history understanding and evidence localization into a specialized lightweight model, rather than relying on increasingly expensive retrieval pipelines.

(3) KV caching in multi-turn interactions. We agree that Full can benefit from persistent KV caching when processing repeated queries over a shared history. However, this optimization is also applicable to SMILE. The S-MLLM can also cache previously processed historical prefixes, avoiding redundant computation across subsequent queries. Crucially, the cached history is maintained by the lightweight S-MLLM, while the expensive L-MLLM still receives only the brief and compact context. Therefore, KV caching can further reduce SMILE's cost without eliminating its fundamental computational advantage over Full, whose large reader must maintain and process the full historical context.

AuthorsOpenReview ↗

Response to Reviewer 4hqb (3/4)

(Part 3/4)

[Q1] (1) Why do we choose a 9B model as the training interpreter, and does it introduce a bias toward the 9B reader? (2) Can using a stronger interpreter further improve performance?

We sincerely thank the reviewer for this thoughtful question.

(1) Choice of the 9B interpreter and potential bias. We choose Qwen3.5-9B as the frozen interpreter primarily to balance supervision quality and training efficiency, as it provides effective handoff supervision while keeping the CHPO cost relatively low. Importantly, the resulting S-MLLM generalizes well to substantially stronger inference backbones without retraining. On Persona-MME, pairing the same S-MLLM with Qwen3.5-27B and GPT-5.6-Sol improves accuracy from 82.82% to 84.20% and 84.28%, respectively. Notably, the results of these models already approaches its reasoning ceiling. (We estimate the reasoning ceiling of Qwen3.5-27B by providing it with oracle contexts, achieving 87.63%, compared with 84.20% achieved by SMILE). More importantly, this cross-backbone scaling benefit is even more pronounced on PersonaMem-v2. On the 32K setting, replacing the 9B reader (L-MLLM) with Qwen3.5-27B and GPT-5.6-Sol improves ACC-M by 27.4% and 31.7%, and ACC-O by 12.8% and 61.9%, respectively, in relative terms. These substantial gains across different backbones in a plug-in-and-play manner demonstrate that the distilled contexts can effectively support substantially stronger readers.

Regarding potential interpreter bias, it is worth clarifying that the interpreter does not directly supervise the entire optimization of S-MLLM process. The location reward is computed independently from the ground-truth historical interactions, while the handoff reward evaluates the final answer through its probability margin rather than directly enforcing a particular response style or context format. Moreover, to isolate context handoff quality from the interpreter's own reasoning limitations, we exclude samples that it cannot answer correctly even with the ground-truth context when computing the handoff reward. This design mitigates the potential confounding effect of the interpreter's reasoning limitations.

(2) Can a stronger interpreter further improve performance? In principle, a stronger interpreter could provide more reliable handoff supervision by reducing its own reasoning errors, and could also make more training samples eligible for the handoff reward through the aforementioned ground-truth-context filtering. However, this comes at additional training-time computational cost.

Nevertheless, to further resolve your concerns, we conduct an additional experiment using Qwen3.5-27B as the CHPO interpreter and evaluate the resulting S-MLLM with the same Qwen3.5-27B reader, on PersonaMME benchmark. As shown in the table below, the stronger interpreter brings an additional performance improvement, supporting our analysis. Nevertheless, the gain is relatively modest, partly because some questions remain beyond the reasoning capability of the 27B reader even when provided with correct contexts (as discussed above). This suggests that the 9B interpreter already provides an effective and computationally economical supervision signal, enabling the resulting S-MLLM to benefit from substantially stronger readers in a plug-and-play manner.

CHPO InterpreterInference L-MLLMOverall ACC (%)
Qwen3.5-9BQwen3.5-27B84.20
Qwen3.5-27BQwen3.5-27B84.93
Reviewer_6DKLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SMILE, a small-large collaboration framework for long-term MLLM personalization. It resolves the efficiency and effectiveness issues of prior ahead-of-time and just-in-time methods. Specifically, a lightweight small model scans the full raw multimodal history to distill query-dependent clues (about 2% of the original tokens), while a frozen large model generates the final response. Trained with a two-stage SFT plus RL recipe, SMILE achieves good results across four benchmarks.

Critical strengths

(a)Simple and Effective Method: Using a small model to read the entire history and reserving the large model for reasoning provides a practical balance between cost and information retention.

(b)Sound Training and New Benchmark: The SFT and RL design is intuitive and well-ablated. The newly constructed LongPersona-MME benchmark also offers a valuable testbed for 256K ultra-long context.

(c) Memory-Construction-Free: The method avoids the overhead and potential loss of building persistent memory databases, outperforming baselines like RAG and PersonaVLM.

Critical weaknesses

There are several limitations that need to be clarified: (a) Efficiency and scalability: Although it avoids building memory, feeding the entire raw history into the small model may still cause latency and scalability issues as interactions continue to grow.

(b) Context Accuracy: Since there is no multi-round retrieval, it is unclear how accurate the one-pass context extraction is, and how much the internal CoT specifically contributes to locating evidence.

(c) Limited gain from RL: Table 3 indicates that the major performance jump comes from the SFT stage, while the additional gain from RL appears rather marginal.

Questions for authors

There are several limitations that need to be clarified: (a) Efficiency and scalability: Although it avoids building memory, feeding the entire raw history into the small model may still cause latency and scalability issues as interactions continue to grow.

(b) Context Accuracy: Since there is no multi-round retrieval, it is unclear how accurate the one-pass context extraction is, and how much the internal CoT specifically contributes to locating evidence.

(c) Limited gain from RL: Table 3 indicates that the major performance jump comes from the SFT stage, while the additional gain from RL appears rather marginal.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6DKL (1/2)

We sincerely thank the reviewer for the thoughtful questions and highly positive assessment of our work. We are particularly encouraged by the reviewer’s recognition of the simplicity and effectiveness of our small–large collaboration paradigm, the soundness of our SFT and RL training design, the value of LongPersona-MME as a new benchmark for 256K ultra-long-context personalization, and the benefits of avoiding persistent memory construction while retaining fine-grained historical information. Below, we address the remaining concerns and provide further clarifications.

[W(a)/Q(a)] How does SMILE scale with increasingly long interaction histories?

First, we emphasize that SMILE is specifically designed to accommodate continuously growing interaction histories. Its computational cost can be expressed as:

FLOPsSMILE=O(NSTS)+O(NLTL),\mathrm{FLOPs}_{\mathrm{SMILE}}=O(N_S T_S)+O(N_L T_L),

where the S-MLLM processes the complete history, while the expensive L-MLLM only receives the compact context. Consequently, as the history grows, the L-MLLM's computational cost remains bounded as long as the handed-off context is maintained within a compact budget, while only the substantially cheaper S-MLLM incurs additional computation. This is precisely the motivation behind our small–large division of labor.

Our empirical results further validate this design. As history length increases from up to 128K tokens on Persona-MME to 256K tokens on LongPersona-MME, SMILE's FLOPs relative to Full decrease from 65.6% to 55.5%, demonstrating an increasingly favorable computational advantage under longer histories. Notably, Full is already a strong baseline in this setting. Regarding RAG, conventional semantic retrieval is inherently inadequate for highly personalized and context-dependent queries. A more suitable history-aware RAG would need to access and understand the broader interaction history for query expansion before retrieval, introducing additional computational costs that can be comparable to or even higher than those of SMILE.

Finally, we emphasize that any effective method accessing continuously growing histories inevitably incurs additional computation. SMILE's key advantage is not eliminating this cost, but strategically shifting most of the additional history-reading burden to the substantially cheaper S-MLLM (as also explicitly analyzed in Appendix C, Complexity Analysis).

[W(b)/Q(b)] How accurate is one-pass context extraction, and how much does the proposed CoT contribute to evidence localization?

First, we would like to clarify that SMILE does not simply perform a conventional single-round retrieval. Rather, its design follows a different paradigm: the specialized S-MLLM directly accesses the complete interaction history and performs query-conditioned evidence localization, eliminating the need for iterative retrieval over separately indexed historical records. For histories exceeding the S-MLLM's context window, we further employ chunk-based processing.

To quantitatively validate its localization accuracy, we additionally evaluate the evidence recall against ground-truth supporting interactions. SMILE achieves 81.3% evidence recall on Persona-MME, demonstrating its ability to reliably identify the historical evidence required by personalized queries.

Furthermore, to isolate the contribution of P2C-CoT, we introduce an additional ablation that skips the explicit two-stage CoT process and directly supervises the S-MLLM with the final compact contexts. We compare this variant with our original P2C-CoT SFT on Persona-MME, as shown below.

SFT VariantPersona-MME ACC (%)
Direct Compact-Context SFT (w/o P2C-CoT)75.84
P2C-CoT SFT (Ours)80.89

As shown, directly supervising the final compact context leads to substantially worse performance. This is because, for a relatively small model, simultaneously learning to locate relevant evidence from long histories and construct compact contexts is challenging without explicit intermediate guidance. P2C-CoT decomposes this process into evidence localization and context pruning, providing more fine-grained supervision that better facilitates the acquisition of reliable personalized context extraction capabilities.

AuthorsOpenReview ↗

Response to Reviewer 6DKL (2/2)

(Part 2/2)

[W(c)/Q(c)] How should we interpret the additional gains brought by RL?

First, we emphasize that the benefits of RL are not limited to Persona-MME. As shown in Table 3, combining both CHPO rewards consistently improves over SFT across all five benchmark settings. In particular, on LongPersona-MME, the performance increases from 67.20% to 73.76%, yielding a substantial gain of 6.56 percentage points.

Moreover, relatively modest improvements on certain benchmarks are expected. On the one hand, as discussed in our response to Reviewer 4hqb, Persona-MME is already approaching the reasoning ceiling of the L-MLLM, leaving limited room for further improvements through better context handoff. On the other hand, SFT primarily teaches the S-MLLM to imitate offline-annotated evidence localization and context pruning trajectories, whereas RL further optimizes its behavior for effective collaboration with the L-MLLM under actual inference conditions. Therefore, its benefits become particularly evident when generalizing to more challenging and substantially longer interaction histories. For example, the S-MLLM trained on Persona-MME transfers directly to LongPersona-MME without any additional training, achieving 73.76% accuracy, compared with 67.20% for the SFT-only variant, a substantial gain of 6.56 percentage points. This further demonstrates the effectiveness of RL in improving collaborative context localization and handoff under a substantially different history distribution.

Across all five benchmark settings, CHPO achieves an average absolute improvement of 2.56 percentage points over SFT, further validating the consistent benefits of jointly optimizing evidence localization and context handoff.

We sincerely appreciate the reviewer’s constructive feedback and thoughtful suggestions, which have helped us further clarify and strengthen our work. We hope our responses and additional experimental evidence have adequately addressed the remaining concerns. Thank you again for your time and consideration. We would be delighted to continue the discussion if any further questions arise.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary of SMILE

Dear ICLR 2027 Conference Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank you for the time and care devoted to reviewing our submission. As the discussion draws to a close, we would like to briefly summarize our contributions and the additional evidence and clarifications provided in response to your feedback.

SMILE makes personalized assistance over long multimodal interaction histories more practical. A specialized 4B S-MLLM reads the interaction history, locates and prunes query-relevant evidence, and passes only 2.58% of the history on average to a frozen L-MLLM. The same trained context distiller can be directly paired with 9B, 27B, and proprietary readers, allowing SMILE to benefit from stronger reasoning capabilities without additional training. On Persona-MME with the 9B reader, SMILE improves accuracy from 76.95% to 82.82%, while requiring only 65.6% of Full's inference FLOPs, accounting for the computational costs of both models.

In response to the reviewers' feedback, we have further strengthened the paper in four aspects:

  1. Transferability and benchmark accessibility. In response to Reviewer EvXu's follow-up questions, we evaluated the S-MLLM trained exclusively on Persona-MME directly on PersonaMem-v2 at both 32K and 128K, without any target-benchmark training or fine-tuning. SMILE outperformed the reported baselines on both metrics at both context lengths. We have incorporated these results into Appendix E.5 and released a representative subset of LongPersona-MME through our anonymous repository for independent inspection.

  2. LongPersona-MME evaluation and evidence requirements. In response to Reviewer qDNF's follow-up questions, we clarified the evaluation protocol and standardized the scoring of Full and SMILE across the same 869 evaluation items. Under the consistent scoring protocol, SMILE outperforms Full by 3.22 and 5.75 percentage points with Qwen3.5-27B and GPT-5.6-Sol, respectively. We also clarified the annotation criteria and reported that 29.0% of questions were judged to require historical images, while 40.9% were judged to require evidence spanning multiple historical interactions (Reviewer qDNF, W2/Q2 and W4/Q4).

  3. Understanding the sources of improvement. We isolated the effect of pruning by holding the selected interactions fixed, clarified our checkpoint-selection procedure using a separate validation split, and further explained PersonaVLM's episodic memory mechanism. We also added evidence-recall analyses and comparisons between structured reasoning supervision and direct supervision on the final compact context (Reviewer qDNF, W1, W3, and Q5; Reviewer 6DKL, W(b)).

  4. Comparisons with context compression and query rewriting. Appendix E.6 now includes additional comparisons with LLMLingua-2 and PPRO. On Persona-MME with the 9B reader, SMILE achieves 82.82% overall accuracy, compared with 67.94% for LLMLingua-2 and 75.19% for PPRO. We also clarified that the 6.7% context ratio reported in Table 7 specifically corresponds to Persona-MME (Reviewer 4hqb, W1 and Q3; Reviewer EvXu, Q3).

We are particularly encouraged by the positive assessments from Reviewers 4hqb and 6DKL, both of whom initially rated the paper 3 (Weak Accept). Reviewer 4hqb specifically recognized SMILE's modular design and its ability to achieve substantial accuracy gains with lower computational cost, and subsequently confirmed that our rebuttal had addressed all of their concerns and decided to raise the score (initially 3). We are also grateful for the further questions from Reviewers EvXu and qDNF, which motivated additional transfer experiments, greater benchmark accessibility, and a more detailed clarification of the LongPersona-MME evaluation.

At its core, our main contribution SMILE is the first framework for long-term personalization that explicitly divides labor between a specialized small MLLM for personalized evidence localization and a frozen large MLLM for downstream reasoning. As an additional contribution, we will also release the complete LongPersona-MME benchmark to facilitate further research on ultra-long multimodal personalization.

We sincerely appreciate the reviewers' constructive feedback and the opportunity to further strengthen our work. Thank you again for your time, thoughtful consideration, and valuable suggestions.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.