acceptodds
Under review as a conference paper at ICLR 2027

Know Me To Recommend Me: Personalized Recommendation from Long-Term Human-MLLM Assistant Interactions

Abstract

Imagine that in the near future, when you ask your personal Multimodal Large Language Model (MLLM) assistant, "Recommend a laptop that would suit me," it can recover your preferences from sparse, incidental cues accumulated through everyday interactions and directly identify the right item, without requiring you to repeatedly spell out all details. To bridge this research gap, we propose LIP-Rec, which studies how everyday experience between human and their MLLM assistants can be leveraged for personalized recommendation. To systematically evaluate such capability, we introduce PersonaRec, a large-scale benchmark that lays the foundation for moving personalized recommendation from modeling behavioral signals within recommendation platforms toward a new paradigm grounded in daily interactions with MLLM assistants. It comprises 200 distinct personas and over 30K human-refined candidate items, spanning three common daily-life domains: Fashion, Food, and Movies, and 21 sub-domains. For each persona, we construct 83.5K length daily multimodal interactions, where item preferences are sparsely and naturally revealed through explicit, implicit, and evolving preference patterns, and also be grounded in multimodal clues. We benchmark frontier MLLMs and find that they perform poorly on PersonaRec: they struggle to recover item preferences from long interaction histories and translate them into effective retrieval queries. Therefore, we propose Yo'Rec, a simple baseline method that targets these limitations and enhances open-source MLLMs on PersonaRec. Empirical results show its efficacy, offering a promising direction for LIP-Rec.

Primary area:
datasets and benchmarks
Keywords:
long-term human-MLLM assistant interactions, Personalized Recommendation, Personalized MLLM Assistants

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fZAROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LIP-Rec, a research task studying personalized recommendation from long-term human–MLLM assistant interactions. To support this task, it presents PersonaRec, a benchmark comprising 200 personas, over 30K candidate items across three domains, and LLM-synthesized multimodal interaction histories. The authors show that frontier MLLMs perform poorly on this benchmark and further propose Yo'Rec, which improves open-source MLLMs through a three-step CoT-based SFT, followed by GRPO-style training with preference and recommendation rewards.

Critical strengths
  • The proposed task addresses a clear and practically relevant problem, as personal assistants are gaining increasing attention for shopping and content recommendation.
  • The benchmark is substantial in scale and covers explicit, implicit, and evolving preferences in both text-only and multimodal sessions, providing a useful resource for future research.
Critical weaknesses

W1: Insufficient ablation of Yo'Rec's design. (1) In Table 3, the transition from the base model to the PRPL-CoT SFT variant (average R@1 from 4.03 to 11.45) changes both the inference procedure (three-step reasoning) and the model parameters through SFT. It is also unclear whether self-reranking is applied to each variant. Therefore, the contributions of structured reasoning and SFT are not clearly disentangled (see Q2). (2) Furthermore, the effectiveness of the proposed three-step reasoning structure itself remains unclear. Section 3.1 claims that explicit session localization reduces interference from irrelevant history, but this is not directly validated against simpler SFT variants, such as directly rewriting the query or extracting preferences without first localizing sessions (see Q3).

W2: Insufficient analysis of the benchmark's key properties. The paper highlights diverse preference dynamics (explicit, implicit, and evolving) and visually grounded inference as two key characteristics of PersonaRec, but Tables 2 and 3 only report domain-level aggregate results. There is no breakdown by preference type or by text-only versus multimodal interactions, making it unclear which properties contribute most to the benchmark's difficulty or where Yo'Rec provides improvements. In addition, performance on Movies is consistently much lower than on the other domains and relatively close to the Query-Only baseline. For example, under U-Query, GPT-5.6-Sol achieves an R@1 of 2.00 on Movies, compared with 8.20 on Fashion and 8.53 on Food, while Query-Only achieves 0.75 on Movies. However, the paper provides limited analysis of this performance gap.

Questions for authors

Q1: Section 5.2 discusses "Shopping Companion (Tou et al., 2026)", but the cited reference is ShoppingComp (arXiv:2511.22978). The work described appears to be Shopping Companion [1], which also recovers user preferences from multi-session histories for product recommendation and uses SFT followed by GRPO-style training with preference-related rewards. Could the authors correct the citation and clarify how their work differs from Shopping Companion in both benchmark design and training methodology?

[1] Yu, Zijian, et al. "Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks." arXiv preprint arXiv:2603.14864 (2026).

Q2: For Table 3, could the authors clarify whether self-reranking is consistently applied across all variants? Also, could the authors report results for the base model using the same three-step PRPL-CoT pipeline without SFT, while keeping the retrieval and reranking settings identical?

Q3: To validate the claim in Section 3.1, could the authors compare PRPL-CoT SFT with simpler SFT variants, such as directly rewriting the query, or extracting preferences without first localizing sessions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer fZAR (1/3)

Thank you for the careful reading and insightful questions and comments. We sincerely thank you for highly recognizing that (1) our proposed task, LIP-Rec, addresses a clear and practically relevant problem in the era of personalized AI assistants, and (2) PersonaRec is a large-scale benchmark that provides a valuable resource for future research. Below, we carefully address your remaining concerns.

[W1/Q2/Q3] (1) Whether self-reranking is consistently applied across all variants. (2) Comparison with a variant performing PRPL-CoT without SFT. (3) Comparisons with simpler SFT variants: (i) directly rewriting the query; (ii) extracting preferences without first locating relevant sessions.

Regarding (1), yes. We consistently apply the same self-reranking procedure across all designed ablation variants, controlling for reranking as a potential confounding factor and allowing the ablation to more directly reflect the contributions of SFT and individual RL rewards. We sincerely apologize for the lack of explanation.

Regarding (2) and (3), we conduct additional experiments comparing Yo'Rec with three variants: PRPL-CoT w/o SFT, Rewriting Only SFT, and w/o Session Localization SFT. The results are summarized below. We observe that none of these variants outperforms the full Yo'Rec.

Method (U-Query, averaged across three domains)R@1R@10N@10
Qwen3.5-9B + Self-Reranking6.2715.5110.39
PRPL-CoT w/o SFT7.2017.1011.80
Rewriting Only SFT9.1820.4614.39
w/o Session Localization SFT10.2721.5215.47
Yo'Rec (Full)13.5124.2818.79

Specifically, PRPL-CoT w/o SFT directly applies our three-step reasoning pipeline at inference time. However, general-purpose MLLMs are not necessarily equipped with such task-specific reasoning capabilities during pretraining, and thus struggle to execute this pipeline reliably without SFT.

For Rewriting Only SFT and w/o Session Localization SFT, directly processing the long histories makes it difficult to identify and aggregate sparse, query-relevant preferences amid abundant irrelevant or potentially misleading interactions. The former lacks explicit supervision for intermediate preference identification, while the latter skips the localization of relevant sessions. These results validate the necessity of our three-step PRPL-CoT design.

AuthorsOpenReview ↗

Response to Reviewer fZAR (2/3)

(Part 2/3)

[W2] (1) Tables 2 and 3 report only domain-level results, with no breakdown by preference type (explicit, implicit, evolving) or by text-only versus multimodal interactions. (2) Movies performance is much lower than the other domains and close to Query-Only, with limited analysis.

(1) Preference-type and modality-level analysis. We sincerely thank you for this valuable suggestion. The reason we do not originally report such fine-grained breakdowns is that preference types (explicit, implicit, and evolving) and modalities (text-only and multimodal) characterize individual interaction sessions, whereas each target item is associated with multiple preference-revealing sessions, potentially involving different preference types and modalities. Thus, a single recommendation instance may combine multiple such characteristics, making a direct final performance breakdown of item-level recommendation metrics less computable.

Nevertheless, following your suggestion, rather than analyzing only the final recommendation performance, we introduce an intermediate metric, Category-wise Preference Recovery Rate (CPRR), to examine preference recovery at the fine-grained level you suggested. Specifically, CPRR measures the proportion of ground-truth preferences successfully recovered within each category. We report CPRR across different preference-expression types (explicit, implicit, and evolving) and evidence modalities, categorized by where the preference value is conveyed: text-only, multimodal with the value in text, and multimodal with the value only in the image.

CPRR (%)ExplicitImplicitEvolvingText-onlyMultimodal (value in text)Multimodal (value only in image)
GPT-5.6-Sol70.2059.8054.4068.1061.3049.80
Qwen3.5-27B52.4040.6035.8049.7041.3029.60
Qwen3.5-9B31.6020.3015.9028.4021.6013.70
Yo'Rec62.7049.8043.6058.2050.4036.90

We observe that implicit and evolving preferences are more challenging to recover, while multimodal evidence also increases the difficulty. Nevertheless, Yo'Rec consistently improves the preference recovery performance of the base Qwen3.5-9B model across these challenging scenarios.

(2) Movies. We apologize for the limited discussion due to space constraints. Unlike the concrete attributes of fashion and food, movie preferences often involve abstract aspects (e.g., narrative relationships and emotional intensity; Appendix J), making them harder to infer and map to catalog attributes. This challenge persists even with oracle preferences (Table 4). Nevertheless, human evaluation suggests the task remains solvable, and Yo'Rec substantially improves Movie R@1 from 1.62% to 7.50%. We have revised the paper to add this analysis.

AuthorsOpenReview ↗

Response to Reviewer fZAR (3/3)

(Part 3/3)

[Q1] Section 5.2 discusses "Shopping Companion (Tou et al., 2026)", but the cited reference is ShoppingComp (arXiv:2511.22978). Please correct the citation and clarify how this work differs from Shopping Companion in benchmark design and training methodology.

We sincerely thank you for pointing out this citation issue. Due to the highly similar titles, we apologize for this mistake and have corrected the citation. Our original discussion in Section 5.2 was indeed intended to refer to Shopping Companion. Below, we further clarify the differences between the two works.

Benchmark Design.

  1. Task Definition and Evaluation. Shopping Companion considers a shopping-assistant task. However, its final evaluation uses an LLM-judged Success Rate (Succ.), which assesses whether the recommended product(s) satisfy a set of user needs and recovered preference constraints. Importantly, exact matches to the reference products are not required, as any alternative satisfying these criteria is accepted. Thus, its evaluation focuses on preference- and constraint-satisfying shopping outcomes, rather than retrieving a specific target item. In contrast, LIP-Rec aims to retrieve a specific target item from a fixed candidate pool based on underspecified queries and long-term assistant interactions, enabling controlled target-item ranking evaluation with Recall@K and NDCG@K. This more directly reflects the core objective of recommendation: identifying which candidate item is the best match for the user.

  2. Persona Modeling. More fundamentally, PersonaRec explicitly models coherent personas, whereas Shopping Companion does not explicitly construct persistent persona identities with cross-target preference consistency. Instead, it generates target-product-conditioned preference dialogues and plants them into existing LongMemEval histories. In contrast, we construct the entire interaction history for each persona, never expose target items during generation, and reveal only a subset of their attributes through individual sessions. Preferences associated with six targets of the same persona remain mutually consistent across the history.

  3. Preference Dynamics and Distractors. PersonaRec systematically models explicit, implicit, and evolving preferences, together with diverse distractors, including domain-related interactions without preference expression and negative preferences. This enables more fine-grained investigation of preference recovery under diverse and potentially misleading signals (you also pointed out this).

  4. Modality. Shopping Companion builds on textual LongMemEval histories. In contrast, 57% of PersonaRec sessions are multimodal, including cases where preference values are conveyed only through images (Figure 3). This enables the evaluation of visually grounded preference inference, an increasingly important capability for personal MLLM assistants in this era.

Training Methodology.

Shopping Companion trains an interactive tool-using agent with SFT and RL, employing outcome-level and annotation-free tool-wise rewards to supervise memory retrieval and product search. In contrast, Yo'Rec does not train an interactive tool-using policy; it follows a fixed three-step PRPL-CoT pipeline: session localization, preference extraction, and query rewriting. Leveraging PersonaRec's fine-grained provenance, PRC-PO employs F1-based rewards for session localization and preference recovery, and NDCG@K for recommendation-query effectiveness (Section 3.2 and Appendix F). Finally, we emphasize that Yo'Rec is presented as a simple baseline; our primary contributions are the new LIP-Rec task and its dedicated benchmark PersonaRec.

If our responses, additional experiments, and revisions have addressed your remaining concerns, we would be sincerely grateful if you would kindly consider revisiting your assessment, especially in light of your encouraging recognition of our work. We would also be delighted to further discuss any questions about our work.

Reviewer_ZfxTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces LIP-Rec: an assistant receives an underspecified request ("recommend me a jacket") and must recover attribute preferences scattered across a long, multimodal chat history. PersonaRec, the accompanying benchmark, pairs 200 personas with ~83.5K-token synthetic histories and 30K real Amazon/MovieLens items, and each target is uniquely identified by its revealed attributes. Frontier MLLMs perform poorly (best U-Query R@1 ≈ 7%, Table 2), and the diagnostics separate failures in locating evidence from failures in writing the query (Fig. 6, Table 4). Yo'Rec trains Qwen3.5-9B with a locate–extract–rewrite CoT, first with SFT and then with GRPO-style RL, reaching 13.5 R@1.

Critical strengths

S1. Traceable benchmark design. Each target attribute is linked to its revealing sessions, which allows stage-wise diagnosis that real logs cannot support: models lose 22–41% of preferences before evidence is even located (Fig. 6).

S2. Careful anti-leakage choices. Generators never see the target, and distractors include dislikes. Training uses disjoint personas and holds out the evaluation domain (App. F), which makes the gains more credible.

S3. Clear writing and easy to understand.

Critical weaknesses

W1. This work is very similar to VisualLens [NeurIPS 2025] , but there is no discussion about the work or experimental results.

Additionally, LoCoMo (ACL 2024), LaMP (ACL 2024), He et al. (CIKM 2023), and HyDE (ACL 2023) are not discussed.

W2. Construction of the benchmark is mostly synthetic, making the benchmark and evaluation much less convincing. What if the problem is just made up with the benchmarks?

W3. Novelty is limited, since it's mainly about testing basic SFT and RL on the benchmark. This is especially so given W1 and W2.

W4. Yo'Rec's gains may partly reflect catalog-aligned formatting. Only Yo'Rec's prompts request catalog attribute names and the catalog's sentence template (Prompts 8–9), while baselines write free-form profiles (Prompt 11). SFT on ~40 instances already gives most of the gain (Table 3). Running the training-free Prompts 7–10 for baselines would clarify the source.

W5. It is unclear how much scores reflect preference understanding versus generation artifacts. "Implicit" sessions must still name the attribute value (Fig. 13) and reuse stock phrases. The authors' own audit reports 78.75% preference correctness, with no inter-annotator agreement (Table 5). A shortcut probe and external annotators would strengthen the benchmark.

Questions for authors

Address concerns in weakness session.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZfxT (1/4)

We sincerely thank you for recognizing the traceable benchmark design, careful anti-leakage choices, and clarity of our work. We carefully address your remaining concerns below.

[W1] The discussion of relevant work VisualLens (NeurIPS 2025), LoCoMo (ACL 2024), LaMP (ACL 2024), He et al. (CIKM 2023), and HyDE (ACL 2023).

We sincerely thank you for suggesting these relevant works. These works study fundamentally different settings: photo-history-based local-business recommendation (VisualLens), long-term conversational memory (LoCoMo), personalized NLP tasks (LaMP), recommendation-oriented dialogues (He et al.), and generic zero-shot retrieval (HyDE) (even not in personalization scenarios). None addresses the research problem studied in LIP-Rec: retrieving personalized target items by recovering incidental preference signals from users' long-term everyday interactions with their personal MLLM assistants. We further clarify the distinctions below.

VisualLens (NeurIPS 2025). VisualLens leverages users' past personal photos to recommend local businesses, whereas LIP-Rec leverages long-term multimodal interactions between humans and their personal MLLM assistants, where item preferences are incidentally revealed. Moreover, VisualLens ranks 30–100 retrieved nearby businesses, while LIP-Rec retrieves a target item from a large candidate pool, more closely reflecting the conventional e-commerce recommendation setting.

LoCoMo (ACL 2024) and LaMP (ACL 2024). LoCoMo evaluates long-term conversational memory through QA, event summarization, and dialogue generation, while LaMP studies personalized NLP tasks such as text classification and generation. Neither addresses personalized item recommendation from long-term assistant interactions.

He et al. (CIKM 2023). He et al. study conversational recommendation, where users actively express their preferences through recommendation-oriented dialogues on recommendation platforms. As discussed in our original manuscript, the interaction setting is fundamentally different, leading to distinct interaction patterns. In contrast, LIP-Rec focuses on preferences incidentally revealed through everyday interactions with personal assistants, often long before a recommendation request is made, reflecting the emerging role of MLLM assistants as everyday companions.

HyDE (ACL 2023). HyDE proposes a generic zero-shot retrieval technique by generating hypothetical documents from queries, without any personalization.

To further address your concern, we have also incorporated appropriate citations and discussions of these works into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ZfxT (2/4)

(Part 2/4)

[W2] Concerns regarding the synthetic nature of PersonaRec and the real-world relevance of the LIP-Rec task it is designed to evaluate.

We sincerely thank you for your insightful question. We first emphasize that LIP-Rec addresses a realistic and practically important emerging scenario: as MLLM assistants become increasingly integrated into everyday life, users naturally reveal personal preferences through prolonged interactions and expect their assistants to leverage these signals for subsequent recommendations (The other two reviewers also highlighted this strength.). Indeed, industry efforts such as OpenAI's ChatGPT Shopping Research are already exploring personalized shopping based on users' past interactions. Yet, a clearly defined research task and a dedicated benchmark for systematically evaluating this capability remain lacking in academia, which motivates our work. Below, we further elaborate on the motivation and rationale behind using synthetic data in certain stages of our benchmark construction.

Practical Constraints. As discussed in our Limitations, directly collecting real users' long-term assistant interaction histories at scale is fundamentally impractical, as these records inevitably contain highly sensitive personal information, and collecting or publicly releasing them would pose severe privacy risks and potentially violate applicable data-protection requirements. Moreover, densely annotating sparse preference evidence and its evolution over months or years would incur prohibitive costs.

Real-world Grounding. To minimize the gap between our constructed benchmark and real-world scenarios, we ground multiple stages of PersonaRec in real-world data: (i) candidate items and target products are sampled from real recommendation datasets, including Amazon-Reviews-2023 and MovieLens-32M, with manually refined attributes; (ii) personas are derived from PersonaMem-v2/PersonaHub; and (iii) interaction scenarios are seeded from existing real-world human–LLM interaction datasets.

Human Validation. The LLM-generated scenarios and complete interaction histories are extensively reviewed and refined by human volunteers to ensure naturalness, plausibility, and consistency. Furthermore, our independent human post-evaluation (Appendix D) reports 90% preference naturalness, 78.75% preference correctness, and 85% target-item identifiability, providing further evidence of the benchmark's quality.

We sincerely appreciate your concern and the opportunity to clarify these efforts. Given the privacy concerns, we have made extensive efforts to build a realistic, human-validated, privacy-preserving, and controllable benchmark for systematically studying this practically important problem.

AuthorsOpenReview ↗

Response to Reviewer ZfxT (3/4)

(Part 3/4)

[W3] Novelty of Yo'Rec and the contribution of this work.

We sincerely thank you for raising this question. We would first like to clarify that the primary contributions of our work lie in introducing LIP-Rec as a new research task and constructing PersonaRec, a comprehensive and human-validated benchmark dedicated for it. Yo'Rec is deliberately positioned only as a simple baseline for this newly defined task (in our manuscript).

Nevertheless, we believe Yo'Rec has its own methodological novelty, which stems from our understanding of the unique challenges of LIP-Rec. Specifically, we decompose the task into three essential steps: (i) locating query-relevant sessions from long-term interactions, (ii) extracting the corresponding item preferences, and (iii) rewriting them into retrieval-effective recommendation queries. This task-driven decomposition forms our PRPL-CoT reasoning framework, around which we further design SFT and PRC-PO. Thus, Yo'Rec provides a task-specific reasoning and optimization design tailored to LIP-Rec.

Finally, we reiterate that our primary goal is to timely establish this new and practically important research direction through a clearly defined task and a comprehensive and dedicated benchmark. We hope PersonaRec will encourage future research to develop stronger methods for LIP-Rec, ultimately advancing more capable personal MLLM assistants for real-world recommendation scenarios.

[W4] Yo'Rec's gains may partly reflect catalog-aligned formatting: only its prompts request catalog attribute names and sentence templates (Prompts 8–9), while baselines write free-form profiles (Prompt 11). Running the training-free Prompts 7–10 for baselines would clarify the source.

We sincerely thank you for this insightful question. First, catalog-aligned output in Prompts 8–9 is an intentional design of Yo'Rec, aiming to translate recovered preferences into retrieval-effective queries. Nevertheless, we argue that simply adopting such formatting is far from sufficient to solve LIP-Rec, as evidenced below.

(1) Existing Empirical Evidence. Appendix B reveals two intrinsic challenges: (i) preference localization and recovery remain difficult even for frontier MLLMs (Figure 6); (ii) even with oracle preferences, frontier MLLMs achieve limited recommendation performance (Table 4), demonstrating the difficulty of preference-to-query transformation beyond formatting.

(2) Additional Experiment. We introduce a new ablation, w/ Catalog Prompt, using the same Qwen3.5-9B backbone as Yo'Rec. Specifically, we retain the baseline's original Prompt 11 reasoning and only replace its final PROFILE instruction with the catalog-aligned attribute vocabulary and sentence template of Prompts 8–9, without applying Yo'Rec's PRPL-CoT pipeline, SFT, or RL. In addition, we include PRPL-CoT w/o SFT, as requested, which uses the same Prompts 7–10 as Yo'Rec but without any SFT or RL. (same variant reported in our response to Reviewer fZAR). The results show that catalog-aligned prompting alone brings only marginal improvements and remains substantially inferior to Yo'Rec. This demonstrates that LIP-Rec requires more than merely catalog-aligned formatting. PRPL-CoT w/o SFT also falls short of Yo'Rec, indicating that task-specific training is essential for reliably executing the proposed reasoning pipeline.

MethodR@1R@10N@10
Qwen3.5-9B (Backbone)4.0314.188.46
w/ Catalog Prompt4.4614.848.91
PRPL-CoT w/o SFT7.2017.1011.80
Yo'Rec13.5124.2818.79
AuthorsOpenReview ↗

Response to Reviewer ZfxT (4/4)

(Part 4/4)

[W5] It is unclear how much scores reflect preference understanding versus generation artifacts. "Implicit" sessions must still name the attribute value (Fig. 13) and reuse stock phrases. The audit reports 78.75% preference correctness with no inter-annotator agreement (Table 5). A shortcut probe and external annotators would help.

We sincerely thank you for this insightful concern. We clarify the following four aspects.

(1) Expression Diversity. First, our templates do not prescribe fixed linguistic expressions. Instead, they specify preference dynamics (Explicit, Implicit, and Evolving) and modality, while the concrete interaction content is determined by diverse scenarios. We collect seed scenarios from existing real-world human–LLM interaction benchmarks and adapt them to specific item attributes and values, resulting in 17,760 diverse scenarios. All scenarios and generated interaction histories are further reviewed and refined by human volunteers to ensure naturalness and diversity.

(2) Implicit Preference Definition. We intentionally require implicit sessions to mention the target attribute and value while leaving the preference conclusion unstated. This ensures that preference inference is grounded in actual interaction evidence, rather than relying on unsupported guesses about users' preferences, which is not an expected personalized behavior.

(3) Inter-Annotator Agreement. We appreciate your suggestion regarding annotation reliability. In addition to the strong Preference Correctness score (78.75%), we further report Krippendorff's alpha (α\alpha) of 0.81 across independent annotators, demonstrating consistent human judgments and providing further evidence of the reliability of our benchmark.

(4) Shortcut Probing Test. We further introduce a simple shortcut baseline that directly matches attribute-value mentions in the history to the target query and treats them as user preferences, without explicit preference reasoning. Its performance remains substantially below Yo'Rec. This is because, particularly for implicit sessions, merely mentioning an attribute value does not necessarily indicate a preference, as it may arise from ordinary daily conversations unrelated to personal preferences. Moreover, negative preferences also mention attribute values but express rejection rather than preference. We have explicitly accounted for both cases in PersonaRec's construction (Appendix A), by incorporating domain-related sessions without preference expression and sessions expressing dislikes. Consequently, simple matching fails to distinguish genuine preference signals from incidental mentions and negative feedback, highlighting the need for contextual preference understanding.

MethodR@1R@10N@10
Qwen3.5-9B (Backbone)4.0314.188.46
Shortcut Baseline2.289.855.53
Yo'Rec13.5124.2818.79

We hope our clarifications and additional evidence have further strengthened your confidence in our work, and would be sincerely grateful if you would consider reassessing your evaluation. We would also be delighted to discuss any remaining questions.

Reviewer_LRHHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LIP-Rec, a task focused on leveraging long-term daily multimodal interactions between users and their personal MLLM assistants for personalized recommendation. To support this, the authors construct PersonaRec, a synthetic benchmark containing 200 personas, over 30K refined candidate items across three domains, and 83.5K-length interaction histories per persona where item preferences are sparsely and naturally revealed. Benchmarking frontier MLLMs reveals that they struggle to recover scattered preferences and translate them into effective retrieval queries. Consequently, the authors propose Yo’Rec, a baseline method consisting of a Preference Location Chain-of-Thought SFT stage and a GRPO-based policy optimization stage. Experiments show that Yo’Rec improves open-source MLLMs on the proposed benchmark.

Critical strengths

Strengths:

1.The problem formulation is timely and conceptually interesting, shifting the focus of conversational recommendation from platform-specific chatbots to long-term, incidental preference extraction from personal MLLM assistant interactions.

2.The construction of the PersonaRec benchmark is highly systematic, incorporating diverse preference expression types (explicit, implicit, evolving), multimodal cues, and distraction sessions to simulate realistic long-context noise.

3.The diagnostic evaluation of frontier MLLMs is well-structured, clearly isolating the two primary failure modes: incomplete preference recovery from long histories and suboptimal transformation of recovered preferences into retrieval queries.

4.The proposed Yo’Rec baseline provides a logical and interpretable solution pipeline, combining structured chain-of-thought SFT with reinforcement learning to address the identified bottlenecks step-by-step.

Critical weaknesses

Weaknesses:

  1. The benchmark relies entirely on LLM-synthesized interaction histories generated via role-playing prompts. Despite human auditing, these dialogues inevitably exhibit structural artifacts and predictable linguistic patterns. A model might learn to exploit the generative template's stylistic signatures rather than genuinely performing long-term preference reasoning, severely threatening the ecological validity of the benchmark.

  2. The absolute performance of the proposed Yo’Rec remains remarkably low (e.g., an average R@1 of 13.51% and N@10 of 18.79% under U-Query). While the authors frame the benchmark as "challenging," such poor performance raises concerns about whether the task formulation is practically solvable or if the evaluation setup—specifically relying on an external frozen retriever—is fundamentally misaligned with the generative capabilities being optimized.

  3. The training data scale for Yo’Rec is critically insufficient. Fine-tuning and applying RL on merely 10 additional personas (and only using cross-domain data, meaning even fewer samples per model) is statistically fragile. This makes the reported improvements highly susceptible to overfitting the specific idiosyncrasies of those 10 synthetic personas rather than learning generalizable preference-recovery skills.

  4. The PRC-PO reward design conflates two distinct cognitive tasks by simply combining an F1-score for preference extraction and NDCG@K for final retrieval. Because the retrieval score depends heavily on the frozen external embedding model (Qwen3-VL-Embedding-2B), optimizing the LLM's generation to maximize NDCG forces it to adapt to the retriever's embedding space biases rather than purely improving its preference understanding, creating a problematic optimization bottleneck.

Questions for authors

Questions:

  1. Given that all interaction histories are synthetically generated by LLMs, did you conduct any analysis to ensure that Yo’Rec is not simply memorizing the specific prompting templates or linguistic artifacts used during the session construction phase?

  2. How much of Yo’Rec’s performance gain is attributable to adapting to the specific frozen retriever used during RL? Have you evaluated whether the PRC-PO trained model transfers to a completely different retrieval backbone without retraining?

3.Since the RL stage is trained on only 10 personas, what evidence do you have that the learned policy generalizes robustly, and how does performance variance look across different random seeds for this small training set?

5 comments
AuthorsOpenReview ↗

Response to Reviewer LRHH (1/3)

Thank you for your thoughtful assessment and constructive comments. We sincerely appreciate your (1) recognition of the timeliness and conceptually interesting of LIP-Rec, (2) the systematic construction of PersonaRec, (3) the diagnostic analysis of frontier MLLMs, and (4) the interpretable design of Yo'Rec. Below, we carefully address your remaining concerns.

[W1/Q1] (1) Whether LLM-synthesized interactions introduce template-specific linguistic patterns that may facilitate shortcut learning. (2) Whether Yo'Rec learns such generation patterns rather than generalizable preference-reasoning capabilities.

We sincerely thank you for raising this concern. We respectfully clarify two aspects.

(1) Interaction Diversity. The LLM's role in benchmark construction is not unconstrained creation, but controlled realization of diverse scenarios. Specifically, the diversity of interactions primarily comes from our 17,760 scenarios, seeded from real-world human–LLM interaction benchmarks and adapted to specific item attributes and values, rather than fixed linguistic templates. Our templates only specify the preference dynamics (explicit, implicit, or evolving) and modality, while the concrete interaction content and expression are determined by the diverse scenarios. Furthermore, both scenarios and complete interaction histories undergo extensive human review and refinement. Our independent human evaluation (Appendix D) further supports their quality, with 2.75/3 naturalness, 2.25/3 coherence, and 90% preference naturalness.

(2) Yo'Rec and Potential Style Memorization. More importantly, our training protocol explicitly prevents memorization of evaluation-specific interaction patterns through two safeguards (Appendix F): (i) the 10 training personas are strictly disjoint from the 200 evaluation personas; and (ii) we adopt leave-one-domain-out training, completely excluding the evaluation domain from both SFT and PRC-PO. Thus, Yo'Rec cannot simply memorize evaluation personas, their histories, or domain-specific interaction patterns, and must generalize its learned reasoning capabilities across unseen personas and domains.

AuthorsOpenReview ↗

Response to Reviewer LRHH (2/3)

(Part 2/3)

[W2/W4/Q2] (1) Whether LIP-Rec is practically solvable given Yo'Rec's relatively low absolute performance. (2) Whether using an external frozen retriever leads to a potential mismatch with the generative capabilities being optimized, and a bottleneck. (3) Whether Yo'Rec's gains generalize to different retrieval backbones or are specific to its training-time retriever.

We sincerely thank you for these important questions. In brief, LIP-Rec is practically solvable, the frozen retriever serves as a controlled and replaceable evaluation component rather than an intrinsic constraint of the task, and we further demonstrate that Yo'Rec generalizes beyond its training-time retriever.

(1) Task Solvability. Our human study (Appendix D, Table 6) demonstrates that humans can successfully solve LIP-Rec, achieving 68.75% R@1 in the 10-candidate selection setting even with long and noisy histories, although they spend substantial time. Furthermore, when provided with oracle preferences, frontier MLLMs achieve substantially higher recommendation performance (Appendix B, Table 4), confirming that the task is challenging but far from unsolvable.

(2) Evaluation with a Frozen Retriever. Our evaluation in this study mainly focuses on improving the MLLM's ability to understand long-term preferences and formulate personalized recommendation queries. We therefore keep the external retriever frozen and shared across methods to isolate the MLLM-side contributions under a controlled evaluation setting. As the first study to propose this new task and the dedicated benchmark, this is just an evaluation choice, not a constraint or bottleneck imposed by LIP-Rec: retriever optimization is complementary and remains an important direction for future work. Furthermore, we have also investigated alternative retrievers (Appendix G.1). Our experiments show that BM25 is less effective for LIP-Rec, as preference-enriched queries primarily convey semantic information rather than exact lexical matches. In contrast, the dense Qwen3-VL-Embedding-2B retriever provides greater potential for this task. While further retriever optimization is certainly possible, it also raises concerns about overfitting to specific catalogs or domains and potentially compromising generalization, which we leave for future work.

(3) Cross-Retriever Generalization. First, we clarify that Yo'Rec does not merely learn retriever-specific optimization. Both its SFT supervision and PRC-PO preference rewards explicitly target the intermediate capabilities of LIP-Rec, including item-preference-relevant-session localization and fine-grained item-preference recovery, rather than solely optimizing retrieval outcomes. Nevertheless, to further address your concern, we replace the original dense retriever with BM25 and directly evaluate the trained Yo'Rec without retraining. As shown below, Yo'Rec still achieves clear improvements over the backbone, demonstrating its generalization beyond the training-time retriever.

Method (w/ BM25)R@1R@10N@10
Qwen3.5-9B2.4210.835.98
Yo'Rec9.1719.8313.82
AuthorsOpenReview ↗

Response to Reviewer LRHH (3/3)

(Part 3/3)

[W3/Q3] (1) Whether training on only 10 additional personas provides sufficient supervision and enables generalization beyond their specific characteristics, rather than overfitting to these personas. (2) The sensitivity and performance variance of Yo'Rec across different random seeds.

We sincerely appreciate your concerns regarding our training data scale and model robustness. Despite using only 10 additional training personas (40 target-level instances per domain-specific model), Yo'Rec achieves clear improvements on 200 unseen evaluation personas and held-out domains, providing direct evidence of its generalization capability. We further report experiments across multiple random seeds to assess training stability. Below, we provide detailed analyses.

(1) Training Scale and Generalization. First, each of the 10 training personas is associated with six target items across three domains. Under our leave-one-domain-out protocol, each domain-specific model is trained on the other two domains, yielding 40 training instances (10×410 \times 4), rather than merely 10 samples. More importantly, these training personas are strictly disjoint from the 200 evaluation personas, and the evaluation domain is completely excluded from both SFT and RL. Thus, the improvements on unseen personas and held-out domains demonstrate that Yo'Rec learns transferable LIP-Rec capabilities rather than memorizing the specific characteristics of its training personas. This data-efficient design is also intentional: LIP-Rec builds upon the pretrained MLLM's existing language, multimodal understanding, and reasoning capabilities, and we aim to equip it with task-specific reasoning through minimal supervision, paving the way for future training-free solutions. In addition, large-scale collection of such long-term personal interaction data is inherently impractical in real-world settings due to privacy constraints. We therefore deliberately adopt a small training set to reflect this practical constraint and investigate how LIP-Rec can be effectively learned with limited supervision.

(2) Random-Seed Sensitivity. To further assess training robustness, we repeat Yo'Rec training with 3 different random seeds and report the mean and standard deviation of its performance. The results show that the gains are not attributable to a particular random initialization or training trajectory.

MethodR@1R@10N@10
Yo'Rec (Seed 1)13.5124.2818.79
Yo'Rec (Seed 2)13.1823.9518.50
Yo'Rec (Seed 3)13.6724.5218.96
Yo'Rec (Mean ±\pm Std)13.45±0.2513.45 \pm 0.2524.25±0.2924.25 \pm 0.2918.75±0.2318.75 \pm 0.23

We hope our clarifications and additional evidence have adequately addressed your concerns and further demonstrated the significance of our work. We would be truly grateful if you would consider these points in reassessing your evaluation, and would be delighted to continue discussing any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

LIP-Rec · Final Rebuttal Summary

Dear ICLR 2027 Conference Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank you all for the time, care, and thoughtful consideration devoted to evaluating our submission. As the rebuttal discussion draws to a close, we would like to summarize the reviewers' feedback, our responses, and the corresponding improvements to our work.

We are grateful that Reviewers fZAR and LRHH recognized the practical relevance of LIP-Rec, which explores how everyday interactions with personal MLLM assistants can inform personalized recommendations. Reviewer fZAR also highlighted the scale and diversity of PersonaRec, our newly constructed benchmark. Reviewer ZfxT appreciated its traceable preference evidence, careful safeguards against data leakage, and the clarity of our presentation. Reviewer LRHH further recognized the systematic benchmark design, diagnostic evaluation of frontier MLLMs, and interpretable design of Yo'Rec.

Throughout the rebuttal, we conducted additional experiments and provided further analyses to address the reviewers' concerns. We summarize our key responses and contributions below:

  1. Benchmark Realism and Reliability. We clarified how PersonaRec grounds its constructed interaction histories in real-world item catalogs, personas, and scenarios seeded from human–assistant interaction benchmarks. We additionally reported inter-annotator agreement (Krippendorff's α = 0.81) and introduced a shortcut baseline based on attribute-value matching, which achieved only 2.28% R@1, compared with 13.51% for Yo'Rec (Reviewer ZfxT, W2 and W5).

  2. Method Ablations and Design Validation. We conducted additional controlled ablations, consistently applying self-reranking across variants, including training-free PRPL-CoT, query-rewriting-only SFT, and SFT without session localization. We further isolated the contribution of catalog-aligned formatting, showing that formatting alone brings only marginal gains (Reviewers fZAR, W1/Q2/Q3, and ZfxT, W4).

  3. Retriever Generalization and Training Robustness. We evaluated the trained Yo'Rec using BM25 without retraining, achieving 9.17% R@1 compared with 2.42% for the backbone. We also repeated training across three random seeds, obtaining an average R@1 of 13.45% with a standard deviation of only 0.25 percentage points (Reviewer LRHH, W2/W3/W4/Q2/Q3).

  4. Fine-Grained Benchmark Analysis. We reported preference recovery rates across explicit, implicit, and evolving preferences, as well as different evidence modalities. We also provided further analysis explaining the particular challenges of the Movie domain (Reviewer fZAR, W2).

  5. Related Work and Manuscript Improvements. We corrected the Shopping Companion citation and expanded the related-work discussion in Section 5.2. We also added the Movie-domain analysis in Section 4.2 and clarified the reranking protocol in Table 3. All additional experiments and analyses will be incorporated into the revised manuscript.

We are particularly grateful that, following the discussion, Reviewer LRHH kindly raised their rating from Weak Reject to Weak Accept. Reviewer fZAR maintained their Weak Accept and reaffirmed their positive assessment of our work, noting that our additional experiments and clarifications addressed most of their concerns and describing the work as "a useful contribution to the community." We sincerely appreciate their thoughtful consideration and encouraging feedback.

As committed, we will release the full PersonaRec benchmark to the research community to facilitate further research on LIP-Rec.

Once again, we sincerely thank the Program Chairs, Senior Area Chairs, Area Chairs, and all reviewers for your time, constructive feedback, and thoughtful engagement throughout the discussion. Your comments have helped us substantially strengthen both the empirical evidence and presentation of our work. We truly appreciate the opportunity to clarify our contributions and improve the paper.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.