acceptodds
Withdrawn from ICLR 2027

PrefMoE: Mitigating Preference Collapse in Personalized Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized multimodal large language models (MLLMs) aim to generate user-specific responses, but existing methods mainly rely on profile-level information and overlook diverse user preferences. We identify group preference collapse, where multi-user personalized MLLMs become insensitive to individual preferences and drift toward dominant population-level choices due to suppressed preference signals and unreliable preference use during generation. We propose PrefMoE, a preference-centric framework that separates stable profile information from preference-related representations. PrefMoE decomposes preferences into shared prototypes and personalized residuals, preserves individualized residuals with imbalance-aware learning, counterfactual pseudo-user augmentation, and residual decorrelation, and routes profile and preference factors through separate LoRA adaptation paths. Experiments across multiple MLLM backbones show that PrefMoE improves preference-sensitive personalization while substantially reducing preference collapse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Multi-User Personalization, User Preference Modeling, Preference Learning, Group Preference Collapse, Mixture of Experts, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F8pSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at personalization in multimodal LLMs at the level of user preferences rather than user identity. The authors point out that when a model is trained on many users together, it tends to give the popular answer instead of the one the specific user prefers, and they call this group preference collapse. To deal with it, PrefMoE keeps a learned state for each user, splits it into profile and preference parts, and models each preference as a shared prototype plus a personal residual. On top of that there are a few extra losses (an imbalance-aware contrastive loss, pseudo-user augmentation, decorrelation) and a MoE-LoRA router. Experiments are on a cleaned version of MMPB and cover several backbones of different sizes.

Critical strengths
  • I think the problem is worth studying. Pointing out that personalized models can drift toward the majority answer, and that overall accuracy can hide this, is a useful observation.
  • The improvements over full fine-tuning are consistent across quite a few backbones and model sizes in Table 1, so they don't seem to be specific to one model.
  • There's a lot of analysis beyond the main table (routing variants, removing facet labels, number of pseudo users, an extra domain, sequential updates).
  • The paper is fairly honest about its limitations, e.g. it reports that PrefMoE degrades more than a baseline under sequential updates.
Critical weaknesses
  1. My main concern is that the ablation only partly supports the story of the paper. The introduction argues that existing methods focus on who the user is and not what they prefer. But in Table 2, the single largest improvement, on both accuracy and collapse, comes from the step labeled profile factor learning. As far as I can tell this step mainly adds the profile encoders and losses, although it may also introduce the factorized user state; the paper should clarify what exactly it adds. Of the preference-specific components, pseudo-user augmentation and decorrelation do reduce collapse noticeably, but the imbalance-aware residual term barely changes it, and their effect on accuracy is small compared with the profile step. So a good part of the gain seems to come from better modeling of the user's identity, which doesn't quite match the motivation. The ablation is also only cumulative; leave-one-out results would make the contribution of each part much clearer.
  2. I'm not convinced the paper has identified where collapse comes from. It frames collapse as arising under joint multi-user learning, and the method mainly targets preference signals being suppressed during representation learning. But the models without any fine-tuning already collapse a lot. In Figure 1, fine-tuning actually seems to reduce collapse for every group, and Table 1 shows the same trend for almost all backbones. That suggests the base model's prior (or just a common default answer) may be a big part of the problem. The paper's second explanation, that the model may follow generic or population-dominant patterns during generation, partly covers this, but the two sources are never separated experimentally. There's also a confound between popularity and semantic similarity: Pilates is the least popular option but also the one closest to Yoga, and it's generally the most collapsed. It would help to compare base and fine-tuned models under the same prompt, or to change which preference is the majority and see whether the collapse direction follows.
  3. The comparison with baselines mixes different things. The test users are the same as the training users, PrefMoE uses a learned per-user state at inference, and the fine-tuning baselines only see the preferences as text in the prompt. It's also not clear how the PEFT baselines get user information. To be fair, the basic user-embedding row in Table 2 suggests an ID embedding alone isn't enough. Still, I'd like to see a simple LoRA + user-ID embedding baseline and some evaluation on held-out users.
  4. The collapse metric could use more support. The number of negative samples it's computed on isn't reported, and from the table values it looks fairly small. Since it only counts false "yes" answers on negatives, it could on its own be lowered by a more conservative model. The accuracy gains make this less likely to be the whole story, but reporting overall yes-rates or recall would make it clear. All results also seem to be from a single run without variance.
  5. Some closely related work isn't discussed, for example shared + individual user representations in personalized alignment (P-RLHF, PAL), MoE-LoRA user routing (iLoRA), and popularity bias in recommender systems. The term "preference collapse" has also been used before in the RLHF literature. This makes it hard to judge what is actually new here.
  6. The benchmark is small (50 users) and the preference labels were reconstructed by the authors, so it's hard to know how well the results generalize. The authors do acknowledge this.
  7. Minor:
    • Several method details are missing from the paper: the profile encoders and losses, how the experts are conditioned on the user state, how pseudo users get residuals, and key hyperparameters such as the temperature and loss weights. These may be in the supplementary code, but they should be in the paper.
    • Figure 1 is a bit confusing. The options aren't really comparable (Yoga and Pilates overlap, and essential oils is a product rather than an activity), and the Yoga bar, which counts correct predictions, sits next to the collapse bars, so it's easy to misread.
Questions for authors
  1. What exactly does the profile factor learning step in Table 2 add on top of the basic user embedding? Does it also introduce the separate preference factors?
  2. Do you have results on held-out users, and a LoRA + user-ID embedding baseline with a similar number of parameters?
  3. How many negative samples is the collapse rate computed on, and what are the overall yes-rates or false-acceptance rates for each method? Any variance across seeds?
  4. How are the PEFT baselines conditioned on the user, and how were they tuned?
  5. How was full fine-tuning done for the 72B models with the reported hardware? And why does fine-tuned Qwen2.5-VL-7B do worse than the untrained one on preference accuracy in the 0-turn setting?
Reviewer_KHuGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the problem of preference collapse in personalized multimodal large language models (MLLMs). The authors argue that existing models often rely on common population-level preferences rather than individual user preferences, especially when multiple users are trained jointly. To address this issue, the paper proposes PrefMoE, which separates user profiles from preferences and further decomposes preferences into shared prototypes and user-specific residuals. The framework combines imbalance-aware contrastive learning, pseudo-user augmentation, residual decorrelation, and hierarchical mixture-of-experts routing with LoRA adapters to improve personalized response generation. The main claimed contribution is a preference-centric personalization framework that preserves individual preference differences and reduces predictions toward population-dominant preferences.

Critical strengths

I appreciate the paper's focus on the distinction between individual preferences and population-level preferences. This is important for personalized AI systems. The finding that MLLMs may produce popular answers even when user-specific preference information is provided is interesting, and the authors attempt to quantify this phenomenon.

Critical weaknesses
  1. Limited evidence of genuine preference learning or collapse.

My main concern is the validity of the evaluation. The benchmark includes only 50 users with predefined preference annotations, and training and evaluation involve the same users. PrefMoE retrieves learned representations by user ID, so the improvements may largely reflect memorizing and retrieving known preferences rather than learning them from actual behavior. The additional evaluation is also small and constructed, without testing unseen users. Moreover, the collapse metric can mechanically improve when overall false-positive errors decline. The paper does not sufficiently distinguish reduced preference collapse from improved prediction accuracy.

  1. Limited novelty and insufficient comparisons with simpler methods.

Separating population preferences from individual deviations has been extensively studied in recommendation and personalized learning. The proposed framework combines existing techniques, including contrastive learning, reweighting, and MoE routing, but offers limited new methodological insights. Furthermore, synthetic pseudo-users may not represent realistic preferences, and performance deteriorates under longer dialogues and sequential updates.

Overall, I find the contribution and evidence for generalizable preference learning insufficient.

Questions for authors
  1. Preference learning versus memorization. Since the training and evaluation sets involve the same 50 users, how can the authors establish that PrefMoE learns generalizable individual preferences rather than memorizing the existing preference annotations? Can the method generalize to entirely new users whose preference representations were not optimized during training?

  2. Validity of the collapse metric. How much of the reduction in preference collapse comes from improvements in overall prediction accuracy or lower false-positive rates? Could the authors compare population-directed errors while controlling for general error rates and the popularity distribution of the alternatives?

  3. Incremental methodological contribution. Can the authors compare PrefMoE with simpler approaches, such as user-specific preference embeddings, explicit memory retrieval, or shared-plus-individual latent factor models combined with standard LoRA? Given the substantial improvement from profile factor learning alone, what is the additional benefit of the proposed preference residuals and hierarchical MoE architecture?

  4. Generalization to realistic preferences. How well would PrefMoE perform when preferences must be inferred from actual user interactions rather than explicitly annotated descriptions? Also, what evidence supports the assumption that randomly recombined pseudo-user preferences represent meaningful counterfactual preference profiles?

Reviewer_k1T7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores personalization of multimodal LLMs, motivated by a failure mode introduced in the paper called group preference collapse. This is a failure mode that arises when a single model is trained with personalization context across multiple users and then tends to collapse its predictions to popular preferences, sometimes even preferring these aggregate preferences over explicit preferences encoded in the context during inference time. To address this issue, the paper proposes a few key architecture and training changes on top of a frozen MLLM backbone, called PrefMoE. The first is a user-specific learned state that is a function of a user image profile vector, text profile vector, and preference vectors across 5 facets: entertainment, fashion, lifestyle, shopping and travel. The user-specific facet vectors are a combination of a shared prototype plus a personal residual, which is trained with additional losses (a frequency-weighted contrastive loss, synthetic pseudo-users crafted by recombining preferences from real users, and a decorrelation penalty) to keep rare preferences from being absorbed by popular ones. The second is a two-level router that selects facet- and profile-specific LoRA adapters on the frozen MLLM to apply for each image and question. At inference time, PrefMoE uses the user identifier and retrieves the learned user state. The method is trained and evaluated on the same set of 50 users on MMPB-Clean across 9 different frozen MLLM backbones, and additionally on two constructed datasets using images from Food-101 and Fashion Product Images and 20 users. Across all backbones, they show an improvement in preference accuracy and a reduction in preference collapse compared to full fine-tuning. The primary finding of the paper is that explicitly separating per-user preference representations reduces the tendency to default to popular preferences.

Critical strengths

The paper identifies a critical personalization failure mode for MLLMs, illustrates it clearly, and evaluates its proposed solution across many MLLM backbones. Key strengths include:

  1. The problem is well-motivated and relevant. Population-level preferences being emphasized over individual preferences is a clear challenge and examples of this are provided in the paper.
  2. PrefMoE shows improvements in preference accuracy and collapse over no training and full fine-tuning across all the MLLM backbones (different sizes and model families) evaluated in the paper.
  3. Several ablations are conducted to understand the impact of each component in the proposed architecture, the number of pseudo-users used during training, facet labels, and adaptive versus fixed routing.
Critical weaknesses
  1. If I understand the setup correctly, training and evaluation use the same 50 users because PrefMoE must learn the state for each user during training and simply look up this state during inference. This brings up two concerns: 1) how does the technique scale to new users or shifts in user preference? (The sequential update experiments in Appendix H also show PrefMoE degrading more than TG-LLaVA when user preferences change.) and 2) fair comparison to the baselines. The baselines only see the user preferences as text/image context during both training and inference, while PrefMoE uses a learned per-user state retrieved by the user’s unique id. This may enable PrefMoE to simply memorize instead of reasoning about preferences. If the technique can scale to held-out users, it would be useful to include an evaluation of this.
  2. Some baselines are weak or not comparable, and some are missing. One concern is that the full fine-tuning performance is sometimes worse than no training. This may indicate that the training is not properly tuned. Also since PrefMoE freezes the backbone, perhaps a standard LoRA finetuned baseline with a similar number of trainable parameters should also be reported as that may be more comparable. Also in Table 1, it is unclear what backbones are used for LLaVA-NeXT, LOVA3 and TG-LLaVA. Table 10 suggests 7B versions, but it is not clear. This is needed to understand the comparisons being shown in the table better. The paper mentions other personalization approaches but does not evaluate these and it is unclear why not (e.g. MyVLM and PVIT).
  3. The tasks themselves that are evaluated on don’t seem to require significant or complex preference reasoning. From the examples provided, it seems like the preference lists are explicit and short and the queries are simple yes/no or multiple choice queries.
Questions for authors
  1. Clarification: For the food and fashion set, how were the 20 users and their profiles (images and descriptions) and preferences constructed, and was the model trained only on this set or adapted from the MMPB-Clean trained model?
  2. Clarification: Do the preference items and image groups in the evaluation split also appear in training for the same users? How does PrefMoE perform on users not seen during training?
  3. Clarification: How were the full fine-tuning baselines tuned, given that some perform worse than no training?
  4. Suggestion: Consider running multiple training runs and reporting mean and variance to reduce noise in the metrics (especially since the evaluation sets are small).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.