acceptodds
Under review as a conference paper at ICLR 2027

VIPSim: A Dual-Role User Simulation and Judgment Framework for Personalized VLM Alignment

Abstract

Personalized vision-language model (VLM) alignment has become an increasingly important as general-purpose VLMs still fall short of satisfying individual needs. However, the user perspective in this area remains predominantly overlooked. Existing approaches confine interactions and evaluation to limited, static, and single-turn settings. They fail to capture the complexity of real-world multi-turn dialogues, where users' needs vary across individuals and continuously evolve throughout conversations. Furthermore, user simulators remain largely unexplored for multi-modal personalized conversational settings. Prior work has shown that simply prompting assistant-aligned models to act as users often produces unsatisfactory and unrealistic behaviors. In this paper, we introduce VIPSim, a novel dual-role simulation and judgment framework designed to bridge the gap between static evaluation and real-world personalized VLM alignment. Conditioned on minimal personalized profiles and visual assets, our simulator dynamically synthesizes realistic, multi-turn interactions centered around personalized entities. It also serves as a judge to provide internal, multi-dimensional, session-level feedback on the assistant's performance in personalized scenarios. Experiments demonstrate that our fine-tuned VIPSim significantly outperforms both open-source baselines and proprietary models in simulating human-like interactions. Notably, our approach also substantially mitigates hallucination and sycophancy compared to its vanilla counterpart. By leveraging personalized user simulation, our method enables evaluation of personalized VLM alignment performance in realistic settings, bringing personalized VLM alignment one step closer to satisfying individual needs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
VLM, User Simulator, Personalized Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yRpNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a framework that enables the creation of user simulators based on minimal data. Besides the user simulator this work shows that by applying further optimization is possible to turn the user simulator into a judge model. The contribution of the paper is the method to produce personalized user simulators and turn them into judges achieving better results than the baselines and reducing hallucination and sycophancy.

Critical strengths
  • The motivation, problem formulation and simulation pipeline are easy to follow.
  • Despite the training data being synthetic the presence of humans reviewing the data used for training ensure an alignment step in the pipeline, as shown in Table 2.
  • The overall pipeline can be used to improve the evaluation of VLMs.
Critical weaknesses
  • The data construction isn't well motivated and could have more depth in the explanation.
  • During Section 4 or 5 there is no reference to the data used on the main experiences (Table 1 and Table 2) and there is a lack of clarity in metrics that use LLMs as a Judge.
  • Baselines used for evaluation are dated and the baseline for the method could be a more recent model too. Updating the backbone would show generalization of the method while evaluating against more recent baselines would show how it keeps up with more recent models.
Questions for authors
  • Why use a single 2B model as a captioner for the only two user simulator baselines? This should be ablated to ensure the bottleneck is not on the captioner.
  • How would the model compare with more recent open and close source baselines?
  • Can the training method generalize to other recent models such as Qwen3.5-4B or is it tied to the model?
  • Does the gains scale with bigger models such as Qwen3.5-9B?
  • Which LLMs where used for the Binoculars and Turing Test metrics?
  • What was the data split used for evaluation? Does it only contain samples generated by the pipeline or does it contain samples from other datasets?
  • Which LLM was used to generate the dataset?
Reviewer_wuBwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VIPSim, a dual-role framework for personalized VLM alignment and evaluation through multi-turn user simulation and session-level judgment. Using personalized profiles and visual assets from MMPB, the authors construct synthetic dialogues across three categories of personalized scenarios. They fine-tune Qwen3-VL-4B to simulate users capable of dynamically referencing images, responding to assistant outputs, and autonomously terminating interactions. The simulator is further adapted into a session-level judge through preference optimization, assessing assistant performance in terms of helpfulness, preference adherence, personalization, and naturalness.

Critical strengths

The paper explores a relevant problem of multi-turn user simulation for personalized VLM evaluation. The proposed framework integrates user simulation and session-level judgment in a unified pipeline. The paper includes comparisons with various baselines and additional analyses of simulation quality.

Critical weaknesses

Many baselines are outdated, such as LLaVA-1.5, Qwen2-VL, and GPT-4o. Comparisons with stronger recent open-source and proprietary VLMs are needed to better establish the competitiveness of VIPSim. The main experiments are conducted on an evaluation set constructed by the authors rather than a public benchmark, but important details about the evaluation process are missing, such as how task completion and termination are determined. I also checked the released evaluation code and found that several metrics do not seem to measure what they claim. Specifically, Termination Rate counts a conversation as successful whenever the simulator outputs <|endconversation|> after the first turn, even if the task has not been completed. Similarly, Task Completion Rate only checks whether any user message contains an <image_id:N> token, without verifying whether the actual task goal was achieved. This makes the reported metrics much less convincing. Moreover, the LLM-based scores in Table 1 are all given by GPT-4o, which is itself included as a baseline. More importantly, GPT-4o is only given the text conversation, without access to the actual images, and even the image-reference tokens are removed. This makes it difficult to see how the judge can reliably evaluate visual grounding in a multimodal setting. These issues raise serious concerns about the reliability of evaluation. The necessity of DPO is unclear. The SFT model already performs both user simulation and judgment, so the additional benefit of DPO is not well justified. As shown in Table 2, DPO mainly improves JSON output success rate, while judgment correlations show limited or even negative changes. Meanwhile, Table 1 shows a noticeable drop in simulation quality after DPO. These results make it unclear whether DPO is actually beneficial to the proposed framework. The training dialogues are generated by an LLM simulating both the user and assistant, rather than collected from real interactions. It's unclear whether learning from such pre-generated conversations can capture how real users react to different or unexpected assistant responses, especially given the paper's emphasis on realistic user simulation. Minor writing issues. There are several typos in both the main text and figures, such as “Perference” and “Ancher”.

Questions for authors

Could you clarify the training and evaluation splits? The released SFT data appears to contain the 320 evaluation dialogues and their ratings. Could you provide more details on the evaluation metrics in Table 1, particularly how token-based checks reflect actual task completion and appropriate termination? Is there any additional evidence that VIPSim captures real user behavior beyond the naturalness of its generated dialogues?

Reviewer_uoX1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces VIPSim, a dual-role framework that extends personalized VLM evaluation from static question answering to multi-turn, multimodal interaction conditioned on specific entities and their preferences. Built on Qwen3-VL-4B, the framework comprises two core roles:

  • VIPSim-Simulator (user simulator): Trained through user-side SFT on synthesized and human-refined dialogues, it learns progressive querying, selection and delivery of existing visual assets, and autonomous termination to generate interaction trajectories around personalized goals.
  • VIPSim-Judge (session-level judge): Built upon the simulator through preference optimization with an NLL regularizer, it produces scores and justifications for helpfulness, preference adherence, personalization, and naturalness, supporting assistant evaluation and downstream training-data selection.

The main novelty lies in the personalized multimodal user-simulation formulation and the integration of interaction generation with session-level judgment through these two roles; the contribution centers on task and framework design rather than a new architecture or optimization algorithm.

Critical strengths

Well-motivated task and useful interaction design

Single-turn personalized QA cannot adequately test progressively revealed needs, image references across turns, or corrective follow-ups. The paper operationalizes these capabilities in an executable interaction pipeline and supports both first-person and acquainted-third-party perspectives. This gives the task a clear contribution: evaluating interactive behavior that static questions do not readily capture.

Extensive experiments demonstrate strengths in simulation, judgment, and selected robustness metrics

Extensive experiments covering Simulator Performance, Judge Model Performance, Hallucination, and Sycophancy demonstrate the effectiveness of specialized training. In simulator evaluation, VIPSim achieves leading results against the evaluated mainstream open-source and proprietary VLMs on key metrics including termination, naturalness, personalization, and interaction quality. The judge substantially improves structured-rating success and agreement with reference ratings over the original backbone. On hallucination and sycophancy benchmarks, VIPSim also demonstrates advantages on MMHal and selected sycophancy subcategories. Together, these results support its value in interaction simulation, session-level judgment, and selected aspects of robustness, without implying uniform superiority across all metrics.

Critical weaknesses

None

Questions for authors

Q1. What can be learned from Gemini-2.5-Flash’s higher completion rate?

In Table 1, Gemini-2.5-Flash achieves 91.87% Task Completion, compared with 77.81% for VIPSim-Simulator. The Gemini-2.5-Flash’s high task completion rate need to be systematically analyzed, what skills can VIPSim learn from it to improve its own performance?

Q2. How sensitive is the judge baselines’ JSON-scoring success to prompt design?

The proprietary models in Table 2 have relatively low success rates for JSON-formatted scoring. Can better prompt design improve these rates? The analysis of Judge Model Performance is somewhat limited and would benefit from further explanation.

Additional Comments: On page 8, the discussion of Table 4 is placed between Figure 3 and Table 5, disrupting the reading flow. Please adjust the layout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.