- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a framework that enables the creation of user simulators based on minimal data. Besides the user simulator this work shows that by applying further optimization is possible to turn the user simulator into a judge model. The contribution of the paper is the method to produce personalized user simulators and turn them into judges achieving better results than the baselines and reducing hallucination and sycophancy.
Critical strengths
- The motivation, problem formulation and simulation pipeline are easy to follow.
- Despite the training data being synthetic the presence of humans reviewing the data used for training ensure an alignment step in the pipeline, as shown in Table 2.
- The overall pipeline can be used to improve the evaluation of VLMs.
Critical weaknesses
- The data construction isn't well motivated and could have more depth in the explanation.
- During Section 4 or 5 there is no reference to the data used on the main experiences (Table 1 and Table 2) and there is a lack of clarity in metrics that use LLMs as a Judge.
- Baselines used for evaluation are dated and the baseline for the method could be a more recent model too. Updating the backbone would show generalization of the method while evaluating against more recent baselines would show how it keeps up with more recent models.
Questions for authors
- Why use a single 2B model as a captioner for the only two user simulator baselines? This should be ablated to ensure the bottleneck is not on the captioner.
- How would the model compare with more recent open and close source baselines?
- Can the training method generalize to other recent models such as Qwen3.5-4B or is it tied to the model?
- Does the gains scale with bigger models such as Qwen3.5-9B?
- Which LLMs where used for the Binoculars and Turing Test metrics?
- What was the data split used for evaluation? Does it only contain samples generated by the pipeline or does it contain samples from other datasets?
- Which LLM was used to generate the dataset?