- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper mainly checks whether the behavioral fidelity of LLM user simulators predicts how reliably they select recommendation policies. It proposes a multi-view fidelity measurement: calibration, conditional-response prediction and treatment-response matching, then compares these views with policy ranking, pairwise decision correctness and regret to relate the behavioral fidelity to policy evaluation. For the experiments, KuaiRand is the primary dataset, with supplementary OBD and Amazon analyses. Controlled transformations and matched statistical baselines provide counterexamples to particular proxy-to-decision inferences, and the results cannot sufficiently prove the effectiveness of the proposed method.
Critical strengths
- The proposed research question matters, and the study is organized in a clear way. Consistent with prior work on simulator evaluation, the paper recognizes that fidelity of simulator and selecting the best policy are different tasks. It tests its hypothesis through a staged sequence of research questions: whether the fidelity views agree, whether fidelity improvements carry over to decisions and whether fidelity predicts decision errors. Evaluating regret and pairwise correctness against a randomized-log reference is more useful for the stated deployment purpose than reporting calibration or AUC alone.
Critical weaknesses
- Part of the claimed novelty/contribution is overstated, and some relevant prior work is neither cited nor compared against. This paper’s first contribution is to reframe simulator evaluation from behavioral similarity to decision reliability. However, this distinction has already been formalized before. For example, some prior work separates simulators for training, which aim at behavioral similarity, which should predict real system performance [1][2]. Recent work also validates LLM-based simulators by whether their system ratings agree with human judgments [3], and validates recommender simulations against real A/B test outcomes [4]. The author should cite more related works and clarify the contribution relative to them.
- The central experiment is uninformative and proves nothing. Mainly in RQ3, it never observes genuine simulator errors: every disagreement falls on a pair where the real-user reference itself is unstable. The authors acknowledge this in the Limitations, yet the abstract presents the null as a negative finding about fidelity. The abstract should state that the question remains unresolved.
- The design choices lack empirical justification, and there is little analysis of why the results come out as they do. The experiments do not directly show what the proposed framework contributes. There is no evidence that the F1-F3 decomposition is reasonable nor that each view captures something distinct and relevant to decisions.
- Beyond the controlled transformations, there is no ablation over the policy family, the simulator prompt, or the backbone in the main analysis. More experiments of this kind could reveal where the approach breaks down and would be a clear guidance for improvement. The paper would also benefit from qualitative analysis of the simulator outputs and of the disagreement cases, showing what the simulators actually get wrong.
- The design, experiments, and analysis appear ad hoc, and the paper reaches no solid conclusion or solution. The authors state that “the F1 definition and predictive analysis were revised after seeing results” around Lines 477, yet the paper repeatedly describes its analysis as pre-registered. This makes it hard to tell which conclusions are confirmatory. Beyond that, the conclusions are mostly null or non-identifiable results. The work leaves substantial room for improvement.
[1] Bernard, Nolwenn, and Krisztian Balog. "Towards a formal characterization of user simulation objectives in conversational information access." Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval. 2024.
[2] Balog, Krisztian, et al. "Theory and toolkits for user simulation in the era of generative AI: user modeling, synthetic data generation, and system evaluation." Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2025.
[3] Dou, Yao, et al. "SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
[4] Bougie, Nicolas, et al. "Beyond offline a/b testing: Context-aware agent simulation for recommender system evaluation." arXiv preprint arXiv:2604.09549 (2026).
Questions for authors
- What is the conceptual contribution beyond previous works, as stated in Weakness 1?
- What is the reason for the negative experimental result? what do the authors believe is the reason this study fails to identify a fidelity view that certifies policy selection, and which component of the proposed framework is responsible?
- The logistic and LightGBM baselines reach PDA 0.017–0.051 and Kendall τ ≈ −0.6. This is a near-complete reversal of the reference ordering. What is the mechanism behinds it?