acceptodds
Withdrawn from ICLR 2027

Does Behavioral Fidelity Predict Reliable Policy Selection? Decision-Anchored Validation of LLM User Simulators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are increasingly used to simulate user responses in recommender systems, supporting agent training and policy evaluation. Existing studies primarily assess the behavioral similarity between simulated and real users, yet it remains unclear whether such fidelity reliably reflects the quality of recommendation policy selection. To address this question, we characterize LLM-based user simulators from three complementary perspectives: distributional fidelity, conditional response fidelity, and treatment-response fidelity. We further construct simulator conditions with systematically different fidelity profiles through probability calibration, behavior cloning, treatment-aware training, and controlled information perturbations. Using randomized real-user logs, we estimate reference values for supported recommendation policies and directly compare simulator fidelity with policy ranking, pairwise decision correctness, and decision regret. Experiments and cross-domain stress tests show that making a simulator more behaviorally similar to real users does not necessarily make it more reliable for comparing or selecting recommendation policies. The relevant code can be obtained at https://anonymous.4open.science/r/SimFidelity-523F.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Recommender Systems, Recommendation Policies, Fidelity, Information Perturbation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ihaNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper mainly checks whether the behavioral fidelity of LLM user simulators predicts how reliably they select recommendation policies. It proposes a multi-view fidelity measurement: calibration, conditional-response prediction and treatment-response matching, then compares these views with policy ranking, pairwise decision correctness and regret to relate the behavioral fidelity to policy evaluation. For the experiments, KuaiRand is the primary dataset, with supplementary OBD and Amazon analyses. Controlled transformations and matched statistical baselines provide counterexamples to particular proxy-to-decision inferences, and the results cannot sufficiently prove the effectiveness of the proposed method.

Critical strengths
  1. The proposed research question matters, and the study is organized in a clear way. Consistent with prior work on simulator evaluation, the paper recognizes that fidelity of simulator and selecting the best policy are different tasks. It tests its hypothesis through a staged sequence of research questions: whether the fidelity views agree, whether fidelity improvements carry over to decisions and whether fidelity predicts decision errors. Evaluating regret and pairwise correctness against a randomized-log reference is more useful for the stated deployment purpose than reporting calibration or AUC alone.
Critical weaknesses
  1. Part of the claimed novelty/contribution is overstated, and some relevant prior work is neither cited nor compared against. This paper’s first contribution is to reframe simulator evaluation from behavioral similarity to decision reliability. However, this distinction has already been formalized before. For example, some prior work separates simulators for training, which aim at behavioral similarity, which should predict real system performance [1][2]. Recent work also validates LLM-based simulators by whether their system ratings agree with human judgments [3], and validates recommender simulations against real A/B test outcomes [4]. The author should cite more related works and clarify the contribution relative to them.
  2. The central experiment is uninformative and proves nothing. Mainly in RQ3, it never observes genuine simulator errors: every disagreement falls on a pair where the real-user reference itself is unstable. The authors acknowledge this in the Limitations, yet the abstract presents the null as a negative finding about fidelity. The abstract should state that the question remains unresolved.
  3. The design choices lack empirical justification, and there is little analysis of why the results come out as they do. The experiments do not directly show what the proposed framework contributes. There is no evidence that the F1-F3 decomposition is reasonable nor that each view captures something distinct and relevant to decisions.
  4. Beyond the controlled transformations, there is no ablation over the policy family, the simulator prompt, or the backbone in the main analysis. More experiments of this kind could reveal where the approach breaks down and would be a clear guidance for improvement. The paper would also benefit from qualitative analysis of the simulator outputs and of the disagreement cases, showing what the simulators actually get wrong.
  5. The design, experiments, and analysis appear ad hoc, and the paper reaches no solid conclusion or solution. The authors state that “the F1 definition and predictive analysis were revised after seeing results” around Lines 477, yet the paper repeatedly describes its analysis as pre-registered. This makes it hard to tell which conclusions are confirmatory. Beyond that, the conclusions are mostly null or non-identifiable results. The work leaves substantial room for improvement.

[1] Bernard, Nolwenn, and Krisztian Balog. "Towards a formal characterization of user simulation objectives in conversational information access." Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval. 2024.

[2] Balog, Krisztian, et al. "Theory and toolkits for user simulation in the era of generative AI: user modeling, synthetic data generation, and system evaluation." Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2025.

[3] Dou, Yao, et al. "SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[4] Bougie, Nicolas, et al. "Beyond offline a/b testing: Context-aware agent simulation for recommender system evaluation." arXiv preprint arXiv:2604.09549 (2026).

Questions for authors
  1. What is the conceptual contribution beyond previous works, as stated in Weakness 1?
  2. What is the reason for the negative experimental result? what do the authors believe is the reason this study fails to identify a fidelity view that certifies policy selection, and which component of the proposed framework is responsible?
  3. The logistic and LightGBM baselines reach PDA 0.017–0.051 and Kendall τ ≈ −0.6. This is a near-complete reversal of the reference ordering. What is the mechanism behinds it?
Reviewer_YK4XOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether behavioral fidelity of LLM-based user simulators predicts the quality of recommendation policy selection. The authors define three fidelity views: F1 probability calibration, F2 conditional response prediction, and F3 treatment response. They construct simulator conditions with systematically different fidelity profiles via probability calibration, behavioral cloning, treatment-aware training, and controlled information perturbations. Using randomized real-user logs to estimate reference policy values, they compare simulator fidelity against policy ranking, pairwise decision correctness, and regret. The main finding is that better behavioral fidelity does not consistently translate into better policy selection in the evaluated settings, and the RQ3 predictive analyses do not establish stable incremental predictive validity for pairwise decision correctness.

Critical strengths

The paper addresses an important practical question: whether behavioral fidelity is a useful indicator of a user simulator’s reliability for recommendation policy selection.

Critical weaknesses
  1. The instability of the reference rankings weakens the conclusions of RQ3. All observed disagreements are confined to the 28 policy pairs with unstable reference rankings, while the 31 stable pairs exhibit no disagreements, leaving AUROC undefined for the stable subset. This makes it difficult to determine whether the disagreements reflect simulator judgments or uncertainty in the reference ranking.
  2. Although the authors acknowledge the different sampling and weighting schemes used to estimate real-user and simulator policy values, they do not quantify how these differences affect the results. It therefore remains unclear how much of the observed discrepancy is attributable to simulator predictions and how much to the evaluation setup.
Questions for authors
  1. In RQ3, given that all observed disagreements occur on policy pairs with unstable reference rankings, how can unreliable simulator judgments be distinguished from uncertainty in the reference rankings?
  2. How sensitive are the reported policy-value comparisons to the different sampling and weighting schemes used for real-user and simulator policy values?
Reviewer_QUiVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In modern recommender systems and agentic AI, researchers increasingly replace expensive human trials with LLM-based user simulators to compare, rank, and select candidate recommendation algorithms. The implicit assumption across almost all existing literature is: Higher Behavioral Fidelity (looks like real users) implied Better Policy Selection (ranks algorithms correctly). The paper demonstrates that: Fidelity is multi-dimensional and contradictory; Improving fidelity can hurt decision outcomes; Fidelity metrics do not predict decision validity; Decisional Anchoring Contract. While the conceptual motivation and candor of the paper are strong, a critical audit reveals several methodological shortcomings, evaluation vulnerabilities, and reporting concessions

Critical strengths

As LLM user simulation surges in recommender systems and agent benchmarks, scrutinizing the foundational assumption that "behavioral fidelity implies reliable policy selection" is exceptionally important.

Demonstrating "responsiveness collapse" in behaviorally cloned LLMs (losing 99% of within-user variance) and the breakdown of classical ML baselines on policy ranking despite superior fidelity scores provides genuine technical value.

The authors are honest about non-preregistered changes, data overlap artifacts, and null statistical results, which is refreshing for an empirical conference submission.

Critical weaknesses

Evaluation Setup is Heavily Simplified Relative to Real-World LLM Simulators. The LLM simulators in this study do not act as conversational, interactive, or multi-turn agentic simulators. They are reduced to evaluating soft log-probabilities on binary tokens ("0" / "1") for a single item exposure in a fixed offline context.

Modern LLM simulators (e.g., Agent4Rec, RecUserSim, conversational recommendation) engage in dialogue, multi-step search, open-ended feedback, or profile reflection. By stripping the simulator down to a binary logit evaluator over tabular offline logs, the paper's critique might not fully capture the dynamics of true multi-turn agentic simulators.

The authors admit an experimental artifact; “an artifact audit found 6,787 of 18,939 early full-Probe records belong to Probe-A and 12,152 to Decision-B; this is not training leakage, but the intended row-level separation was not achieved”. Consequently, all full-Probe results in RQ1 and RQ2 are demoted to purely "descriptive," and RQ3 had to be patched with "Probe-A-only recomputation".

The authors explicitly state that “The F1 definition and predictive analysis were revised after seeing results (not pre-registered)”. Specifically, they removed the slope term from F1 after seeing the data. While their transparency is commendable, modifying metric definitions and evaluation pipelines after inspecting results weakens the confirmatory rigor of the statistical claims.

Because disagreements occur only where the real-user reference itself is noisy and statistically uncertain, the paper cannot definitively prove whether the simulator made a decision mistake, or whether the SNIPS off-policy estimator on the logged data was simply too noisy to establish the true ground truth ordering. The authors concede this: “Disagreements therefore cannot be separated from real-user ranking uncertainty”.

In KuaiRand, candidate policies simply select a top-100 list from 7,583 candidate items and recommend uniformly at random. Real-world recommender systems employ rank-weighted, slate-based, or sequential exposure. Evaluating policies as flat uniform distributions over 100 items is an artificial simplification.

Only 15 out of 25 policies satisfied the basic support criteria. Most of these are basic heuristic variants (popularity blending, item-KNN, co-visitation), limiting the breadth of algorithmic paradigms tested.

Questions for authors

The simulators evaluated are essentially offline logit-scorers over single-token binary outputs ("0"/"1") given historical text context. How do the authors expect these conclusions to hold for interactive, multi-turn, conversational agent simulators where error compounding and dynamic feedback loops dominate?

The F1 composite and predictive regression setups were modified after inspecting results , and the resulting AUROC increments reverse sign under influence-point deletions (Section 4.4). Could the authors clarify the statistical robustness of these findings outside the KuaiRand environment?

The candidate policies are restricted to uniform random draws over a top-100 item pool. How sensitive are these findings to more realistic top-k ranked slate presentations (e.g., position-biased slates of size 5 or 10)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.