- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the concept of individual fidelity, distinguishing it from steering in the context of user behavior simulation. The authors argue that existing work mostly focuses on steering, which measures how much a simulator’s behavior changes in response to supplied user information. In contrast, individual fidelity measures the degree to which a simulator preserves behavioral differences specific to the intended user. To evaluate this, the authors propose Attribution Trials, a method inspired by randomized controlled trials, to test how much the model learns from the specific user’s information. The results show that current user simulators do not exhibit individual fidelity, and that prompting-based simulation is ineffective.
Critical strengths
This paper discusses an important problem and, through very concrete experiments, proves that prompting LLMs does not work.
Critical weaknesses
Novelty of the prompting finding. The observation that prompted LLMs make limited use of user profiles is not new, including on the same dataset. The OPeRA benchmark already found that adding personas to the prompts of off-the-shelf LLMs does not consistently improve exact next-action accuracy, with gains limited to coarser action-type and click-type metrics, and concluded that current models have limited ability to integrate persona information into step-level decisions.
Questions for authors
N/A
Thank you for the positive review and for pointing us to OPeRA [1]. Your one weakness credits us with OPeRA's finding, which is not ours; we sincerely hope you will re-evaluate our work in light of this difference.
Moreover, we have released the code of Attribution Trials at https://anonymous.4open.science/r/Attribution-Trials-ECC7/; it contains every experiment in the paper, including those added in this rebuttal, and the download steps for every dataset, and we ensure that all reported results are reproducible.
W1: the prompting finding is not new
Our finding is not OPeRA's. OPeRA compares a persona with no persona, which is our gain over no information. We ask whether a profile that helps does so because it is this user's: on our 22 frozen-prompt combinations, the profile beats no profile in 14, and a matched imposter's profile does almost as well in all 14 (Section 5.1). A persona-versus-none comparison cannot see this: it keeps 100.6% of its score when the imposter's profile replaces the target's (Section 5.2). Nor is the finding specific to prompting: none of the 84 combinations passes, including trained per-user models and released simulators. Revision: Section 5.1 cites OPeRA's finding.
References
[1] Wang et al. OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation. ACL 2026.