- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the discrepency between how well LLMs predict what people say versus what people do. It introduced the concepts of fidelity asymmetry and target ambiguity as indicators of "behavioral underspecification." It then conducted hypothesis tests on seven general-purpose models and three fine-tuned models with a constructed dataset. The results reveal that behavioral underspecifications as measured by both fidelity asymmetry and target ambiguity are common, and fine-tuned human-behavior models do not consistently reduce this phenamenon.
Critical strengths
- The research question is important in this domain, it also has many implications such as how to fine-tune LLMs to better distinguish what people say vs what people do, and how do we better frame behavioral tasks to the models.
- The problem is well formulated, with formal definitions and notations clearly given.
- The application of behavioral science theory, the theory of planned behavior, to alleviating the fidelity asymmetry seems to be inspiring and promising.
Critical weaknesses
- Evaluation data: Though the dataset contains 98 matched say-do pairs across multiple domains and over 37,000 human responses as described, it seems to be limited in its form. If I understand correctly, the "do" data here are all binary voting records. As voting is highly contextualized, and follows a very specific process, it may hardly cover the vast behavioral space and could introduce bias using it represent human behavior, as this paper scoped. I think the authors could consider narrowing this paper's scope into human's voting behavior or including more diverse datasets.
- Evaluation metrics: While the formula supports arbitrary distributional distance metrics, the experiments used only the TVD. It might be helpful to try another distance metric as a robustness check.
- The discussion of fine-tuned models lacks formal evidence support. The comparison between fine-tuned and general-purpose LLMs are informative, but with the fine-tuning data and process varying a lot, it is difficult to conclude the dynamics of behavior under specification and the impact of fine-tuning. I think the paper could benefit from having at least one controlled experiment, where an LLM is particularly fine-tuned by what people say vs what people do in the dataset under a controlled process. With this controlled experiment, it would be clearer how fine-tuning influences.
- There are also models particularly tuned on what people do. The authors may consider including them into the comparison:
[1] Lu et al. "Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
[2] Xie et al. "Be. fm: Open foundation models for human behavior." arXiv preprint arXiv:2505.23058 (2025).
[3] Lei et al. "HumanLLM: Towards personalized understanding and simulation of human nature." Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1. 2026.
Questions for authors
- Since there are several categories of say-do pairs in the dataset, I'm curious how's the results differ among the categories? Is there a category where models tend to be more behaviorally underspecified?
- Page 3 remarks that this paper focuses on the behavioral underspecification at the population level. I wonder even though we don't consider the individual-level heterogenity at this stage, how will the individual-level results differ from what we have at the distributional level?
2 comments
We appreciate the reviewer's consideration of the additional results and explanations. It would be great if the reviewer could clarify what concerns still remain. To summarize, we:
• Clarified that “voting" is only one of the 98 say-do pairs we study
• How TVD is the most appropriate, and widely used metric for this task (equivalent to Wasserstein in our formulation)
• Identified ways in which fine-tuning can mitigate the problem
• Discussed how behavioral underspecification is robust to the type of say-do pair
• Highlighted that behavioral underspecification is even more prominent of models are evaluated at the individual response level
We are happy to address any questions that still remain unanswered.
Thanks again for taking the time to review our paper and go through our rebuttal.
I appreciate the authors' efforts in responding to my concerns and providing the additional results. However, I think my concerns on the evaluation data and metrics still remain. I will keep my score.