acceptodds
Under review as a conference paper at ICLR 2027

BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks

Abstract

Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics. While these models show promise in tasks such as survey response prediction and human-subject experiment simulation, there remains no systematic understanding of how well they perform across diverse behavioral science tasks. We introduce BehaviorBench, a comprehensive benchmark that evaluates foundation models along four core capabilities: (1) behavior prediction and simulation, (2) strategic decision-making, (3) subject-trait inference, and (4) behavioral knowledge application. Crucially, BehaviorBench evaluates model outputs at both the individual and distributional levels, capturing not only per-subject accuracy but also population-level alignment, an essential requirement for behavioral validity. Our evaluation shows that BehaviorBench remains challenging for leading general-purpose LLMs and behavior foundation models that are specifically trained with behavioral data. We find that individual-level and distributional performance do not always align. General-purpose LLMs tend to underestimate the diversity of human responses, whereas behavior foundation models often lag behind at individual-level prediction. Our investigation further demonstrates how fine-tuning on diverse behavioral data can improve both individual-level prediction and distributional alignment, balancing these two objectives. Our results highlight the importance of evaluation at both individual and distributional levels, establishing BehaviorBench as a foundation for developing and assessing behaviorally aligned AI systems.

Primary area:
datasets and benchmarks
Keywords:
large language models, foundation models, human behavior simulation, distributional alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uEP1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BehaviorBench, a benchmark for evaluating foundation models on behavioral-science tasks. The benchmark contains 12 tasks grouped into four capability categories: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application. A main feature is that models are evaluated at both the individual level and the distributional level.

The paper finds that these two evaluation levels can give quite different model rankings. General purposed LLMs often do well on individual prediction but underrepresent the diversity of human responses, while behavior-specialized models can better match population-level variation. The paper also fine-tunes Qwen3-4B and Llama-3.3-70B on a broad mixture of behavioral data and reports improvements at both levels, with larger gains on the distributional metrics.

Critical strengths
  1. The individual-level vs. distributional-level distinction is useful.

A model can make reasonably good point predictions while still collapsing toward an “average” person and missing population heterogeneity. BehaviorBench makes this failure mode visible, and the resulting differences between general-purpose and behavior-specialized models are interesting.

  1. The benchmark covers a fairly broad range of behavioral-science tasks.

The benchmark includes forward behavior prediction, strategic decision-making, subject-trait inference, and behavioral knowledge application. This gives a broader picture than benchmarks focused on only one behavioral task family.

Critical weaknesses
  1. Several tasks are standard structured or sequential prediction problems, but there are no task-specific non-foundation baselines.

Examples include predicting age from personality scores, predicting survey responses from demographics or previous responses, and predicting future actions from previous game rounds. These are presented to foundation models as prompts, but the underlying problems are still ordinary supervised prediction tasks.

Because of this, I found some of the low absolute scores hard to interpret. If a simple supervised model also performs poorly, then the task may simply contain little predictive signal. If a small task-specific model performs much better, then the result would instead suggest that the foundation models are failing to use information that is actually available.

I would like to see representative baselines such as mean/majority/last-action predictors, gradient-boosted trees, and neural models such as SOTA NLP predictors, GRU, or Transformer for sequential inputs.

  1. The generalization benefit of behavioral fine-tuning is not fully isolated in the current analysis.

The paper reports that behavioral fine-tuning improves both individual-level prediction and distributional alignment, and the fine-tuning setup is motivated as encouraging generalization across tasks and contexts. However, the post-training results are mostly reported in aggregate, making it difficult to assess how much of the improvement carries over to settings that are less directly represented in the fine-tuning mixture. The paper includes some useful held-out settings, but their contribution to the reported fine-tuning gains is not analyzed separately.

Questions for authors
  1. Could the authors add task-specific non-foundation baselines?

For a few representative tasks, I would like to see comparisons with task-specific predictors, including gradient-boosted trees, SOTA NLP predictors or GRU/Transformer. Personality-to-age prediction, sequential or masked survey prediction, and multi-round game prediction would already be useful examples. For distributional tasks, an empirical training-population distribution could also be a useful reference.

This would help clarify whether low performance comes from weak predictive information, from limitations of the foundation models, or both.

  1. Could the authors provide a more direct breakdown of the fine-tuning gains on held-out or less directly represented settings?

For example, reporting backbone-versus-fine-tuned results separately on a held-out game such as Push/Pull would help clarify how much of the improvement reflects transfer beyond the task settings represented during fine-tuning.

8 comments

We thank the reviewer for the questions and comments!

"I would like to see representative baselines such as mean/majority/last-action predictors, gradient-boosted trees, and neural models such as SOTA NLP predictors, GRU, or Transformer for sequential inputs."

“For distributional tasks, an empirical training-population distribution could also be a useful reference. This would help clarify whether low performance comes from weak predictive information, from limitations of the foundation models, or both.”

"Could the authors add task-specific non-foundation baselines?"

Thank you for suggesting these additional baselines. Following your advice, we have run these baselines on the tasks you mentioned:

  • Mean and majority: choosing the mean or the most common answer in the fine-tuning set.

  • Last action: directly answering using the subject's most recent answer or action in the input.

  • Training-population distribution: answering with one random draw from the training data, as a reference for the Wasserstein distance.

  • XGBoost (fine-tuned): gradient-boosted trees fine-tuned on features parsed from the input.

  • ModernBERT-large (fine-tuned): a recent encoder-only transformer model (395M parameters) with leading performance on classification and retrieval tasks (Warner et al., 2025). We fine-tune a separate ModernBERT-large model on each task, using only that task’s part of our fine-tuning dataset, as shown in Table 5 (Appendix B.3).

The results are shown below.

ModelPers. To Demo. (MAE ↓)Seq. Resp. Pred. (Acc. ↑)Masked Resp. Pred. (Acc. ↑)Multi-Round Pred. (MAE ↓)
Claude Opus 59.1 ± 0.340.6% ± 1.449.0% ± 1.617.0 ± 0.4
GPT 5.6 Sol11.5 ± 0.343.1% ± 1.548.0% ± 1.620.1 ± 0.5
Mean8.9 ± 0.228.7% ± 1.427.1% ± 1.421.7 ± 0.3
Majority9.8 ± 0.331.8% ± 1.432.7% ± 1.529.7 ± 0.6
Last action-26.7% ± 1.4-16.4 ± 0.5
XGBoost (fine-tuned)8.5 ± 0.242.0% ± 1.541.7% ± 1.513.5 ± 0.4
ModernBERT-large (fine-tuned)8.2 ± 0.345.7% ± 1.749.3% ± 2.013.5 ± 0.4

Table 1: Individual-level performance of the baselines and two general-purpose frontier LLMs. The best result in each column is in bold. ± is the standard error over 1,000 bootstrap resamples of subjects. A dash means the baseline does not apply.

At the individual level, the general-purpose frontier LLMs still lag task-specific models such as ModernBERT-large (fine-tuned). This shows that the input holds information that the frontier LLMs do not fully use, indicating a limitation of current foundation models. We also note that the gap between the leading frontier LLM and ModernBERT-large (fine-tuned) is small on some tasks (e.g., Masked Resp. Pred.).

ModelPers. To Demo. (W ↓)Seq. Resp. Pred. (W ↓)Masked Resp. Pred. (W ↓)Multi-Round Pred. (W ↓)
Claude Opus 53.70 ± 0.220.48 ± 0.030.34 ± 0.0211.4 ± 0.5
GPT 5.6 Sol7.19 ± 0.290.46 ± 0.030.40 ± 0.0313.9 ± 0.5
Training-population distribution0.83 ± 0.400.40 ± 0.040.42 ± 0.042.3 ± 0.6

Table 2: Distributional-level performance of the training-population distribution and two general-purpose frontier LLMs, in Wasserstein distance (W, lower is better). The best result in each column is in bold.

At the distributional level, the training-population distribution is a strong baseline. It has the lowest W on three of the four tasks. This suggests substantial room for improving foundation models.

We have added these results and discussions in Appendix J.

Warner et al., 2025: Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference, ACL.

"The generalization benefit of behavioral fine-tuning is not fully isolated in the current analysis."

“Could the authors provide a more direct breakdown of the fine-tuning gains on held-out or less directly represented settings?”

Here we report a more detailed comparison of the fine-tuned models and their backbone models on the held-out settings (discussed in Appendix B.3, lines 958-960). In the held-out Push/Pull game (which was unseen in training) we evaluate models’ performances on multi-round behavior prediction. This game contains 13,991 players and 52,786 data points from the MobLab dataset.

ModelRound 2Round 3Round 4Round 5Round 6Round 7Round 8+
Qwen3-4B0.580.610.620.590.610.610.55
Fine-tuned Qwen3-4B0.520.590.630.650.720.710.74
Llama-3.3-70B0.430.480.480.460.460.480.48
Fine-tuned Llama-3.3-70B0.600.640.670.670.740.740.77

Table 3: The performance on the held-out Push/Pull game, in macro F1 (higher is better). The best result in each round is in bold.

Both fine-tuned models show improved prediction performance as the round number increases, indicating that they leverage accumulated player behavior more effectively with more historical records. This experiment shows that fine-tuning on diverse behavioral data can generalize to a new game scenario.

We have added the per-task results of the fine-tuned models to Appendix D.4 (Table 6), and the held-out Push/Pull results by round to Appendix K.

Thank you for additional results.

  1. Given that smaller models outperform frontier LLMs in several tasks, and recent some works (e.g. Small Foundation Models of Human Cognition and Behaviour) shows that sub-1B models can match Centaur, what do the authors see as the unique advantage of LLMs for behavioral prediction?

  2. Prior technical reports (Centaur May Have Learned a Shortcut that Explains Away Psychological Tasks) suggest that such foundation models may exploit sequential shortcuts. Could the improvements across rounds in the Push/Pull experiment reflect round effects or simple action persistence rather than better behavioral generalization? What do the authors think? (first-round performance and control for round effects may be convincing evidences.)

Reviewer_1PyQOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark for benchmarking AI models (LLMs & behavioural foundation models) on two human behaviour levels: individual level and population level. The new benchmark, called BehaviourBench, comprises human behaviour data from several sources, notably from behaviour/economics games and some surveys. The paper shows how the new benchmark different from existing one, and how it covers cases that the other ones don't.

The paper also show performance of several LLMs & BFMs on the new benchmark, demonstrating that there's a still a large performance gap when predicting human behaviour, profile at both individual level and population level.

Critical strengths

This is a very well written paper with well-built benchmark and solid experiments.

Critical weaknesses
  • Although using data from games helps create a large scale benchmark, it's unclear whether this data can faithfully represent human behaviour in real life. To answer this, I would expect to find some benefit of finetuning models on game data when evaluated on real data.

  • Following the point above, the benchmark seems quite limited in the sense of domain diversity. Although there are several games in the benchmark, they are artificial (for instance, why should we care about the BeautiContest game -- line 226?). The benchmark covers data on domain of surveys. What could we get solid conclusions on human behaviour prediction (in general) when using this benchmark?

  • Table 2 nicely summarizes BehavorBench and existing benchmarks. Given that there are already several benchmarks in the literature, it's unclear why we don't just combine those into one big benchmark; why we need one more benchmark. Also, in section 2.4, the paper writes "[n]one of them evaluates the inference of subject traits xor the application of behavioral knowledge K", it's unclear whether it's straightforward to extend the existing benchmarks for that.


After rebuttal: the authors have addressed my concerns. I raised the score.

Questions for authors
  1. The authors use both "population level" and "distributional level", how are they connected?

  2. In table 4, why is only one BFM model (Centaur-70B) shown? how about Socrates?

4 comments

We thank the reviewer for the questions and comments!

"it's unclear whether this data can faithfully represent human behaviour in real life. To answer this, I would expect to find some benefit of finetuning models on game data when evaluated on real data."

We agree that economic games are controlled settings and may not fully capture every aspect of decision-making in real life. However, large-scale real-world decision data are limited in the public domain, often incomplete, and very noisy (Salganik, 2017, Section 2.3), which makes it difficult to benchmark models and gain clean scientific insights.

As a common practice in behavioral science literature, economic games provide a controlled environment for studying human decision-making (Falk and Heckman, 2009), and it has been validated that behavior in these games correlates with behavior in real life (Camerer, 2015).

All game data in BehaviorBench are real human decisions from MobLab play records (Mei et al., 2024), with real human subjects from more than 50 countries. After cleaning, the collection contains 379,754 behavioral observations from 76,035 unique subjects, as described in Appendix B (lines 887-888). We therefore argue that these game records provide a good proxy for human decision-making, alongside surveys and other behavioral data in BehaviorBench.

Salganik, 2017: Bit by Bit: Social Research in the Digital Age, Section 2.3, Princeton University Press.

Falk and Heckman, 2009: Lab experiments are a major source of knowledge in the social sciences, Science.

Camerer, 2015: The promise and success of lab-field generalizability in experimental economics: A critical reply to Levitt and List, Handbook of Experimental Economic Methodology, Oxford University Press.

Mei et al., 2024: A Turing test of whether AI chatbots are behaviorally similar to humans, PNAS.

"they are artificial (for instance, why should we care about the BeautiContest game -- line 226?). ... What could we get solid conclusions on human behaviour prediction (in general) when using this benchmark?"

The Beauty Contest game is a classical economic game that tests an important aspect of strategic reasoning in human behavior: level-k reasoning, that is, how many steps a player thinks ahead about what others will choose (Nagel, 1995). In BehaviorBench, a model plays the game against human play records. To win, the model needs to anticipate how human players reason. Therefore, a higher win rate in the Beauty Contest game suggests that the model better understands and predicts the level-k reasoning of human players.

Behavior in the Beauty Contest game also predicts behavior in other strategic interactions. For example, players whose guesses were closer to the winning number obtained better outcomes in a later negotiation (Hedberg, 2021).

Nagel, 1995: Unraveling in guessing games: An experimental study, American Economic Review.

Hedberg, 2021: One Step Ahead in the Game: Predicting Negotiation Outcomes with Guessing-Games Measures, Journal of Behavioral Decision Making.

"it's unclear why we don't just combine those into one big benchmark; why we need one more benchmark. ... it's unclear whether it's straightforward to extend the existing benchmarks for that."

BehaviorBench is not designed to simply add a few more datasets to existing behavioral prediction tasks. BehaviorBench differs significantly from existing benchmarks in two ways, as discussed in Section 2.4 (lines 300-303):

A much more comprehensive set of prediction targets under a unified framework, p(y | x, c; K). For example, BehaviorBench evaluates the prediction of strategic actions, the inference of subject traits (x), and the application of behavioral knowledge (K), which are fundamentally different from the tasks that the previous benchmarks cover (e.g., survey result prediction).

Evaluation at both individual and distributional levels. Previous works evaluate at only one of the two, and within a narrower set of scenarios.

Combining existing benchmarks is also difficult, because they cover very different domains and task formulations. Moreover, each of them evaluates at only one level, either individual or distributional, so a combined benchmark could not compare a model's performance at the two levels. BehaviorBench evaluates both levels under one framework, which shows that they do not always align, as discussed in Section 4.2.

"In table 4, why is only one BFM model (Centaur-70B) shown? how about Socrates?"

Table 4 lists only the models that rank among the top five on either the individual-level or distributional leaderboard, as described in Section 4.2 (lines 417-418). Centaur-70B is the only existing behavior foundation model (excluding our fine-tuned models) among them.

Here are the Table 4 statistics for all six behavior foundation models, with Gemini 3.1 Pro, which ranks 1st at the distributional level, for reference:

σmodel/σhuman\sigma_{\text{model}}/\sigma_{\text{human}}∣μmodel−μhuman∣σhuman\frac{\lvert\mu_{\text{model}}-\mu_{\text{human}}\rvert}{\sigma_{\text{human}}} ↓\downarrow
Gemini 3.1 Pro0.700.31
Centaur-70B0.930.38
Socrates-Qwen-14B-SFT0.830.43
Socrates-Llama3-8B-SFT0.970.63
OSim-8B0.550.48
Socrates-Qwen-14B-DPO0.410.45
Socrates-Llama3-8B-DPO0.390.50

Table 4: The diversity of human responses and the alignment with the human population mean for Gemini 3.1 Pro and all six behavior foundation models.

Centaur-70B and both Socrates SFT models better capture the diversity of human responses than Gemini 3.1 Pro, while OSim-8B and both Socrates DPO models do not. Gemini 3.1 Pro has the best alignment with the human population mean of these seven.

We have added the full results of Table 4 for all 20 models in Appendix L (Table 15).

"The authors use both "population level" and "distributional level", how are they connected?"

Thanks for pointing this out! In this manuscript, "population level" and "distributional level" both stand for modeling the human behavior distribution by considering the underlying population as a whole, in comparison to “individual level”. We evaluate models’ population/distributional level performance by comparing their predictions or simulations directly to the distribution of observed human behavior, using distributional distance metrics such as Wasserstein distance. In the revised draft, we have changed the two mentions of "population level" to "distributional level" in the abstract and Appendix D to ensure consistency.

Reviewer_1PyQOpenReview ↗

response

I would like to thank the authors for the response. I raised the score.

Thank you very much for your valuable feedback that helped us improve our paper!

Reviewer_rNd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents BehaviorBench, a benchmark designed to evaluate four types of LLM capabilities in behavioral science tasks: behavior prediction and simulation, strategic decision-making, subject-trait inference, and the application of behavioral knowledge to reasoning and problem-solving. In particular, the benchmark evaluates LLM performance at both the individual and distributional levels. The paper also fine-tunes two open-weight LLMs and shows that fine-tuning leads to substantially greater improvements at the distributional level than at the individual level.

Critical strengths

The paper provides a comprehensive, large-scale evaluation of LLMs across diverse behavioral science tasks. Specifically, it evaluates 20 LLMs using extensive data from behavioral experiments, personality surveys, scientific literature, and economics contests. The diversity of tasks and data sources strengthens the empirical evaluation and supports reliable comparisons across different LLMs.

Critical weaknesses
  • The paper is primarily an empirical study and makes limited methodological contributions. For example, although the experimental results show that the fine-tuned LLMs still struggle with predicting human behavior and inferring subject traits at the individual level, the paper does not develop a novel method to address these limitations or improve individual-level prediction and inference performance.
  • Some prior studies have already investigated and compared the capabilities of LLMs in behavioral science tasks. Although this paper extends these evaluations to a larger scale, covering more diverse tasks and data sources, the underlying experimental methodology is not substantially different from that of existing studies. Therefore, the novelty of the benchmark mainly lies in the breadth and scale of its evaluation rather than in a new evaluation methodology.
Questions for authors

(i) Could the authors further clarify the methodological contributions of this work, beyond integrating four categories of behavioral science tasks into a single benchmark?

(ii) For the strategic decision-making capability, the paper only evaluates LLMs using the Beauty Contest game. Are there any particular difficulties in incorporating other categories of games into the evaluation? Limiting the evaluation to a single game seems insufficient to support the claim that the benchmark comprehensively assesses the general strategic decision-making capabilities of LLMs.

Minor Suggestion: It would be helpful to include the related work section in the main text rather than placing it entirely in the appendix.

3 comments

We thank the reviewer for the questions!

"The paper is primarily an empirical study and makes limited methodological contributions."

"(i) Could the authors further clarify the methodological contributions of this work, beyond integrating four categories of behavioral science tasks into a single benchmark?"

Our contribution is the new benchmark, and we do not aim to propose a new method. Benchmark papers offer a different and sustainable type of contribution to the field alongside methodology papers. For example, HELM (Liang et al., 2023) and MMLU (Hendrycks et al., 2021) are two highly cited benchmark papers without proposing new methods.

A benchmark on behavioral science tasks is critical because it tests (and guides the advances of) model capacity in understanding and predicting human behaviors, beyond the commonly tested capabilities. In our paper, we find that frontier LLMs usually suffer from misaligned performance in individual-level vs distributional behavioral tasks, as discussed in Section 4.2. This suggests that there is considerable room to improve these methods on both levels at the same time.

Liang et al., 2023: Holistic Evaluation of Language Models, TMLR.

Hendrycks et al., 2021: Measuring Massive Multitask Language Understanding, ICLR.

".., the paper does not develop a novel method to address these limitations or improve individual-level prediction and inference performance."

In our paper, we have demonstrated the opportunity to address these limitations by curating a new fine-tuning dataset of human behavior data and using it to fine-tune Qwen3-4B and Llama-3.3-70B. This dataset contains 649,394 data points across 11 fine-tuning tasks from all three data sources. Both fine-tuned models improve over their backbone models by 4.0%–12.3% at the individual level and 46%–63% at the distributional level (Section 5.2). This preliminary result alludes potentially promising explorations in improving the frontier models, although it is not the main aim of this study.

"Some prior studies have already investigated and compared the capabilities of LLMs in behavioral science tasks."

Beyond integrating tasks, BehaviorBench differs from prior studies in two ways, as shown in Table 2 and Section 2.4.

First, the four capabilities give a more comprehensive evaluation than any single task. They all follow from one framework, p(y | x, c; K). Prior benchmarks evaluate only behavior y, and none of them evaluates the inference of subject traits x or the application of behavioral knowledge K.

Second, BehaviorBench evaluates models at both the individual and distributional levels, while prior works evaluate at only one. This matters because researchers care about both individual outcomes and the outcome distribution of the population. Individual-level and distributional performance also do not always align.

"(ii) For the strategic decision-making capability, the paper only evaluates LLMs using the Beauty Contest game. Are there any particular difficulties in incorporating other categories of games into the evaluation?"

Thanks for the good question! Although the Beauty Contest is currently the only game in strategic decision-making for which we have obtained the human play data, we do plan to expand this category with additional MobLab games about strategic interaction once a new data sharing agreement with MobLab is ready. Note that, most prior work about strategic interaction, such as Huang et al. (2025) and Duan et al. (2024), have LLMs play against other LLMs rather than human players, as discussed in Section 2.2.2. As the purpose of BehaviorBench is to evaluate capacity for human behavioral tasks, comparing with real human decision data is essential.

Huang et al., 2025: Competing Large Language Models in Multi-Agent Gaming Environments, ICLR.

Duan et al., 2024: GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations, NeurIPS.

"Minor Suggestion: It would be helpful to include the related work section in the main text rather than placing it entirely in the appendix."

We thank the reviewer for the suggestion! We would include the Related Work in the main text if the page limitation permits. Currently, the main text briefly compares BehaviorBench with existing benchmarks in Section 2.4 (lines 296-303).

I would like to thank the authors for their responses. I have carefully read their clarifications. I am afraid that I feel the responses do not substantially change my overall assessment of the work. I would prefer to maintain my original score.

I recognize the importance of developing benchmarks for the community, particularly for evaluating capabilities that are of broad interest and practical significance. However, I remain unconvinced that benchmarking human behavior prediction (e.g., in experimental economic games) would generate sufficiently broad interest within the foundation model development community. Furthermore, while fine-tuning LLMs on behavioral datasets can improve their prediction performance, this does not appear to be a novel method.

Thank you very much for your thoughtful comments, which helped us improve our paper! We truly appreciate the time you devoted to reading and responding to our rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.