- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces BehaviorBench, a benchmark for evaluating foundation models on behavioral-science tasks. The benchmark contains 12 tasks grouped into four capability categories: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application. A main feature is that models are evaluated at both the individual level and the distributional level.
The paper finds that these two evaluation levels can give quite different model rankings. General purposed LLMs often do well on individual prediction but underrepresent the diversity of human responses, while behavior-specialized models can better match population-level variation. The paper also fine-tunes Qwen3-4B and Llama-3.3-70B on a broad mixture of behavioral data and reports improvements at both levels, with larger gains on the distributional metrics.
Critical strengths
- The individual-level vs. distributional-level distinction is useful.
A model can make reasonably good point predictions while still collapsing toward an “average” person and missing population heterogeneity. BehaviorBench makes this failure mode visible, and the resulting differences between general-purpose and behavior-specialized models are interesting.
- The benchmark covers a fairly broad range of behavioral-science tasks.
The benchmark includes forward behavior prediction, strategic decision-making, subject-trait inference, and behavioral knowledge application. This gives a broader picture than benchmarks focused on only one behavioral task family.
Critical weaknesses
- Several tasks are standard structured or sequential prediction problems, but there are no task-specific non-foundation baselines.
Examples include predicting age from personality scores, predicting survey responses from demographics or previous responses, and predicting future actions from previous game rounds. These are presented to foundation models as prompts, but the underlying problems are still ordinary supervised prediction tasks.
Because of this, I found some of the low absolute scores hard to interpret. If a simple supervised model also performs poorly, then the task may simply contain little predictive signal. If a small task-specific model performs much better, then the result would instead suggest that the foundation models are failing to use information that is actually available.
I would like to see representative baselines such as mean/majority/last-action predictors, gradient-boosted trees, and neural models such as SOTA NLP predictors, GRU, or Transformer for sequential inputs.
- The generalization benefit of behavioral fine-tuning is not fully isolated in the current analysis.
The paper reports that behavioral fine-tuning improves both individual-level prediction and distributional alignment, and the fine-tuning setup is motivated as encouraging generalization across tasks and contexts. However, the post-training results are mostly reported in aggregate, making it difficult to assess how much of the improvement carries over to settings that are less directly represented in the fine-tuning mixture. The paper includes some useful held-out settings, but their contribution to the reported fine-tuning gains is not analyzed separately.
Questions for authors
- Could the authors add task-specific non-foundation baselines?
For a few representative tasks, I would like to see comparisons with task-specific predictors, including gradient-boosted trees, SOTA NLP predictors or GRU/Transformer. Personality-to-age prediction, sequential or masked survey prediction, and multi-round game prediction would already be useful examples. For distributional tasks, an empirical training-population distribution could also be a useful reference.
This would help clarify whether low performance comes from weak predictive information, from limitations of the foundation models, or both.
- Could the authors provide a more direct breakdown of the fine-tuning gains on held-out or less directly represented settings?
For example, reporting backbone-versus-fine-tuned results separately on a held-out game such as Push/Pull would help clarify how much of the improvement reflects transfer beyond the task settings represented during fine-tuning.
8 comments
"The generalization benefit of behavioral fine-tuning is not fully isolated in the current analysis."
“Could the authors provide a more direct breakdown of the fine-tuning gains on held-out or less directly represented settings?”
Here we report a more detailed comparison of the fine-tuned models and their backbone models on the held-out settings (discussed in Appendix B.3, lines 958-960). In the held-out Push/Pull game (which was unseen in training) we evaluate models’ performances on multi-round behavior prediction. This game contains 13,991 players and 52,786 data points from the MobLab dataset.
| Model | Round 2 | Round 3 | Round 4 | Round 5 | Round 6 | Round 7 | Round 8+ |
|---|---|---|---|---|---|---|---|
| Qwen3-4B | 0.58 | 0.61 | 0.62 | 0.59 | 0.61 | 0.61 | 0.55 |
| Fine-tuned Qwen3-4B | 0.52 | 0.59 | 0.63 | 0.65 | 0.72 | 0.71 | 0.74 |
| Llama-3.3-70B | 0.43 | 0.48 | 0.48 | 0.46 | 0.46 | 0.48 | 0.48 |
| Fine-tuned Llama-3.3-70B | 0.60 | 0.64 | 0.67 | 0.67 | 0.74 | 0.74 | 0.77 |
Table 3: The performance on the held-out Push/Pull game, in macro F1 (higher is better). The best result in each round is in bold.
Both fine-tuned models show improved prediction performance as the round number increases, indicating that they leverage accumulated player behavior more effectively with more historical records. This experiment shows that fine-tuning on diverse behavioral data can generalize to a new game scenario.
We have added the per-task results of the fine-tuned models to Appendix D.4 (Table 6), and the held-out Push/Pull results by round to Appendix K.
Thank you for additional results.
-
Given that smaller models outperform frontier LLMs in several tasks, and recent some works (e.g. Small Foundation Models of Human Cognition and Behaviour) shows that sub-1B models can match Centaur, what do the authors see as the unique advantage of LLMs for behavioral prediction?
-
Prior technical reports (Centaur May Have Learned a Shortcut that Explains Away Psychological Tasks) suggest that such foundation models may exploit sequential shortcuts. Could the improvements across rounds in the Push/Pull experiment reflect round effects or simple action persistence rather than better behavioral generalization? What do the authors think? (first-round performance and control for round effects may be convincing evidences.)
We thank the reviewer for the questions and comments!
Thank you for suggesting these additional baselines. Following your advice, we have run these baselines on the tasks you mentioned:
Mean and majority: choosing the mean or the most common answer in the fine-tuning set.
Last action: directly answering using the subject's most recent answer or action in the input.
Training-population distribution: answering with one random draw from the training data, as a reference for the Wasserstein distance.
XGBoost (fine-tuned): gradient-boosted trees fine-tuned on features parsed from the input.
ModernBERT-large (fine-tuned): a recent encoder-only transformer model (395M parameters) with leading performance on classification and retrieval tasks (Warner et al., 2025). We fine-tune a separate ModernBERT-large model on each task, using only that task’s part of our fine-tuning dataset, as shown in Table 5 (Appendix B.3).
The results are shown below.
Table 1: Individual-level performance of the baselines and two general-purpose frontier LLMs. The best result in each column is in bold. ± is the standard error over 1,000 bootstrap resamples of subjects. A dash means the baseline does not apply.
At the individual level, the general-purpose frontier LLMs still lag task-specific models such as ModernBERT-large (fine-tuned). This shows that the input holds information that the frontier LLMs do not fully use, indicating a limitation of current foundation models. We also note that the gap between the leading frontier LLM and ModernBERT-large (fine-tuned) is small on some tasks (e.g., Masked Resp. Pred.).
Table 2: Distributional-level performance of the training-population distribution and two general-purpose frontier LLMs, in Wasserstein distance (W, lower is better). The best result in each column is in bold.
At the distributional level, the training-population distribution is a strong baseline. It has the lowest W on three of the four tasks. This suggests substantial room for improving foundation models.
We have added these results and discussions in Appendix J.
Warner et al., 2025: Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference, ACL.