acceptodds
Under review as a conference paper at ICLR 2027

SCE-Bench: Multi-scale Behavioral Alignment of LLM Agents in Real-world Microeconomics

Abstract

The adoption of Large Language Models within Agent-Based Modeling enables the simulation of economic behaviors, yet the empirical validity of these LLM-driven agents remains largely unverified. Current benchmarks often emphasize normative optimization or game-theoretic tasks, omitting the temporal consistency and individual heterogeneity necessary for realistic economic modeling. In response, we introduce SCE-Bench, derived from the Survey of Consumer Expectations using over 70,000 nationally representative observations from 2016 to 2023. We establish a hierarchical evaluation protocol that measures alignment at both the aggregate level, through temporal correlation and distribution matching, and the individual level, using a Monte Carlo-based approach to assess individual differences and statistical indistinguishability. Our empirical analysis covers comparisons across model characteristics, prompt variations, and temporal contexts. The results indicate that LLMs often exhibit an “Average Person” bias and lack precise temporal tracking. Meanwhile, cost-effective open-weight models emerge as viable alternatives for large-scale simulations. These findings, along with insights into feature selection, verbalization, role-playing perspective, and temporal contexts, establish practical guidelines for configuring LLM economic agents in realistic simulations. Finally, we further validate the generalization of our framework and findings on cross-domain tasks and downstream simulation systems. Dataset, code, and experimental results are available at https://anonymous.4open.science/r/SCE-Bench-BBA7.

Primary area:
datasets and benchmarks
Keywords:
LLMs, microeconomics, ABM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7QATOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SCE-Bench to evaluate the behavioral alignment of LLMs with humans in spending expectations, job offer acceptance intentions, and credit application intentions. Models generate responses conditioned on respondent profiles, macroeconomic conditions, and survey questions. The evaluation is conducted at both the individual and aggregated levels.

The authors further compare models and several ablations. The paper reports that LLMs often fail to accurately reproduce population heterogeneity and temporal dynamics, and that improvements in general capabilities do not necessarily translate into more faithful behavioral simulation.

Critical strengths
  • The proposed task is meaningful and important in economic simulation
  • Investigating the problem at the individual and aggregated levels is correct and effective.
Critical weaknesses
  • My main concern is that the work doesn't include enough baselines. More baselines can better showcase the dataset's effectiveness. For example, a basic fine-tuning baseline would be useful. For aggregate-level estimation, prediction-powered inference (PPI) or related approaches [1, 2] are also worth considering, since it combines model predictions with a smaller human-labeled sample to correct for prediction bias. Recent work [3] also learns weights for combining synthetic agents’ outputs using human observations. This seems particularly relevant to the paper’s aggregate-level evaluation and would provide a useful comparison beyond raw LLM responses.

  • Some of the reported results are inconsistent. For example, Figure 2(k) on page 5 reports a Spending MAE of 4.38 for Qwen3 without thinking, while Table 9 on page 20 reports 7.69 for the same configuration. Some experimental details also need more justification. Figure 14 asks the model to produce a list of 50 estimates. Although the prompt requests “independent samples,” it is unclear whether an LLM could generate independent samples in this way.

[1] Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. Prediction-Powered Inference. Science, 2023 [2] Wang, Mengxin, Dennis J. Zhang, and Heng Zhang. "Large language models for market research: A data-augmentation approach." Marketing Science 45.4 (2026): 728-751. [3] Zhang, B., et al. Agentic Economic Modeling. arXiv:2510.25743, 2025.

Questions for authors

See the weakness above. Some experimental details need more clarifications.

Reviewer_Pe9HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark, SCE-Bench, derived from the Survey of Consumer Expectations (SCE). Their evaluations cover both aggregate and individual levels, multiple models, prompt variations, temporal contexts, etc. Their results show an “average persona” bias, lack of temporal tracking, different behavior depending on model family, and more.

Critical strengths

The analysis shows depth, even if it uncovers already known results. For example, the authors explore the influence of prompt variations, aggregate vs. individual-level analysis, models, data leakage, and more.

The paper is also easy to read and to follow.

Critical weaknesses

This paper mentions several contributions or observations that are known in the literature.

Starting with [1]: the setup, analysis, and even visualizations (e.g., density distributions, text boxes in the Appendix, etc.) are strikingly similar. It uses SCE, conditions on profiles, shows temporal responsiveness, data leakage, etc. I’m not sure if this is a resubmission.

Regardless of that very similar paper, similar findings have been reported in other publications. For example (to list just a few): (1) the “average person” bias [2, 3], (2) frontier or reasoning models might not always be better [4], (3) irrelevant context hurts performance [5], (4) effects of the prompts [6], etc.

[1] Liu, G., Wang, C., Li, J., Wu, H., & Jiang, C. (2026, July). Do LLM Agents Really Mimic Humans? Diagnosing and Aligning Microeconomic Behaviors in Macro-ABMs. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 36102-36122).

[2] Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023, July). Whose opinions do language models reflect?. In International conference on machine learning (pp. 29971-30004). PMLR.

[3] Hu, T., Baumann, J., Lupo, L., Collier, N., Hovy, D., & Röttger, P. (2026, April). Simbench: Benchmarking the ability of large language models to simulate human behaviors. In International Conference on Learning Representations (Vol. 2026, pp. 28290-28341).

[4] Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., & Griffiths, T. L. (2024). Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333.

[5] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., ... & Zhou, D. (2023, July). Large language models can be easily distracted by irrelevant context. In International conference on machine learning (pp. 31210-31227). PMLR.

[6] Lutz, M., Sen, I., Ahnert, G., Rogers, E., & Strohmaier, M. (2025, July). The Prompt Makes the Person (a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models. In EMNLP (Findings) (pp. 23212-23237).

Questions for authors

I don’t have any questions that could change my assessment. I think this work mostly finds what we already know and overlaps significantly with related work.

Reviewer_6CawOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper leverages the Survey of Consumer Expectations (SCE) to construct a large-scale dataset of consumer responses concerning spending, credit, and labor. It compares these human responses with those generated by various LLMs, including GPT, Gemini, and Qwen models. The main findings suggest that LLMs tend to exhibit an "average person" bias and struggle to accurately track temporal changes in human economic behavior.

Critical strengths

The paper conducts extensive experiments using a large-scale, real-world consumer survey covering spending, credit, and labor. The dataset is practically relevant and includes observations collected over an eight-year period, allowing the authors to evaluate LLM-generated responses across different economic conditions and temporal contexts.

Critical weaknesses
  • Limited methodological contribution. The paper primarily focuses on comparing real human responses with those generated by different LLMs, without introducing substantial methodological innovations. Although the experiments reveal that LLMs struggle with temporal tracking and individual-level behavioral alignment, the paper does not propose novel solutions with sufficient technical depth to address these limitations. As a result, the contribution is largely empirical rather than methodological.

  • Limited novelty of the main findings. The key conclusions that LLMs exhibit an "average person" bias and have limited ability to accurately capture temporal dynamics are not particularly surprising.

Questions for authors

There appears to be an inconsistency between Figure 2 and Table 9. Table 9 reports the estimated values of different evaluation metrics for the spending task across various LLMs. For example, the reported JS divergence for GPT5m-t is 0.13. However, Figure 2(f) reports a JS divergence of 0.3999 for the same model and task. Could the authors clarify the reason for this discrepancy?

Minor Suggestions (No Rebuttal Response Required):

  • Tables should be explicitly referenced at appropriate points in the main text. Otherwise, it is not always clear when readers are expected to consult them.
  • The related work section would be better placed in the main body of the paper rather than in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.