- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SCE-Bench to evaluate the behavioral alignment of LLMs with humans in spending expectations, job offer acceptance intentions, and credit application intentions. Models generate responses conditioned on respondent profiles, macroeconomic conditions, and survey questions. The evaluation is conducted at both the individual and aggregated levels.
The authors further compare models and several ablations. The paper reports that LLMs often fail to accurately reproduce population heterogeneity and temporal dynamics, and that improvements in general capabilities do not necessarily translate into more faithful behavioral simulation.
Critical strengths
- The proposed task is meaningful and important in economic simulation
- Investigating the problem at the individual and aggregated levels is correct and effective.
Critical weaknesses
-
My main concern is that the work doesn't include enough baselines. More baselines can better showcase the dataset's effectiveness. For example, a basic fine-tuning baseline would be useful. For aggregate-level estimation, prediction-powered inference (PPI) or related approaches [1, 2] are also worth considering, since it combines model predictions with a smaller human-labeled sample to correct for prediction bias. Recent work [3] also learns weights for combining synthetic agents’ outputs using human observations. This seems particularly relevant to the paper’s aggregate-level evaluation and would provide a useful comparison beyond raw LLM responses.
-
Some of the reported results are inconsistent. For example, Figure 2(k) on page 5 reports a Spending MAE of 4.38 for Qwen3 without thinking, while Table 9 on page 20 reports 7.69 for the same configuration. Some experimental details also need more justification. Figure 14 asks the model to produce a list of 50 estimates. Although the prompt requests “independent samples,” it is unclear whether an LLM could generate independent samples in this way.
[1] Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. Prediction-Powered Inference. Science, 2023 [2] Wang, Mengxin, Dennis J. Zhang, and Heng Zhang. "Large language models for market research: A data-augmentation approach." Marketing Science 45.4 (2026): 728-751. [3] Zhang, B., et al. Agentic Economic Modeling. arXiv:2510.25743, 2025.
Questions for authors
See the weakness above. Some experimental details need more clarifications.