- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether llms can accurately simulate heterogeneous treatment effects in social science experiments. The authors construct H-Bench by aggregating 138 existing randomized experiments from five publicly available data sources, resulting in 530 treatment–outcome contrasts and 2,906 subgroup comparisons.
The authors find that models predict ATEs more accurately than subgroup-level treatment interactions. They further report that most evaluated models overestimate treatment-effect heterogeneity. LLM-based targeting policies also offer limited improvement over random treatment assignment.
The paper also investigates potential explanations for these observations, including the cancellation of prediction errors across treatment conditions and the exaggeration of familiar political and demographic differences.
Critical strengths
- The paper attempts to address an important problem in social simulation using llms.
- The dataset is comprehensive and built from real-world experiments, which is needed in this field.
Critical weaknesses
- Several related observations have already been documented in previous literature. For example, some previous work [1] has investigated the discrepancy between LLMs' ability to predict aggregate experimental effects and their substantially weaker performance on demographic interactions. They showed that strong correlations with average treatment effects do not necessarily extend to demographic interaction effects. Other works [2,3,4] have already investigated related failure modes in persona-based simulations and proposed some solutions. Also, Section 5.1 highlights that prediction errors in the treatment and control groups can cancel out, leading to seemingly accurate ATE estimates. This is closely related to existing work on prediction-powered inference (PPI), which addresses biased model predictions by leveraging real observations for statistical correction.
- The paper introduces H-Bench as a new benchmark, but the underlying human experimental data are drawn from existing sources.
[1] Anthis, Jacy Reese, et al. “Position: LLM Social Simulations Are a Promising Research Method.” 2025
[2] Wang, Angelina, et al. “Large Language Models That Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups.” 2025
[3] Cheng, Myra, et al. “CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations.” 2023
[4] Zhang, B., Li, J., Hortaçsu, A., Ye, X., Chernozhukov, V., Ni, A., & Huang, E. W. Agentic economic modeling. 2025
Questions for authors
- Do you think there are temporal effects on the dataset? For example, some datasets were used to train the models you used, but some were not. How could that influence the outcome?
2 comments
Thanks the author for the review. I think the clarification on the related work and the new exposure experiment addressed some of my concerns. I would like to raise my score to weak acceptance.
We thank the Reviewer for the thoughtful comments. We have addressed each point below and marked the corresponding revisions in the main text and appendix in orange.
We respectfully disagree that our main findings have been documented before. To our knowledge, no prior work systematically tests, across models and against randomized human outcomes, whether LLM simulations recover who responds to an intervention. Our paper establishes four results that the cited works do not:
Anthis et al. [1] is a position paper reviewing LLM social simulations; we now cite it in Related Work. Its remark on demographic interaction effects is a one-sentence summary of earlier work in its appendix, concerning a single model. It reports no analysis of the size, direction, cause, or decision cost of the gap, which are results 2–4 above.
Wang et al. [2] and Cheng et al. [3] study how groups are portrayed without an intervention. We study whether differences in response to an intervention are preserved, a causal question. The distinction matters empirically: eight of eleven models recover baseline group differences reasonably well (r = 0.64–0.76), but interaction correlations range only from −0.01 to 0.23. Evaluations of portrayal therefore cannot detect the failure we document.
PPI and Zhang et al. [4] use real observations to correct model predictions and obtain valid inference for a chosen estimand. Section 5 addresses a different question: it is a diagnosis of why validating simulations on average effects can pass while their heterogeneity is wrong, not a correction method. The two are complementary; we now cite both in Related Work.
Using existing randomized experiments is deliberate: a heterogeneity benchmark needs randomized outcomes with respondent-level characteristics, and peer-reviewed, publicly archived experiments are the most credible and reproducible source. H-Bench contributes what the sources do not provide: (i) a harmonized benchmark of 530 contrasts and 2,906 subgroup comparisons; (ii) matched simulations of every observed profile under every arm for eleven models, including frontier proprietary models (GPT-5.6 Luna, Claude Sonnet 5); and (iii) an evaluation framework with noise correction, cross-fitting, and a held-out targeting design. We will release all data, outputs, and code.
If models had memorized published results, exposure could explain why they recover average effects but not heterogeneity. We dated every experiment by the earliest article, working paper, or dissertation reporting its results (searching the literature as of November 2026) and recomputed the correlations for all eleven models within three groups (new Appendix E and Table 9).
The results do not support exposure as an explanation. In all three groups, every model except the smallest (Gemma 3 4B) recovers average effects better than interactions. Models recover average effects on unpublished experiments at least as well as on older ones (median r = 0.77 vs. 0.63), so this accuracy does not depend on having seen published results. Median interaction correlations are higher for experiments published by 2021, which could reflect exposure or study composition, but this would only flatter models on published data.