- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how LLM-simulated participants compare psychologically to real people. It adapts several experiments for existing effects and uses a regression to analyze this with respect to model bias, replication effects, and the gap between the original and replicated results.
Critical strengths
I really like the approach of focusing on the gap between the original and replicated results, and I think it's justified really well. I appreciate the controls and analysis exploring the limits of the method. I think incorporating the model’s own bias into the mixed effects model is a clever way to account for it, and the finding that it is highly impactful, as well as that original and replicated results correlate with simulations at the same time, is compelling evidence that these systems cannot be naively used to simulate people.
Critical weaknesses
I’m concerned about the variance collapse: the finding that simulated participants are nearly identical in their responses. This can make small differences look like huge effects and impact the interpretability of the coefficients of the model. It also means that the test for publication bias, which is primary, is unstable, which we see when we get the max value for multiple effects. Relatedly, some have adopted demographic persona prompting to increase variation in responses by simulating how groups with a particular demographic makeup might respond to studies. This practice Wang et al. show to be deeply flawed. Even so, some readers might feasibly try to use it to address the variance collapse problem, so it might be worth making the relevant experiment more prominent in the main text and examining results between groups. Not exactly a weakness per se, just something to consider. Furthermore, given that most experimental setups were inferred from written descriptions, it may be worth validating that they have similar results to the originals on real people, to rule out that the differences are due to different experimental conditions rather than legitimate differences between humans and simulations.
Questions for authors
- How did you determine that the adapted text experiments reasonably measured the same effects as the original studies?
- What changes did you make to the experimental setup (e.g. raise temperature) to address variance collapse, and how did results change?
4 comments
Author Response to Reviewer qYqm (2/2)
W3 and Q1. Given that most experimental setups were inferred from written descriptions, it may be worth validating that they have similar results to the originals on real people. How did you determine that the adapted text experiments reasonably measured the same effects as the original studies?
TL;DR: We have no human data on the adapted texts. On three tasks run with the original items and scoring, the models still differ from humans in spread and in effect size.
The submission relies on two independent transcriptions of each experiment and on per-template provenance records. It does not validate the adapted texts on human participants, which is why the paper describes each simulation as a text-adapted analogue. The following evidence is added.
- Original items and scoring. Three tasks for which participant-level human data are public were run with the original wording, items, response scales and scoring (Knobe from Many Labs 2, n = 5,676; the smiley study replication, n = 847; the Mellers replication, n = 510). The models were Claude Sonnet 4.5, GPT-5.6-terra and DeepSeek V4.1. The median ratio of simulated to human within-condition SD is 0.03, 0.14 and 0.19 for the three models. Effect sizes differ from the human ones in both directions. On the Knobe task the human contrast is 3.01 scale points and the models give 5.00, 1.34 and 4.70. On the smiley task it is 0.18 and the models give 0.01, 0.35 and 0.15. On the Mellers task it is 8.9 and the models give 2.6, −2.3 and −0.5. Aligning items and scoring thus does not remove these differences. The procedure is still not identical to the human one, since the model receives a participant-role instruction and answers in one text turn, and the human Mellers data do not record the order of the two vignettes.
- Dependence of the main result on reconstruction. The 55 units whose templates are both reconstructions give the same coefficient pattern as the full sample (0.46 [0.14, 0.78] for the model's own bias and 0.93 [0.35, 1.51] for the replication estimate). Four tasks are known to differ from the human study in the endpoint or contrast that is scored, and excluding them leaves the two coefficients at 0.39 [0.16, 0.62] and 1.11 [0.40, 1.85].
The revision separates the results that do not depend on comparability with the human protocol (the associations across effects, the within-condition variance, the refusal pattern) from those that do (comparisons of level with human effect sizes, and direction agreement), and states that the second group holds for the text-adapted analogues only.
Q2. What changes did you make to the experimental setup (e.g. raise temperature) to address variance collapse, and how did results change?
TL;DR: The submission changed only response scales and personas. A new temperature sweep shows that higher temperature does not restore a human-like spread.
In the submission, the changes aimed at the variance collapse were the 0 to 100 response scales and the personas. Sampling was already stochastic. Claude Sonnet 4.5 ran at temperature 1.0, the maximum of its API, and the other models ran at provider defaults.
Temperature has now been varied directly on the three tasks run with the original items and scoring, where the human within-condition SD is known (10 cells, 80 simulated participants per cell and temperature, 7,200 calls).
| Model | Temperatures | Median simulated SD / human SD |
|---|---|---|
| Claude Sonnet 4.5 | 0, 0.5, 1.0 | 0.00, 0.00, 0.03 |
| GPT-5.6-terra | 0.2, 1.0, 1.5 | 0.14, 0.14, 0.13 |
| DeepSeek V4.1 | 0.2, 1.0, 1.5 | 0.20, 0.19, 0.21 |
For Claude Sonnet 4.5, the number of distinct responses per cell rises with temperature, so the parameter is applied, and at the maximum five of the ten cells still have zero variance. GPT-5.6-terra also responds to the parameter on an open-ended control prompt (a request to name any noun, 40 calls per temperature), so its flat ratio is informative. On the DeepSeek endpoint the same control prompt shows no response to the parameter, so we report its values without attributing them to temperature.
Treatment contrasts change little across temperatures. No setting brings the simulated spread above about a fifth of the human one, which supports the paper's treatment of the collapse as a problem of distributional alignment.
Thanks for your response [and additionally, I appreciate the easy-to-follow formatting of the rebuttal :)]. I'm happy with the changes and will keep my positive score.
We are grateful for your time and consideration in reviewing our rebuttal. Your insightful comments are greatly appreciated.
Author Response to Reviewer qYqm (1/2)
We appreciate the reviewer's thoughtful comments and encouraging evaluation.
TL;DR: The variance collapse is itself a finding of the paper. The two coefficients we interpret stay positive when the units at the maximum are dropped.
Section 3 of the revision states that the coefficients describe which effects a simulation makes large and do not measure calibrated transfer.
TL;DR: Both suggestions are adopted. Personas shift group means in a plausible direction but leave the spread at about half the human level.
Persona prompting therefore moves group means while leaving within-group variation far too narrow, consistent with the flattening reported by Wang et al. The persona comparison moves to the main text, which now says that personas do not resolve the variance collapse.