- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes using behavioral hypotheses screened by subsequent independent evidence as the unit for evaluating LLMs as simulated human participants, and builds HUMANSTUDY-BENCH on this basis. Through a human-in-the-loop Filter–Extract–Execute pipeline, the platform reconstructs 12 classic human-subject studies as executable protocols and administers them to 10 models × 4 agent designs, with 6,588 simulated participants per configuration. The paper introduces two metrics: PAS and ECS. The main findings are threefold: test-level inferential agreement is polarized; effect-magnitude agreement is low; and adding participant descriptions changes agreement in different directions across hypotheses, with no clear average effect.
Critical strengths
1. A principled evaluation lens with rigorously designed metrics. Given that classic effects may themselves fail to replicate, screening hypotheses with subsequent independent evidence, distinguishing direct from related support, and excluding hypotheses with mixed or contrary follow-up results is more rigorous than treating original studies as ground truth. The three-state form of PAS penalizes cases where both sides are significant but in opposite directions, and the continuous mapping avoids discontinuities at significance thresholds.
2. Implementation details are clearly presented. The platform supports community extension with new studies and agent designs. The paper reports costs, a prompt-paraphrase ablation, and an AI use statement, and its discussion of limitations (WEIRD samples, stereotyping risks, unsuitability for high-stakes decisions) is candid.
Critical weaknesses
1. Limited novelty and scale relative to prior work. The paradigm of administering human-subject experiments to LLMs and testing whether population-level effects replicate was established before, and some cited works such as Cui et al. (2025), Ashokkumar et al. (2026), and Hu et al. (2026, SimBench) are considerably larger in the number of experiments or in coverage. This paper includes only 12 studies and 45 hypotheses, making its contribution resting mainly on the evaluation lens and the metrics. How this work differs from them in evaluation unit, metrics, and conclusions? Compared with simpler metrics on the same data, do PAS/ECS lead to different conclusions, and what additional information do they provide? If the two are highly correlated, the incremental value of PAS needs to be re-argued.
2. PAS and ECS are mechanically affected by simulated-population variance, which may confound the core findings. A large body of work shows that response diversity in LLM-simulated populations is substantially lower than in humans.
- For PAS: with N fixed, compressed within-group variance inflates t/F statistics and pushes Bayes factors toward extremes. The polarization (25% of tests > 0.9, 31% < 0.1) may therefore mainly reflect variance collapse in agents rather than a property of agreement itself.
- For ECS and RQ2: standardized effect sizes d divide by the standard deviation. Variance compression systematically inflates d, which can directly explain the finding that overestimation is more common than underestimation, independent of whether the mean differences themselves are larger.
3. Training-data contamination is not addressed. All 12 studies are textbook classics. The agreement may come from memorized findings rather than behavioral simulation, and disagreement may come from recognizing the classic problems.
4. LLM components in the pipeline may introduce bias and lack validation. For example, free-text answers are converted into structured answers by a DeepSeek formatter model, while DeepSeek V3.2 is itself an evaluated model. The paper reports neither per-model parse-failure or refusal rates nor the agreement between formatter outputs and human annotation.
5. Limited coverage of agent designs. All four designs are prompt-based. The data-based methods the paper itself cites (e.g., Centaur from Binz et al. 2025, and the fine-tuned models of Kolluri et al. 2025) are not evaluated.
Questions for authors
-
RQ2 states that 50% show no effect or the opposite direction. What fraction is no effect and what fraction is the opposite direction? The two have very different implications.
-
Does fixing the agent sample size to the original human N make PAS incomparable across studies? If agent N were scaled up uniformly, would the rankings remain stable?
-
How do agents perform on the hypotheses that were reconstructed but not scored? This would test whether the screening changes the conclusions.
-
The ten-model mixture reaches a PAS close to most single models. Is this mainly because mixing increases response variance?
4 comments
Dear Reviewer i92J,
Thank you again for your constructive feedback. We have provided detailed responses addressing your main concerns, specifically showing that hypothesis screening changes benchmark conclusions, and that our findings hold on raw, unstandardized scales rather than reflecting variance collapse. Could you please confirm if these responses have sufficiently addressed your concerns?
Response to the Authors
Thank you for the detailed responses. The rebuttal addresses several of my original concerns. However, some important concerns remain unresolved. They are central to the benchmark's validity, as they affect whether the reported results genuinely reflect the fidelity of simulated human behavior. In particular:
-
Variance compression and PAS polarization. The breakdown of null-favoring and opposite-favoring tests helps clarify PAS failures, but does not fully address whether variance compression contributes to overly concentrated posterior agreement in some tests. The native-scale analysis is also limited to two contrasts, leaving the robustness of the effect-magnitude findings across the benchmark unclear.
-
Training-data contamination. The authors argue that recalling experimental findings does not necessarily enable agents to reproduce participant-level behavior or native-scale effect magnitudes. This is reasonable, but does not directly rule out contamination as a contributor to the observed agreement. In particular, models may leverage prior knowledge of classic experimental findings to reproduce expected response patterns, rather than generate responses through faithful participant-level behavioral simulation. This makes it difficult to disentangle genuine behavioral fidelity from the effects of memorized experimental knowledge.
Thank you for the follow-up. We address your remaining concerns below.
-
High PAS is not driven by variance compression. 65% of the tests with PAS > 0.9 are choice-proportion tests, where variance is determined by the proportion itself and cannot be compressed independently of the effect. In the remaining tests, the agent effect on the raw scale (without SD division) is close to the human effect (median ratio 0.85).
The effect-magnitude finding does not depend on standardization. We recomputed all SD-based tests on their native scales. Only 35% of agent effects fall within a factor of two of the human effect.
-
Ruling out memorization is beyond the scope of our benchmark. Memorization matters only for work that studies the mechanism behind simulator outputs. Our benchmark evaluates whether simulator outputs reproduce validated human effects, not the mechanism behind them. Widely used benchmarks such as SWE-bench and SimBench take the same approach: they are built from public data, do not rule out contamination, and remain valuable to the community. At the current stage, agents perform poorly on these tasks even though the studies are well-known classics: mean PAS is only .462, and nearly half of all tests show no effect or the opposite direction. If memorization contributes at all, true simulation fidelity would be even lower, which only strengthens our conclusion.
Separating memorization from behavioral simulation is a research question in its own right and would warrant a new paper; no single paper can resolve every open question in the field.
Thank you. Screening drives model rankings, and native-scale checks show our findings are not variance artifacts.
Thank you for recognizing the rigor of our evidence-screened lens and metrics. Below, we address your concerns.
Novelty, scale, and metric value. Scale without reference validation conflates agent failure with unreplicable human studies. Prior benchmarks treat published human findings as ground truth. We audit the reference first—separating direct support for a contrast from related support—and reconstruct all testable hypotheses within each study (68 hypotheses and 151 tests across 12 studies, with 45 validated hypotheses).
This screening materially changes benchmark conclusions. On identical responses, agents score much higher on the 45 supported hypotheses than the 23 excluded ones (mean PAS .462 vs. .304; .396 vs. .304 within the 9 shared studies). Including the excluded targets even flips the top-ranked configuration.
PAS and ECS also catch failures that simpler metrics miss. In anchoring, Gemma A1 matches the human effect sign, yet its posterior favors the null (PAS .038). On 65 shared effect pairs, ECS and mean absolute error select different best configurations of GPT-OSS-20B (A3 vs. A1) because ECS accounts for relative effect sizes. In the revision, we will report simple metrics alongside PAS and ECS.
Why 12 studies. Twelve is enough to show our claim: validated behavioral hypotheses work as a lens for evaluating participant simulation. More studies would make the benchmark bigger, not the argument stronger. Beyond this paper, more than 100 studies have already been reconstructed on our platform and its companion application. They belong to the researchers who built them, who are entitled to credit and to decide how their work is published, so they are not included in this anonymous submission.
Variance collapse does not explain away our findings. First, compressed variance does not manufacture PAS failures: half of all tests fail because agents produce no effect (34.9% null-favoring) or the opposite direction (14.7% opposite-favoring)—errors that tight variance cannot turn into agreement. Moreover, population variance is part of the evaluated behavioral distribution; inferential metrics should reflect variance collapse rather than mask it.
Second, poor effect-magnitude agreement is not a denominator artifact from dividing by a small standard deviation. On raw native scales without SD division, agents still miss human effect sizes in both directions: across all admissible completions and at matched usable sample counts, agents overstate the human intentionality gap (.78–.98 vs. .59) while understating the peer-comfort gap (.55–1.53 vs. 1.71).
Training-data contamination. Contamination does not invalidate our evaluation either way. First, recalling a group finding does not calibrate an agent queried one participant at a time across separate conditions; even on familiar classics, prompted agents miss native-scale magnitudes. Second, if disagreement stems from recognizing a classic problem, that is still a simulation failure: real participants are naive to the literature, so an agent that breaks role-play on recognizable setups is not a faithful surrogate.
Formatter bias. Formatting noise does not drive our results. Across ten studies, we independently recovered deterministic labels for nearly 92% of archived entries; substituting them changes fixed-panel mean PAS and ECS by less than .002. Only free-prose and incomplete replies remain outside this check, which we will state clearly.
Agent designs. Our four designs evaluate the prompted-agent paradigm that researchers deploy off the shelf across new protocols. Data-grounded and fine-tuned approaches (e.g., Centaur, Kolluri et al.) are far more costly. We will clarify this scope and leave fine-tuned models to future evaluations.
No effect vs. opposite direction. On the study-weighted archive split (posterior classifications), 34.9% of tests are null-favoring and 14.7% are opposite-favoring. We will report this breakdown explicitly in the revision.
Sample-size scaling and comparability. PAS measures posterior agreement at the original human N, intentionally reflecting each study's empirical precision alongside effect differences. Across four half-unit draws (preserving participants, sessions, and dyads) on 115 fixed tests, mean PAS stays nearly unchanged (.469 vs. .453–.457) and rankings remain stable (rank correlations .947–.970).
Performance on excluded hypotheses. As shown above, screening separates supported from excluded targets (.462 vs. .304 overall; .396 vs. .304 in the 9 shared studies) and flips the top configuration.
Ten-model mixture. On the peer-comfort contrast, differences between model means explain 47–57% of the variance in mixed paired gaps across the four designs. Because this decomposition covers one contrast, the driver of the suite-wide mixture score remains open, which we will note explicitly.