acceptodds
Under review as a conference paper at ICLR 2027

Validated Behavioral Hypotheses as a Lens for Evaluating Participant Simulaton

918 views

Abstract

We propose using validated behavioral hypotheses as a lens for evaluating LLM agents as simulated human participants. This approach makes behavioral agreement in participant simulation measurable and decomposable, revealing which human effects agents reproduce, where they diverge, and how agent design changes that agreement. To operationalize this idea, we build HumanStudy-Bench, an open benchmarking platform that reconstructs experimental protocols from published human studies and administers these protocols to agents serving as silicon participants. The benchmark compares population-level effects derived from agent responses with the corresponding published human findings using two metrics: the Probability Alignment Score (PAS) for inferential agreement and the Effect Consistency Score (ECS) for effect-magnitude agreement. We apply HumanStudy-Bench across 12 human studies, evaluating 10 models under four agent designs, with 6,588 simulated participants per agent configuration. By making published human studies and their validated behavioral hypotheses reusable for evaluating simulated participants, we hope to make the behavioral assumptions underlying LLM-based social simulations and their sensitivity to agent design explicit and empirically testable.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Social Simulation, Participants Simulation, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_i92JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using behavioral hypotheses screened by subsequent independent evidence as the unit for evaluating LLMs as simulated human participants, and builds HUMANSTUDY-BENCH on this basis. Through a human-in-the-loop Filter–Extract–Execute pipeline, the platform reconstructs 12 classic human-subject studies as executable protocols and administers them to 10 models × 4 agent designs, with 6,588 simulated participants per configuration. The paper introduces two metrics: PAS and ECS. The main findings are threefold: test-level inferential agreement is polarized; effect-magnitude agreement is low; and adding participant descriptions changes agreement in different directions across hypotheses, with no clear average effect.

Critical strengths

1. A principled evaluation lens with rigorously designed metrics. Given that classic effects may themselves fail to replicate, screening hypotheses with subsequent independent evidence, distinguishing direct from related support, and excluding hypotheses with mixed or contrary follow-up results is more rigorous than treating original studies as ground truth. The three-state form of PAS penalizes cases where both sides are significant but in opposite directions, and the continuous mapping avoids discontinuities at significance thresholds.

2. Implementation details are clearly presented. The platform supports community extension with new studies and agent designs. The paper reports costs, a prompt-paraphrase ablation, and an AI use statement, and its discussion of limitations (WEIRD samples, stereotyping risks, unsuitability for high-stakes decisions) is candid.

Critical weaknesses

1. Limited novelty and scale relative to prior work. The paradigm of administering human-subject experiments to LLMs and testing whether population-level effects replicate was established before, and some cited works such as Cui et al. (2025), Ashokkumar et al. (2026), and Hu et al. (2026, SimBench) are considerably larger in the number of experiments or in coverage. This paper includes only 12 studies and 45 hypotheses, making its contribution resting mainly on the evaluation lens and the metrics. How this work differs from them in evaluation unit, metrics, and conclusions? Compared with simpler metrics on the same data, do PAS/ECS lead to different conclusions, and what additional information do they provide? If the two are highly correlated, the incremental value of PAS needs to be re-argued.

2. PAS and ECS are mechanically affected by simulated-population variance, which may confound the core findings. A large body of work shows that response diversity in LLM-simulated populations is substantially lower than in humans.

  • For PAS: with N fixed, compressed within-group variance inflates t/F statistics and pushes Bayes factors toward extremes. The polarization (25% of tests > 0.9, 31% < 0.1) may therefore mainly reflect variance collapse in agents rather than a property of agreement itself.
  • For ECS and RQ2: standardized effect sizes d divide by the standard deviation. Variance compression systematically inflates d, which can directly explain the finding that overestimation is more common than underestimation, independent of whether the mean differences themselves are larger.

3. Training-data contamination is not addressed. All 12 studies are textbook classics. The agreement may come from memorized findings rather than behavioral simulation, and disagreement may come from recognizing the classic problems.

4. LLM components in the pipeline may introduce bias and lack validation. For example, free-text answers are converted into structured answers by a DeepSeek formatter model, while DeepSeek V3.2 is itself an evaluated model. The paper reports neither per-model parse-failure or refusal rates nor the agreement between formatter outputs and human annotation.

5. Limited coverage of agent designs. All four designs are prompt-based. The data-based methods the paper itself cites (e.g., Centaur from Binz et al. 2025, and the fine-tuned models of Kolluri et al. 2025) are not evaluated.

Questions for authors
  1. RQ2 states that 50% show no effect or the opposite direction. What fraction is no effect and what fraction is the opposite direction? The two have very different implications.

  2. Does fixing the agent sample size to the original human N make PAS incomparable across studies? If agent N were scaled up uniformly, would the rankings remain stable?

  3. How do agents perform on the hypotheses that were reconstructed but not scored? This would test whether the screening changes the conclusions.

  4. The ten-model mixture reaches a PAS close to most single models. Is this mainly because mixing increases response variance?

4 comments
AuthorsOpenReview ↗

Thank you. Screening drives model rankings, and native-scale checks show our findings are not variance artifacts.

Thank you for recognizing the rigor of our evidence-screened lens and metrics. Below, we address your concerns.

Novelty, scale, and metric value. Scale without reference validation conflates agent failure with unreplicable human studies. Prior benchmarks treat published human findings as ground truth. We audit the reference first—separating direct support for a contrast from related support—and reconstruct all testable hypotheses within each study (68 hypotheses and 151 tests across 12 studies, with 45 validated hypotheses).

This screening materially changes benchmark conclusions. On identical responses, agents score much higher on the 45 supported hypotheses than the 23 excluded ones (mean PAS .462 vs. .304; .396 vs. .304 within the 9 shared studies). Including the excluded targets even flips the top-ranked configuration.

PAS and ECS also catch failures that simpler metrics miss. In anchoring, Gemma A1 matches the human effect sign, yet its posterior favors the null (PAS .038). On 65 shared effect pairs, ECS and mean absolute error select different best configurations of GPT-OSS-20B (A3 vs. A1) because ECS accounts for relative effect sizes. In the revision, we will report simple metrics alongside PAS and ECS.

Why 12 studies. Twelve is enough to show our claim: validated behavioral hypotheses work as a lens for evaluating participant simulation. More studies would make the benchmark bigger, not the argument stronger. Beyond this paper, more than 100 studies have already been reconstructed on our platform and its companion application. They belong to the researchers who built them, who are entitled to credit and to decide how their work is published, so they are not included in this anonymous submission.

Variance collapse does not explain away our findings. First, compressed variance does not manufacture PAS failures: half of all tests fail because agents produce no effect (34.9% null-favoring) or the opposite direction (14.7% opposite-favoring)—errors that tight variance cannot turn into agreement. Moreover, population variance is part of the evaluated behavioral distribution; inferential metrics should reflect variance collapse rather than mask it.

Second, poor effect-magnitude agreement is not a denominator artifact from dividing by a small standard deviation. On raw native scales without SD division, agents still miss human effect sizes in both directions: across all admissible completions and at matched usable sample counts, agents overstate the human intentionality gap (.78–.98 vs. .59) while understating the peer-comfort gap (.55–1.53 vs. 1.71).

Training-data contamination. Contamination does not invalidate our evaluation either way. First, recalling a group finding does not calibrate an agent queried one participant at a time across separate conditions; even on familiar classics, prompted agents miss native-scale magnitudes. Second, if disagreement stems from recognizing a classic problem, that is still a simulation failure: real participants are naive to the literature, so an agent that breaks role-play on recognizable setups is not a faithful surrogate.

Formatter bias. Formatting noise does not drive our results. Across ten studies, we independently recovered deterministic labels for nearly 92% of archived entries; substituting them changes fixed-panel mean PAS and ECS by less than .002. Only free-prose and incomplete replies remain outside this check, which we will state clearly.

Agent designs. Our four designs evaluate the prompted-agent paradigm that researchers deploy off the shelf across new protocols. Data-grounded and fine-tuned approaches (e.g., Centaur, Kolluri et al.) are far more costly. We will clarify this scope and leave fine-tuned models to future evaluations.

No effect vs. opposite direction. On the study-weighted archive split (posterior classifications), 34.9% of tests are null-favoring and 14.7% are opposite-favoring. We will report this breakdown explicitly in the revision.

Sample-size scaling and comparability. PAS measures posterior agreement at the original human N, intentionally reflecting each study's empirical precision alongside effect differences. Across four half-unit draws (preserving participants, sessions, and dyads) on 115 fixed tests, mean PAS stays nearly unchanged (.469 vs. .453–.457) and rankings remain stable (rank correlations .947–.970).

Performance on excluded hypotheses. As shown above, screening separates supported from excluded targets (.462 vs. .304 overall; .396 vs. .304 in the 9 shared studies) and flips the top configuration.

Ten-model mixture. On the peer-comfort contrast, differences between model means explain 47–57% of the variance in mixed paired gaps across the four designs. Because this decomposition covers one contrast, the driver of the suite-wide mixture score remains open, which we will note explicitly.

Dear Reviewer i92J,

Thank you again for your constructive feedback. We have provided detailed responses addressing your main concerns, specifically showing that hypothesis screening changes benchmark conclusions, and that our findings hold on raw, unstandardized scales rather than reflecting variance collapse. Could you please confirm if these responses have sufficiently addressed your concerns?

Reviewer_i92JOpenReview ↗

Response to the Authors

Thank you for the detailed responses. The rebuttal addresses several of my original concerns. However, some important concerns remain unresolved. They are central to the benchmark's validity, as they affect whether the reported results genuinely reflect the fidelity of simulated human behavior. In particular:

  1. Variance compression and PAS polarization. The breakdown of null-favoring and opposite-favoring tests helps clarify PAS failures, but does not fully address whether variance compression contributes to overly concentrated posterior agreement in some tests. The native-scale analysis is also limited to two contrasts, leaving the robustness of the effect-magnitude findings across the benchmark unclear.

  2. Training-data contamination. The authors argue that recalling experimental findings does not necessarily enable agents to reproduce participant-level behavior or native-scale effect magnitudes. This is reasonable, but does not directly rule out contamination as a contributor to the observed agreement. In particular, models may leverage prior knowledge of classic experimental findings to reproduce expected response patterns, rather than generate responses through faithful participant-level behavioral simulation. This makes it difficult to disentangle genuine behavioral fidelity from the effects of memorized experimental knowledge.

Thank you for the follow-up. We address your remaining concerns below.

  1. High PAS is not driven by variance compression. 65% of the tests with PAS > 0.9 are choice-proportion tests, where variance is determined by the proportion itself and cannot be compressed independently of the effect. In the remaining tests, the agent effect on the raw scale (without SD division) is close to the human effect (median ratio 0.85).

    The effect-magnitude finding does not depend on standardization. We recomputed all SD-based tests on their native scales. Only 35% of agent effects fall within a factor of two of the human effect.

  2. Ruling out memorization is beyond the scope of our benchmark. Memorization matters only for work that studies the mechanism behind simulator outputs. Our benchmark evaluates whether simulator outputs reproduce validated human effects, not the mechanism behind them. Widely used benchmarks such as SWE-bench and SimBench take the same approach: they are built from public data, do not rule out contamination, and remain valuable to the community. At the current stage, agents perform poorly on these tasks even though the studies are well-known classics: mean PAS is only .462, and nearly half of all tests show no effect or the opposite direction. If memorization contributes at all, true simulation fidelity would be even lower, which only strengthens our conclusion.

Separating memorization from behavioral simulation is a research question in its own right and would warrant a new paper; no single paper can resolve every open question in the field.

Reviewer_hADBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates LLM agents simulating human participants using behavior hypotheses backed by independent human studies. It builds HUMANSTUDY-BENCH, which recreates published human experiments into test tasks executable by AI agents. Two metrics are proposed: the Probability Alignment Score (PAS) for consistency in statistical inferences between model outputs and human findings, and the Effect Consistency Score (ECS) for alignment in behavioral effect magnitudes.

Critical strengths
  1. The research question is practically relevant. LLM-based agents are now widely adopted for social simulation. However, how to verify whether these agents produce behavioral patterns comparable to humans remains an important and understudied problem. The paper translates this challenge into concrete, testable behavioral hypotheses, which helps improve the interpretability of evaluation.

  2. The experiments cover a broad scope and the analysis offers diagnostic insights. The authors evaluate ten models and four agent configurations, and separate biases in the direction and magnitude of behavioral effects. Notably, the paper finds that richer character profiles do not necessarily improve simulation realism. This finding can inform future agent design.

Critical weaknesses
  1. Limited novelty. The paper builds on existing approaches that evaluate LLM agents using human behavioral experiments. The advantages of PAS and ECS over simpler metrics, such as effect-direction accuracy and effect-size error, are not clearly demonstrated.

  2. Insufficient validation of behavioral fidelity. PAS and ECS measure agreement in aggregate statistics, which does not necessarily imply realistic individual behavior. Moreover, converting human experiments into text-based LLM tasks may alter their original behavioral mechanisms, raising concerns about the validity of the evaluation.

Questions for authors
  1. Compared with existing LLM behavioral evaluation methods built on psychological experiments (such as Turing Experiments and SimBench), what new evaluation capabilities does HUMANSTUDY-BENCH deliver that prior works cannot achieve? What experimentally validated advantages do PAS and ECS have over simpler metrics like directional agreement and effect-size error?
  2. PAS and ECS primarily measure consistency in statistical inferences and effect sizes. Yet identical statistical outcomes can arise from distinct distributions of individual-level behaviors. How do the authors demonstrate that these two metrics effectively capture the behavioral realism of simulated participants, rather than merely matching a small set of aggregate statistics?
  3. Some classic human experiments are reconstructed as text-based interactive tasks, which may alter the original incentives and social interaction conditions. How do the authors disentangle performance gaps caused by limited model behavioral capacity from biases introduced by experiment reconstruction? Is there evidence confirming that the reconstructed experiments still target the same behavioral mechanisms examined in the original studies?
3 comments
AuthorsOpenReview ↗

Thank you. Prior evaluations assume the human result is correct. We do not.

Prior evaluations assume the human result is correct. We do not. Turing Experiments runs classic experiments on models and checks whether the published effect shows up again. SimBench compares how models and humans answer across many survey and behavioral datasets. In both cases, the human result is treated as the right answer. We only score agents on validated behavioral hypotheses: published hypotheses that later independent research supports. That way, when an agent disagrees, we can attribute the disagreement to the agent rather than to the human finding.

This makes a real difference. We scored the same agent answers against 45 hypotheses that later research supported and 23 that it did not. Agents agree with the supported ones much more often (mean hypothesis-level PAS .462 vs. .304). The gap holds within the same 9 studies (.396 vs. .304) and in 35 of 40 model configurations.

PAS and ECS can disagree with simpler metrics. In the anchoring study, Gemma A1 shifts in the same direction as humans, so a direction-only metric counts it as a match. But Gemma’s evidence points to no effect, while the human evidence points to a clear positive effect, so PAS gives it only .038. For effect sizes, ECS and mean absolute error pick different best versions of GPT-OSS-20B (A3 and A1) on the same 65 effect pairs. ECS differs because it also checks whether the agent gets the relative sizes of effects right. In the revision, we will report the simpler metrics alongside PAS and ECS.

Individual behavior. We agree that PAS and ECS measure effects at the population level, not individual realism. We work at this level because the published findings make their claims at this level. Testing individual responses would require participant-level human data, which most published studies do not share. The two kinds of evaluation answer different questions: a simulator can produce realistic individual answers and still miss an experimental effect, and our lens is built to catch that. We will state this scope clearly in the paper.

Reconstruction. Turning strategic tasks into text can change payouts, partners and feedback, and our current design cannot fully separate these changes from model limitations. Appendix E lists the changes we made for each study. Our paraphrase test (Appendix F.8) shows that rewording the prompt changes PAS less than changing the agent design does. In the revision, we will describe these results as behavior under the written protocol, and leave claims about the original mechanisms to comparisons with humans under the same protocol.

Dear Reviewer hADB,

Thank you again for your constructive feedback. We have provided detailed responses addressing your main concerns, specifically explaining what our evaluation adds by not treating the published human result as correct, and showing cases where PAS and ECS disagree with simpler metrics. Could you please confirm if these responses have sufficiently addressed your concerns?

Thank you for clarifying the benchmark’s scope. The added value of PAS and ECS over simpler metrics remains unclear, and the reconstructed tasks still need validation with human participants under the same protocol. I’m keeping my original rating.

Reviewer_F33COpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Authors introduce HUMANSTUDY-BENCH, a benchmark for evaluating LLM-based human (or, specifically, participant) simulations against behavioral findings from published human study results. The benchmark reconstructs 12 studies covering 45 scored hypotheses and evaluates 10 LLMs under four participant specifications (raning from blank to backstory conditioning method). It uses Probability Alignment Score (PAS) for directional agreement checking and Effect Consistency Score (ECS) for size effect. The experiments find substantial discrepancies between simulated and human results and inconsistent benefits from demographic conditioning and generated backstory conditioning.

Critical strengths
  1. Paper is easy to follow. This is the only paper in my reviewer batch that is human-readable. I am happy to see that there are authors in ICLR caring about the writing quality.
  2. The paper addresses an important question of whether LLM-based simulations reproduce relevant human behavior.
  3. Reconstructing experimental protocols and preserving links between responses, analyses, and findings provides useful tools for future research on this field.
Critical weaknesses

Given the strength point 1, I hope I could really give a positive rating to this paper, but can't due to the following reasons.

  1. The contribution beyond (multiple) existing works is insufficient

This is the main weakness point. The central empirical conclusions substantially overlap with prior literature

  • Gao et al. ("Take caution in using LLMs as human surrogates.", 2025) demonstrate discrepancies between LLM and human behavior, and similarly for
  • Tjuatja et al. ("Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design.", 2024)
  • Cui et al. ("Can Large Language Models Replace Human Subjects? A Large-Scale Replication of Scenario-Based Experiments in Psychology and Management.", 2024)
  • Bisbee et al. ("Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.", 2024)
  • Wang et al. ("Large language models that replace human participants can harmfully misportray and flatten identity groups", 2025)

just to name a few. The manuscript cites this literature, but does not explain what its proposed evaluation reveals that existing approaches cannot. There are now just too many papers repeatedly pointing to the shortfall of existing LLM-based simulation methods.

  1. The agent design findings offer limited generalizable insight

The four participant specifications (ranging from blank to backstory conditioning) are reasonable agent prompting conditions. But they provide a limited basis for conclusions about human simulation in general. For instance, generated backstories are not equivalent to personas grounded in interview data, or other behavioral records. The experimental observation that additional context helps some hypotheses prediction and hurts others is useful, but the paper provides limited explanation of which information helps, or when it helps, and (some speculation about) why. Consequently, the agent-design experiments add relatively little actionable knowledge beyond just reporting sensitivity to prompting.

Questions for authors

How can the scale of the benchmark be (semi-) automatically scaled beyond 100, 1000, or even 10k samples?

4 comments
AuthorsOpenReview ↗

Thank you. The key question is what this evaluation reveals that existing work cannot. Here is our answer.

We agree that another paper showing LLMs fall short of humans would add little, and that finding is not our focus. Our contribution is the evaluation lens: scoring agents only on behavioral hypotheses that later independent research supports, so that a disagreement can be read as a property of the agent rather than of the human finding.

Among the works you cite, Bisbee et al. and Wang et al. collect new human data, which must be repeated for every new question, and they study survey responses and identity portrayal rather than experimental effects. Gao et al., Tjuatja et al. and Cui et al. are closer to our setting: they score agents against published human results, and this raises a problem.

When an agent disagrees with a published finding, there are two possibilities: the agent is wrong, or the finding is. Many classic findings fail to replicate (Open Science Collaboration, 2015; Klein et al., 2018), so the second case is common. Evaluations that score against published results cannot tell these apart. By default, every mismatch is charged to the agent. Cui et al. note this exact problem: discrepancies "may reflect limitations in the original studies rather than shortcomings of the AI models."

We separate the two. We score agents only on validated behavioral hypotheses: published hypotheses that independent research evidence has since supported. On these, the human side is far more reliable, so a mismatch mostly reflects the agent.

Agents do worse where the human finding is unsupported. We scored the same agent answers against two kinds of human findings: 45 that later research supported, and 23 that later research did not support. Agents agree with the supported ones much more often (mean hypothesis-level PAS .462 vs. .304). This is not just because the two sets come from different studies. Inside the same 9 studies, the gap is still there (.396 vs. .304), and 35 of 40 model configurations show it. In other words, much of what looks like agent failure shows up exactly where the human finding itself is shaky. In the revision, we will expand the related work to position our contribution against each of these works in detail.

Agent design. Agreed: backstories are not grounded personas. The four designs are probes, secondary to the main claim, and we will present them that way.

Scaling. Simulated sample size scales freely, since every protocol can be run at any size. Validated hypotheses scale through expert review of the research evidence, and this is the step our platform adds. We also built a separate application that helps social scientists turn published human studies into executable programs, and more than 100 studies have already been reconstructed this way. Since it is not the focus of this paper, we do not describe it here.

Dear Reviewer F33C,

Thank you again for your constructive feedback. We have provided detailed responses addressing your main concerns, specifically explaining how scoring only validated hypotheses separates agent failure from unreplicable human findings, and clarifying that our agent designs are probes secondary to the evaluation lens. Could you please confirm if these responses have sufficiently addressed your concerns?

Thank you for the detailed and clear response. If the main contribution is the evaluation lens, as authors mentioned, then I think the contribution is more well-scoped than I originally assessed, and I agree with authors that "there are two possibilities: the agent is wrong, or the finding is". Still I think the paper is more well-suited for alternative venues, such as social science or journals studying human behaviors? I would like to change my view of this paper from 1 to somewhere between 1 and 2 (weak rejection).

AuthorsOpenReview ↗

Thank you for reconsidering your score

To be honest, we don't think social science should be seen as a lesser fit for ICLR. Models now win gold medals in math and coding competitions, but on social tasks like ours, they often do little better than a coin flip.

That gap is exactly why AI for social science deserves more attention from the ML community, not less. Our primary area is also "alignment, fairness, safety, privacy, and societal considerations," which is meant for this kind of work, and similar work has appeared at ICLR, including SOTOPIA (ICLR 2024), CogMir (ICLR 2025), and SimBench (ICLR 2026).

ICLR has always been an open and inclusive venue, and we believe social science methods and AI for social science will become an important direction for the ML community.

Comments on the paper

1 comment
Area_Chair_WpUdOpenReview ↗

AC Reminder: Please Confirm Whether Your Concerns Are Addressed

Dear Reviewers,

The authors replied to each review on 12-13 Nov and followed up today asking for confirmation. Please check the response addressed to your review and post a short reply indicating whether your main concerns, in particular the novelty and the value of the proposed evaluation lens, have been addressed. If concerns remain, please specify which ones and why the response is insufficient, so the authors have a fair chance to answer before the discussion period ends. A confirmation, or an updated rating where appropriate, would also help the final assessment.

Thank you for your time. Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.