Validated Behavioral Hypotheses as a Lens for Evaluating Participant Simulaton

We propose using validated behavioral hypotheses as a lens for evaluating LLM agents as simulated human participants. This approach makes behavioral agreement in participant simulation measurable and decomposable, revealing which human effects agents reproduce, where they diverge, and how agent design changes that agreement. To operationalize this idea, we build HumanStudy-Bench, an open benchmarking platform that reconstructs experimental protocols from published human studies and administers these protocols to agents serving as silicon participants. The benchmark compares population-level effects derived from agent responses with the corresponding published human findings using two metrics: the Probability Alignment Score (PAS) for inferential agreement and the Effect Consistency Score (ECS) for effect-magnitude agreement. We apply HumanStudy-Bench across 12 human studies, evaluating 10 models under four agent designs, with 6,588 simulated participants per agent configuration. By making published human studies and their validated behavioral hypotheses reusable for evaluating simulated participants, we hope to make the behavioral assumptions underlying LLM-based social simulations and their sensitivity to agent design explicit and empirically testable.