acceptodds
Under review as a conference paper at ICLR 2027

What Do LLM-Simulated Participants Track? Model Bias, Replication Estimates, and Collapsed Variance

Abstract

LLM simulations of human participants are usually validated against published effect sizes. Published effects are inflated by publication bias, and a model can also produce an effect because it shows the bias itself, so agreement with the literature cannot tell a faithful simulation from one that echoes the literature or its own habits. We propose a decomposition that separates these sources across effects and apply it at scale. For 61 psychology effects with both an original estimate and a large-sample replication estimate, we run text-adapted analogues of each experiment (four LLMs from three developers, about 43,000 simulated participants in the main protocol) and regress the simulated effect size on the model's own bias, measured by giving it the same task with no participant framing, on the replication estimate, and on the gap between original and replication. When two independent transcriptions of each experiment are averaged, simulated effects covary positively with the model's own bias and with the replication estimate in all three model families, but in one family either coefficient becomes uncertain under a single transcription. Whether simulations also inherit the gap cannot be decided at this scale. The decomposition therefore diagnoses what simulations track but cannot score one simulator. A simulator told to copy the original findings goes undetected when scored alone, while a paired comparison of two simulators on the same effects shows a signal. Simulated samples have almost no within-condition variance, so simulated effects are several times larger than human ones, and where the model shows no bias of its own, agreement with the direction of replicated effects is not distinguishable from chance. One model refuses selectively on aversive treatment arms. In an exploratory construct battery, correlations the literature makes famous are exaggerated relative to a matched human panel. We argue that simulators should be compared on shared replication-anchored protocols with a model-bias control, and that current simulations should not be used to infer effects the model does not already show.

Primary area:
generative models
Keywords:
LLM simulation, publication bias, replicability, simulation validity, training data contamination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qYqmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how LLM-simulated participants compare psychologically to real people. It adapts several experiments for existing effects and uses a regression to analyze this with respect to model bias, replication effects, and the gap between the original and replicated results.

Critical strengths

I really like the approach of focusing on the gap between the original and replicated results, and I think it's justified really well. I appreciate the controls and analysis exploring the limits of the method. I think incorporating the model’s own bias into the mixed effects model is a clever way to account for it, and the finding that it is highly impactful, as well as that original and replicated results correlate with simulations at the same time, is compelling evidence that these systems cannot be naively used to simulate people.

Critical weaknesses

I’m concerned about the variance collapse: the finding that simulated participants are nearly identical in their responses. This can make small differences look like huge effects and impact the interpretability of the coefficients of the model. It also means that the test for publication bias, which is primary, is unstable, which we see when we get the max value for multiple effects. Relatedly, some have adopted demographic persona prompting to increase variation in responses by simulating how groups with a particular demographic makeup might respond to studies. This practice Wang et al. show to be deeply flawed. Even so, some readers might feasibly try to use it to address the variance collapse problem, so it might be worth making the relevant experiment more prominent in the main text and examining results between groups. Not exactly a weakness per se, just something to consider. Furthermore, given that most experimental setups were inferred from written descriptions, it may be worth validating that they have similar results to the originals on real people, to rule out that the differences are due to different experimental conditions rather than legitimate differences between humans and simulations.

Questions for authors
  • How did you determine that the adapted text experiments reasonably measured the same effects as the original studies?
  • What changes did you make to the experimental setup (e.g. raise temperature) to address variance collapse, and how did results change?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer qYqm (1/2)

We appreciate the reviewer's thoughtful comments and encouraging evaluation.

W1. Simulated participants are nearly identical in their responses. This can make small differences look like huge effects and impact the interpretability of the coefficients. It also means that the test for publication bias is unstable, which we see when we get the max value for multiple effects.

TL;DR: The variance collapse is itself a finding of the paper. The two coefficients we interpret stay positive when the units at the maximum are dropped.

  • The publication bias test. Its sensitivity to values at the maximum is the reason the pre-specified verdict is "inconclusive" and the reason the paper adds a power analysis for a decisive design. The paper draws no conclusion from this coefficient.
  • The two coefficients that are estimated. Both remain positive when every unit at the ceiling (|r| above 0.95) is dropped, at 0.20 [0.04, 0.37] for the model's own bias and 0.60 [0.06, 1.13] for the replication estimate (52 units). When the affected effects are remeasured on 0 to 100 scales, the own-bias coefficient is positive in all three families (0.43 for Claude, 0.51 for GPT, 0.44 for DeepSeek). The replication coefficient is positive for Claude and DeepSeek and uncertain for GPT (0.51 [−0.21, 1.21]). On ranks, which do not depend on the inflated scale, the partial correlation of the simulated effect is 0.45 [0.07, 0.74] with the model's own bias and 0.30 [−0.04, 0.64] with the replication estimate.

Section 3 of the revision states that the coefficients describe which effects a simulation makes large and do not measure calibrated transfer.

W2. Some have adopted demographic persona prompting to increase variation in responses. Some readers might feasibly try to use it to address the variance collapse problem, so it might be worth making the relevant experiment more prominent in the main text and examining results between groups.

TL;DR: Both suggestions are adopted. Personas shift group means in a plausible direction but leave the spread at about half the human level.

  • Within groups. Across the 298 experimental arms run both with and without personas, personas raise the within-arm standard deviation by a factor of 1.43 [1.23, 1.68] and lower the share of arms with zero variance from 44% to 31%. In a new run, 80 simulated respondents with personas answered the original items of ten constructs of the Twin-2K-500 panel (GPT-5.6-terra). Their standard deviation is 0.48 [0.42, 0.54] of that of the human panel members.
  • Between groups. In a second new run, each of 80 personas was presented twice with only the stated age changed from 24 to 70. Policy support shifts by −0.28 human SD [−0.36, −0.20], in the direction of the age difference among the human panel members (−0.35 [−0.50, −0.20]). Empathy does not shift, as in the human data (DeepSeek V4.1, the successor of the retired V4 endpoint used in the submission).

Persona prompting therefore moves group means while leaving within-group variation far too narrow, consistent with the flattening reported by Wang et al. The persona comparison moves to the main text, which now says that personas do not resolve the variance collapse.

AuthorsOpenReview ↗

Author Response to Reviewer qYqm (2/2)

W3 and Q1. Given that most experimental setups were inferred from written descriptions, it may be worth validating that they have similar results to the originals on real people. How did you determine that the adapted text experiments reasonably measured the same effects as the original studies?

TL;DR: We have no human data on the adapted texts. On three tasks run with the original items and scoring, the models still differ from humans in spread and in effect size.

The submission relies on two independent transcriptions of each experiment and on per-template provenance records. It does not validate the adapted texts on human participants, which is why the paper describes each simulation as a text-adapted analogue. The following evidence is added.

  • Original items and scoring. Three tasks for which participant-level human data are public were run with the original wording, items, response scales and scoring (Knobe from Many Labs 2, n = 5,676; the smiley study replication, n = 847; the Mellers replication, n = 510). The models were Claude Sonnet 4.5, GPT-5.6-terra and DeepSeek V4.1. The median ratio of simulated to human within-condition SD is 0.03, 0.14 and 0.19 for the three models. Effect sizes differ from the human ones in both directions. On the Knobe task the human contrast is 3.01 scale points and the models give 5.00, 1.34 and 4.70. On the smiley task it is 0.18 and the models give 0.01, 0.35 and 0.15. On the Mellers task it is 8.9 and the models give 2.6, −2.3 and −0.5. Aligning items and scoring thus does not remove these differences. The procedure is still not identical to the human one, since the model receives a participant-role instruction and answers in one text turn, and the human Mellers data do not record the order of the two vignettes.
  • Dependence of the main result on reconstruction. The 55 units whose templates are both reconstructions give the same coefficient pattern as the full sample (0.46 [0.14, 0.78] for the model's own bias and 0.93 [0.35, 1.51] for the replication estimate). Four tasks are known to differ from the human study in the endpoint or contrast that is scored, and excluding them leaves the two coefficients at 0.39 [0.16, 0.62] and 1.11 [0.40, 1.85].

The revision separates the results that do not depend on comparability with the human protocol (the associations across effects, the within-condition variance, the refusal pattern) from those that do (comparisons of level with human effect sizes, and direction agreement), and states that the second group holds for the text-adapted analogues only.

Q2. What changes did you make to the experimental setup (e.g. raise temperature) to address variance collapse, and how did results change?

TL;DR: The submission changed only response scales and personas. A new temperature sweep shows that higher temperature does not restore a human-like spread.

In the submission, the changes aimed at the variance collapse were the 0 to 100 response scales and the personas. Sampling was already stochastic. Claude Sonnet 4.5 ran at temperature 1.0, the maximum of its API, and the other models ran at provider defaults.

Temperature has now been varied directly on the three tasks run with the original items and scoring, where the human within-condition SD is known (10 cells, 80 simulated participants per cell and temperature, 7,200 calls).

ModelTemperaturesMedian simulated SD / human SD
Claude Sonnet 4.50, 0.5, 1.00.00, 0.00, 0.03
GPT-5.6-terra0.2, 1.0, 1.50.14, 0.14, 0.13
DeepSeek V4.10.2, 1.0, 1.50.20, 0.19, 0.21

For Claude Sonnet 4.5, the number of distinct responses per cell rises with temperature, so the parameter is applied, and at the maximum five of the ten cells still have zero variance. GPT-5.6-terra also responds to the parameter on an open-ended control prompt (a request to name any noun, 40 calls per temperature), so its flat ratio is informative. On the DeepSeek endpoint the same control prompt shows no response to the parameter, so we report its values without attributing them to temperature.

Treatment contrasts change little across temperatures. No setting brings the simulated spread above about a fifth of the human one, which supports the paper's treatment of the collapse as a problem of distributional alignment.

Thanks for your response [and additionally, I appreciate the easy-to-follow formatting of the rebuttal :)]. I'm happy with the changes and will keep my positive score.

We are grateful for your time and consideration in reviewing our rebuttal. Your insightful comments are greatly appreciated.

Reviewer_Kj3YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks when AI models serve as human participants in psychology studies, are they truly mimicking human behavior, relying on their innate bias, or just regurgitating research papers they memorized during training.

The authors evaluated 61 psychology findings. They compared human data against two AI setups: LLMs with a participant persona, and LLMs answering the exact same tasks without persona. The main takeaway is that just because an LLM can create a well-known finding from a textbook doesn't mean that it's actually simulating human mindsets.

Critical strengths

The study addresses a useful and challenging question: whether LLM participants reproduce human behavior or reflect the model’s own biases and knowledge of published findings. Another strength of the study is that it combines substantial set of psychological studies, covering 61 psychological effects.

Critical weaknesses

My main concern is whether the regression can separate the explanations the authors care about. Both of my concerns are related to the validity of the measurements.

  1. Asking an LLM a question without a persona doesn't isolate its default reasoning. It just queries a model that has already read the psychology literature. Memorized papers and shared phrasing can easily bleed into both the bare-task and persona responses. While the regression shows these outputs correlate, we do not know why, and we cannot untangle the true source of the behavior. Throwing more data at the model won't solve this underlying structural confound.

  2. Out of 122 task templates, only 20 actually used original study materials. The rest relied on variations. When an LLM's response diverges, we cannot tell whether the model failed or if it simply reacted to a different prompt. To make a valid comparison, humans and models need to complete the exact same protocol.

Questions for authors
  • How does the bare-task control distinguish the model’s own biases from its knowledge of this well-known psychology literature?
  • How to establish that the reconstructed tasks are comparable to the human experiments?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Kj3Y (1/2)

We thank the reviewer for two precise questions. Both concern limits of the design. We begin with the comparison against human data, where the new evidence is most direct, and then turn to the bare-task control.

W2 and Q2. Out of 122 task templates, only 20 actually used original study materials. When an LLM's response diverges, we cannot tell whether the model failed or if it simply reacted to a different prompt. To make a valid comparison, humans and models need to complete the exact same protocol. How to establish that the reconstructed tasks are comparable to the human experiments?

TL;DR: Comparability is not established for the reconstructed templates, so the claims that depend on it are now restricted. On three tasks run with the original items and scoring, the models still differ from humans.

Comparability is not established for the reconstructed templates, and we have no human data on them. For a single reconstructed template, a divergence from the human result therefore cannot be attributed to the model, and the revision makes no claim of that kind. The evidence below shows what is observed once items and scoring are aligned, and how much the prompt itself matters.

  • Original items and scoring, where human data exist. Three tasks for which participant-level human data are public were run with the original wording, items, response scales and scoring (Knobe from Many Labs 2, n = 5,676; the smiley study replication, n = 847; the Mellers replication, n = 510). The models were Claude Sonnet 4.5, GPT-5.6-terra and DeepSeek V4.1. On the Knobe task the human contrast is 3.01 scale points and the models give 5.00, 1.34 and 4.70. On the smiley task it is 0.18 and the models give 0.01, 0.35 and 0.15. On the Mellers task it is 8.9 and the models give 2.6, −2.3 and −0.5. The median ratio of simulated to human within-condition SD is 0.03, 0.14 and 0.19 for the three models. Aligning items and scoring thus does not remove the collapse of variance or the divergence in effect size. The procedure is still not identical to the human one, since the model receives a participant-role instruction and answers in one text turn, and the human Mellers data do not record the order of the two vignettes.
  • How much the prompt matters. On the smiley task in GPT, the original materials give a contrast of 0.099 on a 0 to 1 scale and our two reconstructions give 0.013 and 0.091. A reconstruction can therefore change a simulated effect, which is why the paper runs two independent transcriptions of every experiment, averages them, and reports their agreement (0.68) as a measurement problem.
  • Dependence of the main result on reconstruction. The 55 units whose templates are both reconstructions give the same coefficient pattern as the full sample (0.46 [0.14, 0.78] for the bare-task effect and 0.93 [0.35, 1.51] for the replication estimate). Four tasks are known to differ from the human study in the endpoint or contrast that is scored, and excluding them leaves the two coefficients at 0.39 [0.16, 0.62] and 1.11 [0.40, 1.85].
  • Restricted claims. Three results do not rest on comparability with the human protocol, namely the associations across effects, the near-zero within-condition variance, and the condition-targeted refusal. Two do, namely comparisons of level with human effect sizes and direction agreement. The revision marks the second group as conditional on the adaptation and bases comparisons of level on the three tasks above.

Validating all 61 templates on human samples is outside the scope of this work, and the limitations state that the adapted texts were not validated on human participants. We agree that it is the right standard for any claim that a specific simulated effect matches a human one, and the revision makes no such claim for the reconstructed templates.

AuthorsOpenReview ↗

Author Response to Reviewer Kj3Y (2/2)

W1 and Q1. Asking an LLM a question without a persona doesn't isolate its default reasoning. Memorized papers and shared phrasing can easily bleed into both the bare-task and persona responses. Throwing more data at the model won't solve this underlying structural confound. How does the bare-task control distinguish the model's own biases from its knowledge of this well-known psychology literature?

TL;DR: The control does not distinguish the two, and the revision says so. Its purpose is narrower and does not depend on that distinction.

The bare-task control does not distinguish them. The revision says so, describes all three coefficients as conditional associations, and states that the source of the model's behaviour is not identified. What the decomposition separates is three measured quantities (the bare-task effect, the replication estimate and the gap), and the abstract and Section 3 now say this. We agree that more data of the same kind would not change it.

The purpose of the control is narrower. Simulations are usually validated by agreement between a simulated effect and a published effect. If the model produces the same effect with no participant to simulate, that agreement no longer shows successful simulation, whatever the origin of the effect. This holds whether or not the bare-task response reflects the literature. The part of the design aimed at literature-specific content is the gap between original and replication, and the paper's verdict on it is "inconclusive".

As a supplement, for each of the 38 effects in the estimation sample, three models (Claude Sonnet 5, Claude Sonnet 4.5, GPT-5.6-terra) were asked, outside any simulation, whether the task comes from a published study, what effect size they predict for human participants, and what the original study and any replication found (1,710 calls). The stated prediction was then added to the regression of the simulated effect. It is itself a behavioural measure and its prompt names the expected result, so this tests whether the bare-task effect is redundant with it and does not identify the source of either.

  • The premise holds. The models report recognising the task in 58 of 71 Claude units and 27 of 37 GPT units, and the replication effect sizes they recall correlate 0.73 with the true values. The limitations now state that a gap coefficient near zero cannot separate the absence of literature influence from the influence of a literature that contains the replications.
  • The bare-task effect is not redundant with the stated prediction in two of three models. Its coefficient is 0.51 (from 0.52) for Sonnet 5 and 0.62 (from 0.55) for GPT. For Sonnet 4.5 it falls from 0.39 to 0.07, and we make no claim for that model.
  • The replication coefficient overlaps with the stated prediction in the Claude models. It falls from 1.02 [0.42, 1.62] to 0.29 [−0.60, 1.17], and it is unchanged in GPT. This fits the statement in Section 3.2 that the coefficient "need not mean transfer".
  • Shared phrasing. The bare-task prompts were derived from the first of the two transcriptions of each experiment. When the simulated effect is taken from the second transcription, which was written independently and shares no wording with them, the bare-task coefficient is 0.45 [0.16, 0.74] for GPT, 0.47 [0.24, 0.70] for DeepSeek and 0.26 [−0.03, 0.55] for Claude (Table 3). Shared wording therefore does not produce it in two families, and it is uncertain in the third.

Thank you for the revision, and for the added analysis. Narrowing the claims on reconstructed templates makes sense. However, the new results still leave two things open for me.

  • On Q1, your result shows models recognizes most of the classic studies, and recalls their effect size accurately, and controlling for stated predictions wipes out Claude's replication coefficient. If the bare-task prompt is already recalling relevant papers, it's hard to see what the bare-task coefficient tells us beyond persona framing doesn't override recall.

  • On Q2: Dropping level comparisons for the reconstructed template makes sense. However, the two versions of the same task can vary drastically based on wording, and even original materials flip on Mellers. Given this task-dependent noise, it seems also possible that transcription distort the cross-effect regression?

I think the paper works better as a cautionary note on literature recall and prompt sensitivity in LLM benchmarks. Yet the headline result on decomposition of simulation behavior is less clear.

AuthorsOpenReview ↗

Author Response to Reviewer Kj3Y

We thank the reviewer for the close reading.

The paper's main result does not rest on separating recall from other sources. It is that agreement with published effects is a weak test of a simulator, shown with a bare-task control, replication anchors and a paired comparison that detects planted differences in three model families.

Q1. If the bare-task prompt is already recalling relevant papers, it is hard to see what the bare-task coefficient tells us beyond "persona framing doesn't override recall".

TL;DR: The control shows that agreement with human effects can arise with no participant being simulated, whatever the origin of the direct response. In GPT the bare-task coefficient is unchanged when the model's own recalled values and stated prediction are controlled.

The bare-task response may draw on recalled papers, and the control keeps its value in that case. Simulations are validated by agreement with human effects, and the coefficient shows that this agreement is predicted by what the model produces with no participant to simulate.

To see how much of the coefficient the model's explicit knowledge accounts for, we replaced the true original and replication estimates in the regression with the values each model recalls.

  • GPT. It recalls both values for 34 of 37 effects. On these the bare-task coefficient is 0.65 [0.37, 0.93] with the true estimates, 0.67 [0.40, 0.93] with the recalled ones, and 0.74 [0.49, 1.00] when its stated prediction is added as well. The bare-task effect thus carries an association beyond what the model explicitly reports about the literature.
  • Claude. The analysis with recalled values is possible only for Sonnet 5 on 20 effects, too few to be conclusive. For Sonnet 4.5 alone the interval of the coefficient includes zero before and after the stated prediction is added. For Sonnet 5 the coefficient with the stated prediction added is 0.51 [0.28, 0.74], and the pooled coefficient of the main analysis remains positive at 0.41 [0.16, 0.65].

The replication coefficient overlaps with the stated prediction in the Claude models and not in GPT (1.11 [0.19, 2.03] before and 1.06 [0.10, 2.02] after), and the paper reads it as an association that "need not mean transfer". Appendix M will report these results. The probes measure only what a model can report, so memory of the literature that it cannot report remains a possible explanation, as the first limitation states. The use of the control in validation does not depend on which explanation holds.

Q2. Given this task-dependent noise, could transcription distort the cross-effect regression?

TL;DR: Single units are sensitive to wording, as the reviewer says. The associations across effects hold when any one of the 38 effects is left out, so neither Mellers nor any other single task drives them.

  • Leaving out each effect in turn. The bare-task coefficient stays between 0.36 and 0.51 and the replication coefficient between 0.89 and 1.16, and the lower bound of every one of the 38 intervals is above zero for both. Without the four tasks scored differently from the human study, Mellers among them, the coefficients are 0.39 [0.16, 0.62] and 1.11 [0.40, 1.85]. The leave-one-out result will be added to Appendix G.
  • A bias in the same direction across tasks. This is the one case the check above does not address, for example if well-known effects are described more forcefully in the sources that both transcriptions draw on. The only comparison that bears on it is between units with and without primary-source text. The bare-task coefficient does not differ between them (difference −0.07 [−0.50, 0.36]), with only 16 units having such text.

These checks show that the two associations persist after omitting any single effect and after excluding the four tasks with documented scoring or endpoint mismatches. Transcription choices can still distort estimates across multiple tasks. The provenance comparison is too imprecise to rule out such distortion, so we interpret the slopes within the text-adapted battery.

Reviewer_B5KiOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether agreement with published psychological findings is a reliable way to validate simulated participants. It compares simulated effects with original estimates, replication estimates, and direct model responses without participant framing. The simulations track direct responses and replication estimates, while evidence about the original replication gap remains inconclusive. Further analyses reveal compressed response variance, selective refusal, and exaggerated correlations between familiar constructs.

Critical strengths

s1. The direct response control is a valuable contribution. If a model produces the same effect when answering as itself, successful participant simulation is not the only explanation. This is a simple control with important consequences for how simulation results should be interpreted.

s2. I appreciate that the authors test their own diagnostic method. The instructed copying simulator is not reliably identified when analyzed alone. Reporting that failure, rather than hiding it behind the main regression, makes the paper more credible. The distinction between individual assessment and paired comparison is useful.

s3. The variance and refusal analyses are practically important. Low response variance can inflate standardized effects, and selective refusal can remove one arm of an experiment. These are concrete failure modes that conventional agreement scores can miss.

Critical weaknesses

w1. The decomposition identifies conditional associations more clearly than distinct knowledge sources. Direct model responses may themselves reflect the literature, and they share stimulus content with the simulations. How much source separation is actually possible here? The paper acknowledges the issue, but its framing occasionally suggests a cleaner separation than the design can deliver.

w2. The positive control results limit the practical role of the main decomposition. Detecting a difference between two instructed simulators does not establish that the coefficients can diagnose one simulator reliably. I would center the methodological contribution more explicitly on the paired comparison, where there is clearer evidence of sensitivity.

w3. The direction prediction result is too imprecise to support a strong claim of chance performance. Sixteen agreements among twenty nine observations leave room for some predictive ability. The direct bias measurements are noisy as well. Does uncertainty in the threshold based grouping change the conclusion? Repeated effects across models should also be handled in the uncertainty estimate.

w4. The construct correlation analysis is suggestive, but literature influence is not the only plausible explanation. Familiar relationships may simply seem more intuitive to the model. Wording similarity controls cannot exclude that account. Different measurement instruments and concentrated missing correlations add uncertainty, especially since the least favorable missing value treatment removes the effect.

Questions for authors

See above comments. Overall, this is a good paper.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer B5Ki (1/2)

We thank the reviewer for the careful assessment and the positive feedback.

W1. The decomposition identifies conditional associations more clearly than distinct knowledge sources. Direct model responses may themselves reflect the literature, and they share stimulus content with the simulations. How much source separation is actually possible here?

TL;DR: We agree that the knowledge sources cannot be separated, and the revision says so. A new measurement shows that the direct response effect is not a restatement of what the model predicts for humans.

At the level of knowledge sources, none. The coefficients are conditional associations. What the decomposition separates is three measured quantities (the direct response effect, the replication estimate and the gap), and the revision says so in the abstract and in Section 3. It also adopts the reviewer's term "direct response effect".

In a new experiment, for each of the 38 effects in the estimation sample, three models (Claude Sonnet 5, Claude Sonnet 4.5, GPT-5.6-terra) were asked whether the task comes from a published study, what effect size they predict for human participants, and what the original study and any replication found (1,710 calls). The stated prediction was then added to the regression as a fourth regressor. It is itself a behavioural measure and its prompt names the expected result, so this tests whether the direct response effect is redundant with it and does not identify the source of either.

  • The models report knowing this literature. They report recognising the task in 58 of 71 Claude units and 27 of 37 GPT units, and the replication effect sizes they recall correlate 0.73 with the true values.
  • The direct response effect is not redundant with the stated prediction. Its coefficient goes from 0.48 [0.21, 0.75] to 0.41 [0.16, 0.65] for the Claude models and from 0.55 [0.25, 0.86] to 0.62 [0.29, 0.95] for GPT. Fitted per model, Sonnet 5 is unchanged (0.52 to 0.51), while Sonnet 4.5, whose single-model estimate is imprecise before the addition (0.39 [−0.22, 1.00]), falls to 0.07. In two of the three models the direct response effect thus predicts the simulated effect beyond the model's stated prediction, so asking a model what it expects of humans does not replace the control.
  • The replication coefficient overlaps with the stated prediction in the Claude models. It falls from 1.02 [0.42, 1.62] to 0.29 [−0.60, 1.17], and in GPT it is unchanged. This fits the statement in Section 3.2 that the coefficient "need not mean transfer".
  • Shared stimulus content. The direct response prompts derive from the first transcription of each experiment. With the simulated effect taken from the independently written second transcription, the coefficient is 0.45 [0.16, 0.74] for GPT, 0.47 [0.24, 0.70] for DeepSeek and 0.26 [−0.03, 0.55] for Claude (Table 3), so shared wording does not produce it in two families.

The argument summarised in S1 needs no assumption about sources. If an effect appears when the model answers as itself, agreement between the simulated and the human effect does not show that participants were simulated. In practice, a simulation should be credited with an effect only beyond what the direct response already produces, and the control makes that comparison possible wherever the direct response comes from.

W2. Detecting a difference between two instructed simulators does not establish that the coefficients can diagnose one simulator reliably. I would center the methodological contribution more explicitly on the paired comparison.

TL;DR: The contribution is now centered on the paired comparison, which detects the planted difference in all three model families.

Contribution (1) is now the paired comparison of simulators on a shared, replication-anchored protocol bank, and the single-simulator result is presented as the finding that motivates it. Both instructed simulators were rerun on GPT-5.6-terra and on DeepSeek V4.1, which replaces the retired V4 endpoint of the submission (9,120 calls).

  • The paired comparison detects the planted difference in every family. The slope of the within-unit difference on the gap is 1.16 [0.45, 1.87] for GPT, 1.19 [0.55, 1.82] for DeepSeek and 0.71 [0.11, 1.32] for Claude. Pooled over the three families it is 0.95 [0.37, 1.52] (146 units, 38 effects).
  • Single-simulator diagnosis fails in every family. Fitted alone, the simulator told to copy the original findings has a gap coefficient of 1.29 [−0.35, 2.93] in GPT and 1.24 [−0.36, 2.83] in DeepSeek.
  • Scope. The planted difference is an explicit instruction, and the gap accounts for 11% of the variance in the paired difference (15 to 19% in the new families). The design therefore detects a large difference between simulators across a set of effects, and the revision claims no more.
AuthorsOpenReview ↗

Author Response to Reviewer B5Ki (2/2)

W3. The direction prediction result is too imprecise to support a strong claim of chance performance. Does uncertainty in the threshold based grouping change the conclusion? Repeated effects across models should also be handled in the uncertainty estimate.

TL;DR: We remove the chance claim. The revision states only that recovery of the direction was not shown.

"Not distinguishable from chance" is removed. The count is 13 of 28 units from 13 effects. Of the other units, 8 have the opposite sign and 7 have a simulated effect of exactly zero.

  • Threshold grouping and repeated effects. Resampling whole effects, which keeps the repeated models of an effect together, and propagating measurement error in the direct response effect, which moves units across the threshold, gives 22% to 73%. Thresholds of 0.05, 0.10 and 0.20 give 42%, 46% and 52%.
  • All sources together. Allowing in addition for sampling error in the human estimates, which changes which effects count as replicated and with which sign, and for resampling of the simulated responses, gives 18% to 86%.

Uncertainty in the grouping therefore does change what can be concluded. The range includes one half. It also includes the 81 to 85% that a simulator faithful to the replication estimates would reach at our sample sizes, a benchmark that treats those estimates as true values. The data thus establish neither chance performance nor recovery. For reference, the sign of the original study agrees with the replication in 25 of the 28 units.

The revised statement is that, where the model shows no direct response effect, we found no evidence that simulation recovers the replicated direction. Section 6.1 now recommends that such use wait for evidence, and it no longer refers to the chance level.

W4. The construct correlation analysis is suggestive, but literature influence is not the only plausible explanation. Familiar relationships may simply seem more intuitive to the model. Wording similarity controls cannot exclude that account. Different measurement instruments and concentrated missing correlations add uncertainty.

TL;DR: We agree that the intuitiveness account is not excluded and relabel the finding. The effect remains after two mismeasured constructs are removed.

We agree that the wording control cannot exclude the intuitiveness account. The revision describes the finding as selective exaggeration of well-known relations, keeps it labelled exploratory, and names lay predictions of each correlation as the covariate needed to separate the two accounts.

On the measurement instruments, the paper explains that the item text of the Twin-2K-500 panel cannot be redistributed, so the simulated participants answered items that we assembled from public instruments for the same constructs. We checked each of these item sets against the instrument the panel actually used. For two constructs our items measure a different construct than the panel score, which affects four undiscussed pairs. Without them the fame effect is +0.137 (p = .027, 29 pairs), against +0.156 (p = .007, 33 pairs) in the submission, and the political-content effect stays near zero. The revision reports the human sample size of every pair (735 to 2,058) and retains the missing-value bounds, including the least favorable one under which the effect disappears.

Good resoponse! I will keep my score.

Thank you for kindly taking the time to respond to our rebuttal! We greatly appreciate your valuable comments and suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Remarks

Dear Area Chairs and Reviewers,

Contribution

Agreement with published effects is a weak test of an LLM simulator. Simulated effects covary with what the model produces with no participant to simulate, simulated samples have almost no within-condition variance, and a paired comparison on replication-anchored protocols detects a planted difference that scoring one simulator misses.

Revision Summary

We have responded to every review comment and revised the paper accordingly. Please refer to the updated PDF for the revised sections. Our revisions are summarized below.

B5Ki

  • W1 The abstract and Section 3.2 now state that the decomposition separates three measured quantities and not knowledge sources, and a new experiment compares the direct-response effect with the model's stated prediction (Section 5.5, Appendix M).
  • W2 Contribution (1) is now the paired comparison, which detects the planted difference in three model families (Section 5.4, Appendix I).
  • W3 The chance claim is removed, and a joint sensitivity analysis of direction agreement is reported (Section 5.2, Appendix G).
  • W4 The construct finding is relabeled as selective exaggeration of well-known relations, two mismeasured constructs are removed, and the human sample size of every pair is reported (Section 5.6, Table 11).

qYqm

  • W1 Both interpreted coefficients stay positive without the units at the ceiling (Section 5.3, Appendix F).
  • W2 The persona results are now in the main text, with within-group spread and between-group shifts (Section 5.2, Appendix G).
  • W3, Q1 Three tasks were run with the original items and scoring against public human data (Appendix N), and claims that depend on comparability are restricted to the text-adapted analogues (Section 4.1).
  • Q2 A temperature sweep shows that higher temperature does not restore a human-like spread (Appendix N, Table 10).

Kj3Y

  • W1, Q1 Section 3.2 now states that the control does not separate the model's own bias from literature knowledge and that its use in validation does not need this separation, and Appendix M reports what the models recognise and predict.
  • W2, Q2 Three tasks with original items and public human data are added (Appendix N), the main coefficients hold without the tasks scored differently from the human study (Appendix G), and level and direction claims are restricted to the text-adapted analogues (Sections 4.1 and 6.2).
  • Follow-up The bare-task coefficient in GPT remained positive when the model's own recalled values were controlled (Appendix M), and both associations persist when any single effect is left out (Appendix G).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.