acceptodds
Under review as a conference paper at ICLR 2027

Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas

Abstract

Large language models (LLMs) are increasingly used as synthetic personas representing survey respondents, to reduce the costs of survey research, market analysis, and policy experimentation. The validity of LLM-based synthetic personas as substitutes for particular respondents depends on whether they reproduce individuals' decisions. We examine what information helps synthetic respondents better predict each individual's later choices. We design experiments that evaluate synthetic respondents constructed with progressively richer information about each individual, beginning with no personal information, then adding demographics, personality traits, cognitive scores, and finally the respondents' earlier survey choices as behavioral history, yielding five conditions. We use a two-wave panel of 845 US adults who completed measures of 14 behavioral biases (spanning risk, time preferences, overconfidence, and reasoning), so each respondent's earlier answers provide a human test–retest benchmark. When earlier answers serve as behavioral history, all items that score the target bias are withheld. At the population level, the average number of biases per respondent in every condition is close to the human average (7.1–8.1 biases, against 7.1 for humans). This aggregate similarity, however, masks substantial differences in variance structure: persona descriptions recover only 53–67% of the amount of human between-person variation, whereas adding behavioral history restores the overall amount of variation to approximately the human level. At the individual level, predictive agreement is much weaker: description-based personas achieve only 7–12% of the informedness observed in human test–retest responses, while adding behavioral history raises this to 28%. Across demographic groups, the cumulative condition including behavioral history has the highest estimated informedness in all 17 groups, whereas description-based conditions provide little or no information for some groups. We also find that synthetic responses exhibit stronger education- and income-related differences than human responses. For LLM synthetic personas, a respondent's past answers add more to individual-level prediction than a description of who they are.

Primary area:
applications to neuroscience & cognitive science
Keywords:
synthetic personas, synthetic respondents, human behavior simulation, silicon samples, digital twins, persona prompting, behavioral economics, individual-level prediction, fairness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Tyv4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how different types of information about a person help LLMs predict their survey responses. It finds that previous behavioral history is more helpful than demographics, personality, and cognitive scores, though the improvement in accuracy is very small.

Critical strengths
  • Using real respondents across two survey waves and comparing against human test-retest reliability is a good evaluation setup.
  • Withholding previous questions measuring the target bias helps avoid direct answer leakage.
Critical weaknesses
  • The claim that behavioral history helps is too broad for the scope of the experiments. Both the behaviroal history and the behaviors being predicted are from the same behavioral-economics survey battery.
  • Although informedness improves, accuracy only increases from 56.8% to 56.9%, making the practical signficance of the improvement unclear.
  • Only one model is evaluated, and there's no analysis on which parts of behavioral history actually helps prediction.
Questions for authors
  • How well does behavioral history help predict behaviors in a different domain, rather than questions from the same survey?
  • Given the very small improvement in accuracy, how should we interpret the practical benefit of behavioral history?
  • Which parts of the behavioral history contributes most to the improvement?
7 comments
AuthorsOpenReview ↗

R3.1 — Scope of the claim that behavioral history helps

"The claim that behavioral history helps is too broad for the scope of the experiments."

We thank the reviewer for this comment. To state more precisely when and why behavioral history helps, we ran four new variants of A5 on all 845 respondents, each with the same model, survey screens and scoring as A5, changing only the history (Table 4, p. 7; Appendix I, pp. 24–25):

  • Partial behavioral history: the A5 prompt with only the half of the history least related to the target bias.
  • Demographically matched behavioral history: the respondent's full description and A5's format and instructions, but the history of the respondent most similar in gender, ethnicity, age, education and income.
  • Random behavioral history: the same, but with a randomly chosen respondent's history.
  • Only behavioral history: the respondent's own history, without demographics, personality or cognitive scores.

These variants narrow the claim to what the evidence supports. The gain comes from the respondent's own history: a demographically matched respondent's history (J = 0.043) and a random respondent's history (J = 0.036) perform close to A4 (J = 0.033), and A5 exceeds them by 0.036 [0.013, 0.058] and 0.042 [0.019, 0.065]. More, and more closely related, history helps more: the less related half recovers about 40% of A5's gain (J = 0.051), and the full history adds significantly more (0.028 [0.006, 0.051]). History also needs the description: on its own it performs about as well as the description alone (J = 0.045), and A5, which combines the two, exceeds it by 0.034 [0.013, 0.055]. We summarize this in Section 3.2 (pp. 6–7, from line 314), which concludes (p. 7, lines 343–348):

"The gain therefore comes from the person's own history, not from additional instructions, examples, or information unrelated to the person, such as another person's history, even that of a demographically similar person. […] neither history alone nor the description alone is sufficient, and the full gain requires both."

The gain is also not specific to one model or predictor. With GPT-5.6, history raises informedness by a similar amount (0.051 against 0.045 for Claude; Appendix P), and supervised models given the same information gain from history as well (Appendix C).

In short, the claim the paper supports is specific: a respondent's own earlier answers, combined with a description of the person, improve zero-shot prediction of that person's later answers, and neither another person's history nor additional prompt content reproduces this gain.

AuthorsOpenReview ↗

R3.2 — History and targets come from the same survey battery

"Both the behaviroal history and the behaviors being predicted are from the same behavioral-economics survey battery."

We thank the reviewer for raising this point, and we address it in two ways.

First, we clarify how the two waves and the history relate. The history and the targets come from two separate surveys fielded about three years apart (wave 1 in 2014–15, wave 2 in 2017). For each target bias, the whole block of wave-1 items used to score that bias is withheld, so the model never sees the respondent's earlier answers to the bias it is asked about. It sees only the respondent's answers to the other biases from three years ago and must infer the target behavior from them. The analysis of history variants also shows that the history's relevance matters: the half of the history least related to the target bias recovers only about 40% of the gain (R3.1; Table 4). We now state this explicitly in Section 2.2 (p. 3, lines 130–133):

"The whole item block used to score the target bias is removed, so the model never sees the respondent's earlier answers to that bias. The history thus comes from a separate survey fielded about three years earlier and covers only the other biases, from which the model must infer the target behavior."

Second, we tested prediction of a target outside the battery. We asked the LLM five wave-2 life-outcome questions: life satisfaction, financial stress, spending relative to income, being late on a bill, and assets relative to debts. Given A5's information, the LLM reaches a mean informedness of 0.248 [0.216, 0.280], or 61% of the human test–retest benchmark, against 28% on the biases. It is statistically indistinguishable from a supervised gradient-boosting model trained on the same information and about 675 labeled respondents per fold (0.256 [0.230, 0.281]; difference −0.008 [−0.033, 0.017]), and from logistic regression (0.272; difference −0.024 [−0.050, 0.002]). The LLM therefore predicts individuals well outside the battery, and its low informedness on the biases is specific to those targets.

We report these results in the Discussion (Section 4, p. 10, lines 496–506), with a new Table 7 comparing the LLM, the statistical model and the human benchmark on informedness and balanced accuracy for each outcome (intervals in Appendix O), and in a new Appendix O (p. 30, Table 24), which describes the design and compares the LLM with both statistical models. To make room, the decomposition of explained variation in bias counts (previously Table 7 and its paragraph) moved to Appendix N (p. 29), with a one-sentence summary added at the end of Section 3.3 (p. 10, lines 490–492). The Discussion now reads (p. 10, lines 496–506):

"We also tested A5 on targets other than the biases and compared its performance with that of a statistical model: five wave-2 life outcomes that appear in no wave-1 screen (life satisfaction, financial stress, spending relative to income, being late on a bill, and assets relative to debts; Table 7; Appendix O). Given A5's information, the LLM reaches a mean informedness of 0.248, or 61% of the human test–retest benchmark, against 28% on the biases, and it is statistically indistinguishable from a supervised gradient-boosting model given the same information (0.256; difference −0.008, 95% interval [−0.033, 0.017]). These results suggest that when labeled outcome data are scarce but a person's description and behavioral history are available, a zero-shot LLM persona can predict that person's outcomes as well as a statistical model trained on labeled data. This finding encourages using and testing behavioral history in other experimental settings, such as A/B tests, policy changes, advertising, and promotions."

In short, the history never contains the target bias's own items, and outside the battery the zero-shot LLM predicts life outcomes as well as supervised models trained on labeled data.

AuthorsOpenReview ↗

R3.3 — Practical significance given the small change in accuracy

"Although informedness improves, accuracy only increases from 56.8% to 56.9%, making the practical signficance of the improvement unclear."

We thank the reviewer for this comment, which points to a distinction we have now made clearer in the paper. Raw accuracy is not an informative measure here, because the bias outcomes are strongly imbalanced: their prevalence ranges from 0.17 to 0.85. A predictor that gives everyone the more common verdict can reach high accuracy while saying nothing about the individual. Informedness is the chance-corrected version of balanced accuracy (J = 2 × balanced accuracy − 1): it averages the accuracy on respondents with and without the bias, so it does not reward following the more common verdict (Section 2.3). We have rephrased the corresponding passage in Section 3.2 to make this point clearer (p. 6, lines 274–277):

"However, raw accuracy is misleading here because the outcomes are imbalanced, with bias prevalence ranging from 0.17 to 0.85: a model that gives every respondent the more common verdict can be highly accurate without distinguishing between individuals."

To make this concrete, we added balanced accuracy to Table 3 (p. 6). From A4 to A5, raw accuracy barely changes (0.568 to 0.569), but balanced accuracy rises from 0.517 to 0.539, and informedness more than doubles, from 0.033 to 0.078 (12% to 28% of the human test–retest benchmark). The imbalance between the two outcome classes also falls: |TPR − TNR| drops from 0.72 to 0.53. In other words, A4 mostly assigns the same verdict to everyone, while A5 begins to separate respondents who show a bias from those who do not.

We also established that this improvement is statistically robust and that it carries over to the other levels of the analysis:

  • Uncertainty. A respondent-paired bootstrap gives an A5 − A4 gain of 0.045 [0.024, 0.065] in informedness, equivalent to 0.022 [0.012, 0.032] in balanced accuracy, with no bootstrap draw at or below zero. It remains 0.045 [0.019, 0.071] when generation randomness is simulated (Appendix F, pp. 21–22; Table 11 reports Table 3 with intervals for every cell).
  • Between-person variation. Description-based conditions recover only 53–67% of the human between-person variation, whereas A5 restores about the full amount, in both normalized and absolute terms (Table 2, p. 5).
  • Demographic groups. The A5 − A4 gain is positive in all 17 demographic groups, averaging 0.039 [0.017, 0.059] (Section 3.2, p. 8, lines 381–383; Table 5; Appendix F, Table 12).
  • Attribution. The gain requires the respondent's own history: another respondent's history, even a demographically matched one, does not reproduce it (Table 4, p. 7; Appendix I).

In short, the small change in raw accuracy reflects class imbalance rather than a small effect: on balanced accuracy and informedness, behavioral history gives a clear, statistically robust improvement that also restores between-person variation and holds across demographic groups.

Reviewer_v7GeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates whether progressively richer persona information helps an LLM reproduce individual respondents' behavioral biases. Using a two-wave panel of 845 US adults and 14 text-administered biases, it compares five nested conditions for Claude Opus 4.8: no identity, demographics, personality, cognitive scores, and behavioral history added to the complete description. Wave-2 responses are the targets; descriptive fields also come from wave 2, while behavioral history comes from wave 1. The entire earlier item block scoring the target bias is withheld. Description-based conditions achieve mean informedness of 0.020–0.033, compared with 0.283 for human test–retest agreement; adding history raises this to 0.078. Similar average bias counts therefore conceal poor individual fidelity. Variance decomposition, residual correlations, identification, continuous-score analyses, and demographic comparisons further show that history improves some aspects of simulation without reproducing human profiles or demographic relationships. The contribution is a controlled comparison of incremental information blocks and a multidimensional assessment of the gap between aggregate resemblance and individual prediction.

Critical strengths
  1. The evaluation directly tests individual correspondence and avoids several misleading success criteria. The two-wave panel permits matching each synthetic response to the intended person's wave-2 response and comparing it with a human retest benchmark. The paper reports chance-corrected informedness rather than interpreting raw accuracy as individual fidelity, withholds the entire earlier target-bias block, and corroborates the result using residual correlations and identification AUC. For example, A5 achieves informedness of only 0.078 against the human value of 0.283, despite matching the human mean bias count in the count sample. This is consequential evidence against treating aggregate similarity as sufficient validation of individual substitutes.

  2. The empirical diagnosis extends beyond a single favorable metric. The paper reports persistent differences in prevalence, between-bias relationships, and demographic associations even when history improves prediction. Appendix F removes a mechanically confounded bias pair from the correlation-structure analysis, and Appendices G–I examine alternative samples, continuous scores, and respondent-level gains. The demographic-stratified output-reassignment test reduces A5 informedness from 0.078 to approximately 0.026, supporting some respondent-linked signal. These checks make the qualified conclusion substantially more useful than a simple claim that richer personas perform better.

Critical weaknesses
  1. The absence of conventional predictive baselines is a substantial gap in the evaluation. The inputs are largely structured demographic fields, personality and cognitive scores, and earlier survey answers, making regularized logistic regression and a nonlinear predictor such as gradient-boosted trees natural comparators. Yet the paper reports no such predictors of the same wave-2 outcomes with the same A2–A5 information sets and withholding rules. The classifier distinguishing synthetic from human profiles and the regressions explaining bias counts answer different questions. The human retest reference also uses the earlier target-bias outcome, which A5 is intentionally denied, so it does not supply a comparison using the same inputs.

    Consequently, the reader cannot tell whether A5's informedness of 0.078 reflects limited predictability from the available inputs or ineffective use of informative records by the LLM. Nor can the reader assess whether the observed information ladder is also obtained, or substantially exceeded, by simple statistical prediction. This matters directly to the interpretation and practical significance of the central finding, particularly since the Introduction itself discusses prior work with competitive statistical baselines. Comparators across the same information ladder would establish both absolute performance and the incremental value of history. Supervised baselines would use other respondents' labeled outcomes, so that additional supervision must be disclosed when interpreting the comparison. Such results would contextualize the LLM experiment; the current within-LLM improvement alone does not resolve these questions.

  2. The attribution and stability of the history-related gains remain insufficiently established. A5 adds a larger block of task-related examples and extra instructions about using earlier answers, after all descriptive information is already present. There is no history-only condition or rerun with a matched respondent's history while retaining the target respondent's description and the history format and instructions. Permuting completed A5 outputs demonstrates respondent correspondence but does not isolate the history input from the rest of the persona. The authors appropriately acknowledge the sequential interpretation; the evidence therefore concerns history's incremental value in this protocol and does not establish its independent superiority.

    Moreover, each respondent–item–condition combination is generated once, so stable respondent effects cannot be separated from generation randomness. This directly limits the claim that A5 restores human between-person variation. Table 2 compares components normalized by each condition's total sum of squares; corresponding absolute components would clarify the amount of variation recovered. Several appendix analyses report bootstrap intervals, but the primary A5–A4 informedness contrast and the 17 subgroup rankings are presented mainly as point estimates. These limitations leave the reliability and demographic consistency of the central gain less established than the reported rankings suggest.

Questions for authors
  1. Additional evidence essential to the current assessment: conventional prediction baselines. How do regularized logistic regression and a nonlinear tabular predictor perform on the same wave-2 outcomes using the A2–A5 information blocks? Please preserve the target-block and cognitive-score withholding rules, split by respondent, and fit preprocessing, hyperparameters, and decision thresholds using only training/validation data. Report informedness and uncertainty for both absolute performance and the A5–A4 increment on the same evaluation samples. Since supervised baselines use labeled outcomes from other respondents, please distinguish that supervision from the LLM prompting setup. Do these comparisons change the interpretation of the low absolute performance or the incremental benefit of history?

  2. Additional evidence relevant to the current assessment: history attribution. How does A5 compare with a condition that keeps the target respondent's full description and the same history instructions and format, but supplies a demographically matched person's history? A history-only condition would additionally clarify the role of the descriptive blocks. The existing output-reassignment test is useful, but it does not answer this input-level question.

  3. Additional evidence relevant to the current assessment: reliability and uncertainty. Can the authors report respondent-paired bootstrap intervals for the mean A5–A4 informedness difference and assess repeated generations on a representative subset? Please also report absolute between-person variance components alongside the normalized shares in Table 2. These results would affect whether I interpret A5 as recovering stable individual variation or mainly restoring overall dispersion.

A broader comparison across model families and information orderings would improve external validity, but I regard that expansion as future work rather than a prerequisite for the bounded empirical contribution. I would consider raising my score if the authors provide a comprehensive comparison with appropriate predictive baselines using the same input information and withholding rules, and convincing history-attribution controls that isolate the contribution of the target respondent's own behavioral history. If these additional results adequately address the concerns above and support the paper's conclusions, they would materially strengthen the case for acceptance.

10 comments
AuthorsOpenReview ↗

R2.1 — Conventional predictive baselines

"The absence of conventional predictive baselines is a substantial gap in the evaluation. The inputs are largely structured demographic fields, personality and cognitive scores, and earlier survey answers, making regularized logistic regression and a nonlinear predictor such as gradient-boosted trees natural comparators. Yet the paper reports no such predictors of the same wave-2 outcomes with the same A2–A5 information sets and withholding rules."

We thank the reviewer for this suggestion. We trained two statistical models, an elastic-net logistic regression and a gradient-boosted tree classifier. Each was fit to the same information as conditions A2–A5, under the same withholding rules (details in the new Appendix C, pp. 16–18).

The statistical models predict the wave-2 verdicts better than the LLM: mean informedness is 0.189 for gradient boosting with A5 information, against 0.078 for the LLM. However, these models learn from other respondents' labeled answers. They are therefore closer to a few-shot setting, in which a model is shown a subsample's responses and then asked about a new respondent, than to our zero-shot approach. This is now stated in the main text (Section 2.2, p. 3, lines 153–157):

"We study zero-shot prediction and therefore use neither few-shot prompting nor fine-tuning, which can match a human response distribution by imitating its training examples (Gao et al., 2025) without recovering how individuals deviate from each question's average answer (Ahn et al., 2026); supervised models trained on other respondents' labeled answers serve as a reference (Appendix C)."

We added the gradient-boosting results as a new column in Table 3 ("Stat model (A5 information)", p. 6) for comparison, together with a new balanced-accuracy row. We also report the effect of behavioral history for both the statistical model and our zero-shot LLM (Section 3.2, p. 6, lines 281–283):

"In relative terms, history adds far less to a supervised gradient-boosting model given the same information: its informedness rises by 24% from A4- to A5-equivalent information (J = 0.153 to 0.189), against 134% for the zero-shot LLM (J = 0.033 to 0.078; Appendix C)."

In short, models trained on labeled examples extract more individual signal from the same information, but behavioral history is the input that adds most to the zero-shot LLM's predictions.

AuthorsOpenReview ↗

R2.2 — Human retest reference uses different inputs and R2.3 — Limited predictability or ineffective use of the records?

R2.2 — Human retest reference uses different inputs

"The human retest reference also uses the earlier target-bias outcome, which A5 is intentionally denied, so it does not supply a comparison using the same inputs."

We agree, and we clarify that the human retest reference is not intended as a same-input comparison. Its role is to measure internal consistency: about three years separate the two waves, and people can change their minds over that period. On average, a person's wave-1 verdict matches their own wave-2 verdict only about 70% of the time (mean accuracy 0.709, informedness 0.283; Table 3). The retest reference therefore shows how consistent a person's own decisions are over this three-year interval, and gives a practical reference point for the individual-level fidelity a synthetic respondent could reasonably be expected to reach. We have made this role explicit in the introduction (Section 1, p. 2, lines 94–97):

"We evaluate the conditions at the population and individual levels, comparing each synthetic response with the target person's later answer and using the person's own earlier answer as a human test–retest benchmark, a reference point for how consistently the same person decides over the three years between waves."

R2.3 — Limited predictability or ineffective use of the records?

"Consequently, the reader cannot tell whether A5's informedness of 0.078 reflects limited predictability from the available inputs or ineffective use of informative records by the LLM."

We thank the reviewer for this question, which the statistical baselines added in response to R2.1 now answer. Given the same information as A5, the gradient-boosting model reaches informedness of 0.189, and with demographics alone it already exceeds A5 (0.114 against 0.078). The inputs are therefore informative, and A5's low informedness reflects how the zero-shot LLM uses the record, not a lack of predictable signal. Behavioral history still improves the zero-shot LLM, even though the descriptive fields already carry signal it could use. We have added this comparison at the end of the paragraph that opens with "Informedness reveals much larger differences" (Section 3.2, p. 6, lines 283–288):

"Comparing A5 with the statistical model in Table 3 shows that the inputs are informative: with the same information the statistical model reaches J = 0.189, and with demographics alone it already exceeds A5 (J = 0.114; Appendix C). A5's low informedness therefore reflects how the zero-shot LLM uses the record, not a lack of signal in it. Behavioral history still improves the zero-shot LLM, even though the descriptive fields already carry signal it could use."

In short, the information is there but the zero-shot LLM uses it inefficiently, and behavioral history is the input that helps it most.

AuthorsOpenReview ↗

R2.4 — Is the information ladder reproduced or exceeded by statistical prediction?

"Nor can the reader assess whether the observed information ladder is also obtained, or substantially exceeded, by simple statistical prediction. This matters directly to the interpretation and practical significance of the central finding, particularly since the Introduction itself discusses prior work with competitive statistical baselines. Comparators across the same information ladder would establish both absolute performance and the incremental value of history. Supervised baselines would use other respondents' labeled outcomes, so that additional supervision must be disclosed when interpreting the comparison. Such results would contextualize the LLM experiment; the current within-LLM improvement alone does not resolve these questions."

We thank the reviewer for this point. Using the statistical baselines added in response to R2.1, we compared the LLM with an elastic-net logistic regression and a gradient-boosting model at every step of the information ladder, with the same information sets and withholding rules. Both models reproduce the ladder, improving at every step in the same order as the LLM, and behavioral history adds significantly for all three predictors: 0.045 [0.024, 0.065] for the LLM, 0.037 [0.020, 0.051] for gradient boosting and 0.026 [0.012, 0.040] for logistic regression, with no significant difference between them. The level, however, is substantially exceeded: the statistical models reach two to six times the LLM's informedness at every step, and with demographics alone they already exceed A5. The information is therefore present, and the zero-shot LLM uses it less efficiently, although, relative to its starting point, the gain from history is far larger for the LLM (134%, against 24% for gradient boosting). Because these models learn from about 675 labeled respondents per training fold, whereas the LLM receives no labeled examples, they serve as a reference for what the information can support rather than as a like-for-like competitor.

We added the gradient-boosting model with A5 information as a new column in Table 3 (p. 6), whose caption discloses its supervision:

"The last column is a supervised gradient-boosting model trained on other respondents' wave-2 verdicts with the same information as A5 (Appendix C)."

We report the comparison in Section 3.2 (p. 6, lines 281–288):

"In relative terms, history adds far less to a supervised gradient-boosting model given the same information: its informedness rises by 24% from A4- to A5-equivalent information (J = 0.153 to 0.189), against 134% for the zero-shot LLM (J = 0.033 to 0.078; Appendix C). Comparing A5 with the statistical model in Table 3 shows that the inputs are informative: with the same information the statistical model reaches J = 0.189, and with demographics alone it already exceeds A5 (J = 0.114; Appendix C). A5's low informedness therefore reflects how the zero-shot LLM uses the record, not a lack of signal in it. Behavioral history still improves the zero-shot LLM, even though the descriptive fields already carry signal it could use."

We also state the difference in setting in Section 2.2 (p. 3, lines 153–157):

"We study zero-shot prediction and therefore use neither few-shot prompting nor fine-tuning, which can match a human response distribution by imitating its training examples (Gao et al., 2025) without recovering how individuals deviate from each question's average answer (Ahn et al., 2026); supervised models trained on other respondents' labeled answers serve as a reference (Appendix C)."

In short, simple statistical prediction reproduces the information ladder, including the gain from behavioral history, and substantially exceeds the LLM's level, which shows the shortfall lies in the zero-shot LLM's use of the information rather than in the information itself.

Reviewer_SFiiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an analysis of digital twin prompting for more realistic simulations of individuals. The authors show that behavioral history restores the majority of alignment with human behavior.

Critical strengths

This is an interesting insight that using behavioral history makes digital twins more personalized. This is a critical component to have it generalize to other questions. The experiments are well done and relatively clear to reproduce. They clearly fill in the gap Anh et al, 2026 left for behavioral history.

Critical weaknesses

Overall, I think the contribution of the paper is relatively limited. The insight is interesting and the experiments are well executed, but I'd want to see more scenarios (e.g., Gao et al, "Take caution in using LLMs as human surrogates"). I would want to see if this is equivalent for fine-tuning on responses.

Questions for authors

Are the responses better than the co-variance between items? Can you correct for this? In which cases is creating person-level variance really improving simulations? When is this actually important to do? What happens when you add irrelevant historical items?

6 comments
AuthorsOpenReview ↗

R1.1 — Contribution, additional scenarios, and fine-tuning - First part

We thank the reviewer for finding the experiments well executed and for noting that the paper fills the gap that Ahn et al. (2026) left for behavioral history. We take the concern about the size of the contribution seriously. For the revision, we ran new experiments that test the main finding against new baselines, new controls, a new model and new targets, and we revised the text to position the paper more clearly within the literature, including the two studies the reviewer refers to, and to explain why we study zero-shot prediction rather than fine-tuning.

We have now run the following experiments and added them to the paper. To test whether the information we give the LLM is informative in itself, we trained two supervised models, an elastic-net logistic regression and gradient boosting, on the same information as conditions A2–A5 and under the same withholding rules (new Appendix C, pp. 16–18, Tables 9 and 10, and a new column in Table 3, p. 6). With A5's information they reach an informedness of 0.189, so the LLM (0.078) uses about 40% of the available signal, but they gain about as much from behavioral history as the LLM does (difference −0.008 [−0.034, 0.017]). To identify what in the history carries the signal, we ran four new variants of A5 on all 845 respondents, each changing only the history: partial history, the history of a demographically matched respondent, the history of a random respondent, and the person's own history without the description (new Table 4, p. 7; new Appendix I, pp. 24–25). Another respondent's history, whether random (J = 0.036) or demographically matched (J = 0.043), adds no significant gain over the description alone (J = 0.033), whereas the person's own history raises informedness to 0.078; the half of the history least related to the target recovers about 40% of the gain, and history without the description is no more informative than the description alone. To test whether the results depend on the model, we repeated A1–A5 with GPT-5.6 (new Appendix P, pp. 30–31): the history gain is 0.051, against 0.045 for Claude Opus 4.8, and the difference of 0.006 [−0.023, 0.032] is not significant. To test targets outside the bias battery, we asked the LLM five wave-2 life-outcome questions with A5's information (Discussion, p. 10, lines 496–506; new Table 7; new Appendix O, p. 30): its mean informedness of 0.248 is statistically indistinguishable from that of gradient boosting trained on labeled responses (0.256; difference −0.008 [−0.033, 0.017]). Finally, we added respondent-bootstrap intervals for every main result, including a simulation of generation randomness (new Appendix F, pp. 21–22); the history gain of 0.045 has a 95% interval of [0.024, 0.065].

These additions make clearer where the paper stands in the literature. LLMs can reproduce population-level patterns, such as subgroup opinion distributions, classic experimental effects and aggregate economic behavior (Argyle et al., 2023; Aher et al., 2023; Filippas et al., 2024; Hu et al., 2026). Persona prompting, however, compresses human variation, distorts demographic relationships and is sensitive to prompt artifacts (Bisbee et al., 2024; Park et al., 2024b; Wang et al., 2025; Gao et al., 2025). Digital twins built from interviews or survey records (Park et al., 2024a; Toubia et al., 2025) look accurate for individuals mainly because they track each question's average answer, and simple statistical baselines often match or beat them (Peng et al., 2026; Ahn et al., 2026; Chen et al., 2026). Ahn et al. (2026) trace this to persona prompts, which summarize stable attributes and cannot supply how a particular person answers a particular question. Fine-tuning does not close this gap in their tests (de-meaned R² of 2.31% for fine-tuned GPT-4.1, against 4.44% for the best prompt), and they state that they "do not test systems that retrieve a respondent's full response history at prediction time." What this literature leaves open is how much a person's own past behavior adds to a description of the person when the target answers are never shown, whether that gain is specific to the person, and how it compares with how predictable the person is. Our paper addresses this. On a two-wave panel in which every synthetic answer can be compared with the same person's later answer, we give the model, zero-shot and at prediction time, the person's own earlier answers to other questions, so the model must infer the target behavior from how the person answered different questions. We then measure how much person-specific signal this history adds beyond a description of the person.

AuthorsOpenReview ↗

R1.1 — Contribution, additional scenarios, and fine-tuning- Second part

Taken together, the paper contributes to this literature in four respects. The design isolates the value of each type of information: where existing digital twins combine descriptions and personal records in a single prompt, we hold the model, respondents, target questions and scoring fixed and add demographics, personality, cognitive scores and behavioral history in turn (Table 1, p. 3), with the person's own earlier answers as a human test–retest benchmark (J = 0.283) for how predictable each person is. The attribution, through the new history variants, shows that the gain requires the person's own history. Because another person's history in the same format gives no gain, the improvement does not come from imitating examples or from the format of the prompt, the failure modes documented by Gao et al. (2025); it comes from inferring the person's tendencies from their behavior on other questions. This is also a stronger form of the persona-swap control of Ahn et al. (2026), because we replace the history in the prompt and rerun the model, rather than re-pairing existing predictions with other respondents. The evaluation separates population fit from individual fit and quantifies the gap that the first two lines of work describe: the average bias count is close to the human average in every condition (7.1–8.1 against 7.1), yet descriptions recover only 53–67% of human between-person variation and 7–12% of the human informedness, against 28% with behavioral history, and raw accuracy hides this difference (0.550 to 0.569, while informedness rises from 0.001 to 0.078; Tables 2 and 3, pp. 5–6). Finally, because the third line of work finds that simple statistical baselines often do better than LLM twins, we compare the LLM with statistical baselines given the same information: the zero-shot LLM uses only part of the signal available for the biases, but it benefits from behavioral history as much as a trained model does, and it matches trained models on outcomes outside the battery.

On scenarios, where Gao et al. (2025) vary how a model is set up for a single game, our study now spans 14 behavioral biases in six domains (present bias, inconsistent choice, risk, overconfidence, mathematical and statistical reasoning, and attention), each with person-level ground truth, five life outcomes outside the battery, two model families, and nine ways of conditioning the model (A1–A5 and the four history variants). We did not vary the prompt language or role framing, and we did not test open-weight models; we see these as natural extensions.

On fine-tuning, our study is zero-shot by design, and we now state this explicitly in Section 2.2: the prompt contains no labeled example of the target, from the respondent or anyone else, and the model is not fine-tuned. This is the setting in which synthetic personas are meant to be useful, when labeled responses from the target population are scarce; as Gao et al. note, fine-tuning on human responses "necessitates that one already possesses diverse, high-quality, and representative data." Both cited studies tested fine-tuning. Ahn et al. find that fine-tuned GPT-4.1 does worse than prompting at recovering individual deviations, and that a model fine-tuned on a large corpus of survey studies reaches 7.57% on studies it was trained on but only 0.73% on held-out studies. Gao et al. attribute the fine-tuned GPT-4o's match to imitation of its training examples. Fine-tuning on other people's responses thus improves item-level calibration but not the prediction of a particular person, which is what our study targets. The closest comparison with learning from labeled responses that our data allow is the new supervised baselines, which learn from about 675 labeled respondents per training fold; we state in Section 2.2 that we study zero-shot prediction and therefore use neither few-shot prompting nor fine-tuning.

We also made three changes to the text. In the Introduction, we now engage Ahn et al. directly and state the question they leave open, which our study addresses (Section 1, p. 2, lines 73–77):

"In particular, Ahn et al. (2026) show that persona prompts mostly reproduce each question's average answer, and they leave untested whether a person's response history, given at prediction time, supplies the missing person-specific signal. More broadly, it remains unresolved which kind of information about a person (demographics, personality, cognitive scores, or past answers) best helps a model reproduce that person's responses, and whether the model also captures how people differ from one another."

In Section 2.2, we define the zero-shot setting (p. 3, lines 133–134):

"All conditions are zero-shot: the prompt contains no labeled example of the target, from this respondent or anyone else, and the model is not fine-tuned."

AuthorsOpenReview ↗

R1.1 — Contribution, additional scenarios, and fine-tuning- Third part

and we relate our choice not to fine-tune to the findings of the two cited studies (p. 3, lines 153–157):

"We study zero-shot prediction and therefore use neither few-shot prompting nor fine-tuning, which can match a human response distribution by imitating its training examples (Gao et al., 2025) without recovering how individuals deviate from each question's average answer (Ahn et al., 2026); supervised models trained on other respondents' labeled answers serve as a reference (Appendix C)."

In short, the paper addresses a question that the literature on LLM digital twins, including Ahn et al. (2026), leaves open, under controls that rule out the imitation effects documented by Gao et al. (2025): a person's own behavioral history, given zero-shot at prediction time, adds person-specific signal that descriptions do not, the result replicates across two model families, and outside the battery the zero-shot LLM matches supervised models trained on labeled responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.