- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how different types of information about a person help LLMs predict their survey responses. It finds that previous behavioral history is more helpful than demographics, personality, and cognitive scores, though the improvement in accuracy is very small.
Critical strengths
- Using real respondents across two survey waves and comparing against human test-retest reliability is a good evaluation setup.
- Withholding previous questions measuring the target bias helps avoid direct answer leakage.
Critical weaknesses
- The claim that behavioral history helps is too broad for the scope of the experiments. Both the behaviroal history and the behaviors being predicted are from the same behavioral-economics survey battery.
- Although informedness improves, accuracy only increases from 56.8% to 56.9%, making the practical signficance of the improvement unclear.
- Only one model is evaluated, and there's no analysis on which parts of behavioral history actually helps prediction.
Questions for authors
- How well does behavioral history help predict behaviors in a different domain, rather than questions from the same survey?
- Given the very small improvement in accuracy, how should we interpret the practical benefit of behavioral history?
- Which parts of the behavioral history contributes most to the improvement?
7 comments
R3.2 — History and targets come from the same survey battery
"Both the behaviroal history and the behaviors being predicted are from the same behavioral-economics survey battery."
We thank the reviewer for raising this point, and we address it in two ways.
First, we clarify how the two waves and the history relate. The history and the targets come from two separate surveys fielded about three years apart (wave 1 in 2014–15, wave 2 in 2017). For each target bias, the whole block of wave-1 items used to score that bias is withheld, so the model never sees the respondent's earlier answers to the bias it is asked about. It sees only the respondent's answers to the other biases from three years ago and must infer the target behavior from them. The analysis of history variants also shows that the history's relevance matters: the half of the history least related to the target bias recovers only about 40% of the gain (R3.1; Table 4). We now state this explicitly in Section 2.2 (p. 3, lines 130–133):
"The whole item block used to score the target bias is removed, so the model never sees the respondent's earlier answers to that bias. The history thus comes from a separate survey fielded about three years earlier and covers only the other biases, from which the model must infer the target behavior."
Second, we tested prediction of a target outside the battery. We asked the LLM five wave-2 life-outcome questions: life satisfaction, financial stress, spending relative to income, being late on a bill, and assets relative to debts. Given A5's information, the LLM reaches a mean informedness of 0.248 [0.216, 0.280], or 61% of the human test–retest benchmark, against 28% on the biases. It is statistically indistinguishable from a supervised gradient-boosting model trained on the same information and about 675 labeled respondents per fold (0.256 [0.230, 0.281]; difference −0.008 [−0.033, 0.017]), and from logistic regression (0.272; difference −0.024 [−0.050, 0.002]). The LLM therefore predicts individuals well outside the battery, and its low informedness on the biases is specific to those targets.
We report these results in the Discussion (Section 4, p. 10, lines 496–506), with a new Table 7 comparing the LLM, the statistical model and the human benchmark on informedness and balanced accuracy for each outcome (intervals in Appendix O), and in a new Appendix O (p. 30, Table 24), which describes the design and compares the LLM with both statistical models. To make room, the decomposition of explained variation in bias counts (previously Table 7 and its paragraph) moved to Appendix N (p. 29), with a one-sentence summary added at the end of Section 3.3 (p. 10, lines 490–492). The Discussion now reads (p. 10, lines 496–506):
"We also tested A5 on targets other than the biases and compared its performance with that of a statistical model: five wave-2 life outcomes that appear in no wave-1 screen (life satisfaction, financial stress, spending relative to income, being late on a bill, and assets relative to debts; Table 7; Appendix O). Given A5's information, the LLM reaches a mean informedness of 0.248, or 61% of the human test–retest benchmark, against 28% on the biases, and it is statistically indistinguishable from a supervised gradient-boosting model given the same information (0.256; difference −0.008, 95% interval [−0.033, 0.017]). These results suggest that when labeled outcome data are scarce but a person's description and behavioral history are available, a zero-shot LLM persona can predict that person's outcomes as well as a statistical model trained on labeled data. This finding encourages using and testing behavioral history in other experimental settings, such as A/B tests, policy changes, advertising, and promotions."
In short, the history never contains the target bias's own items, and outside the battery the zero-shot LLM predicts life outcomes as well as supervised models trained on labeled data.
R3.3 — Practical significance given the small change in accuracy
"Although informedness improves, accuracy only increases from 56.8% to 56.9%, making the practical signficance of the improvement unclear."
We thank the reviewer for this comment, which points to a distinction we have now made clearer in the paper. Raw accuracy is not an informative measure here, because the bias outcomes are strongly imbalanced: their prevalence ranges from 0.17 to 0.85. A predictor that gives everyone the more common verdict can reach high accuracy while saying nothing about the individual. Informedness is the chance-corrected version of balanced accuracy (J = 2 × balanced accuracy − 1): it averages the accuracy on respondents with and without the bias, so it does not reward following the more common verdict (Section 2.3). We have rephrased the corresponding passage in Section 3.2 to make this point clearer (p. 6, lines 274–277):
"However, raw accuracy is misleading here because the outcomes are imbalanced, with bias prevalence ranging from 0.17 to 0.85: a model that gives every respondent the more common verdict can be highly accurate without distinguishing between individuals."
To make this concrete, we added balanced accuracy to Table 3 (p. 6). From A4 to A5, raw accuracy barely changes (0.568 to 0.569), but balanced accuracy rises from 0.517 to 0.539, and informedness more than doubles, from 0.033 to 0.078 (12% to 28% of the human test–retest benchmark). The imbalance between the two outcome classes also falls: |TPR − TNR| drops from 0.72 to 0.53. In other words, A4 mostly assigns the same verdict to everyone, while A5 begins to separate respondents who show a bias from those who do not.
We also established that this improvement is statistically robust and that it carries over to the other levels of the analysis:
- Uncertainty. A respondent-paired bootstrap gives an A5 − A4 gain of 0.045 [0.024, 0.065] in informedness, equivalent to 0.022 [0.012, 0.032] in balanced accuracy, with no bootstrap draw at or below zero. It remains 0.045 [0.019, 0.071] when generation randomness is simulated (Appendix F, pp. 21–22; Table 11 reports Table 3 with intervals for every cell).
- Between-person variation. Description-based conditions recover only 53–67% of the human between-person variation, whereas A5 restores about the full amount, in both normalized and absolute terms (Table 2, p. 5).
- Demographic groups. The A5 − A4 gain is positive in all 17 demographic groups, averaging 0.039 [0.017, 0.059] (Section 3.2, p. 8, lines 381–383; Table 5; Appendix F, Table 12).
- Attribution. The gain requires the respondent's own history: another respondent's history, even a demographically matched one, does not reproduce it (Table 4, p. 7; Appendix I).
In short, the small change in raw accuracy reflects class imbalance rather than a small effect: on balanced accuracy and informedness, behavioral history gives a clear, statistically robust improvement that also restores between-person variation and holds across demographic groups.
R3.1 — Scope of the claim that behavioral history helps
We thank the reviewer for this comment. To state more precisely when and why behavioral history helps, we ran four new variants of A5 on all 845 respondents, each with the same model, survey screens and scoring as A5, changing only the history (Table 4, p. 7; Appendix I, pp. 24–25):
These variants narrow the claim to what the evidence supports. The gain comes from the respondent's own history: a demographically matched respondent's history (J = 0.043) and a random respondent's history (J = 0.036) perform close to A4 (J = 0.033), and A5 exceeds them by 0.036 [0.013, 0.058] and 0.042 [0.019, 0.065]. More, and more closely related, history helps more: the less related half recovers about 40% of A5's gain (J = 0.051), and the full history adds significantly more (0.028 [0.006, 0.051]). History also needs the description: on its own it performs about as well as the description alone (J = 0.045), and A5, which combines the two, exceeds it by 0.034 [0.013, 0.055]. We summarize this in Section 3.2 (pp. 6–7, from line 314), which concludes (p. 7, lines 343–348):
The gain is also not specific to one model or predictor. With GPT-5.6, history raises informedness by a similar amount (0.051 against 0.045 for Claude; Appendix P), and supervised models given the same information gain from history as well (Appendix C).
In short, the claim the paper supports is specific: a respondent's own earlier answers, combined with a description of the person, improve zero-shot prediction of that person's later answers, and neither another person's history nor additional prompt content reproduces this gain.