- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper deals with the problem of using llm simulated participants for economic study. Traditionally directing asking results in bias. The paper mitigate the bias by sampling multiple simulated people with plausible relevant background and average, which improves prediction.
Critical strengths
- The bias in LLM's simulation is a important issue in applying it
- The predictions are checked with real-world datasets
- Plentiful ablations
Critical weaknesses
- Lack of diagrams in paper make reading harder
- The paper doesn't deal with simulation asking questions that don't have ground-truth answer, which are where in real life this method could be useful, and because non-existing in training data of LLM can lead to different behaviors.
- The scope of simulation is rather narrow
Questions for authors
- Previous works (such as https://arxiv.org/abs/2503.16527) say expanding demographcis to qualitative details like lifestyle introduce bias, which is opposite to this paper's conclusion. How would authors reconcile this.
3 comments
Response to Reviewer pnBu (2/2): W3, Q1
W3 (scope). The paper may seem to focus on the PSID and the lottery experiment, but it covers more than these two. What it covers falls into two areas, household decisions and attitudes. For household decisions, besides the PSID (United States, 2019 and 2023), we use the CHFS (China, 2011) and the Public Opinion Survey on Household Financial Behavior (Japan, 2019). For attitudes we use the GSS (United States, 2022), with 18 questions that include the 2020 presidential vote, abortion and the death penalty. We also run two experiments on human behaviour from the archive of Ashokkumar et al. (2026), and the simulated panel reproduces the direction of 32 of 36 contrasts in one of them. These results are in Section 5 and Appendix D, and the revised introduction now lists them.
Q1 (Li et al., 2025). You ask how our result can be reconciled with Li et al. (2025), who find that the more of a persona the LLM generates, the further the simulated results move from the real outcomes. The two studies use generated detail for different purposes.
In their study each simulated person receives one generated persona, and that persona then stands for the person. If the LLM writes it with a slant, the person carries the slant. In type sampling no single generated description stands for a household. For each survey record we write fifty types, which are different people that the same record could describe, take one answer from each, and combine the answers. Put another way, they condition on one generated description and we average over many. Their personas also start from age, sex, race and state, with the rest generated, while ours start from values the survey recorded, including income and net worth.
Their warning still applies to us in part. Averaging over fifty types limits the influence of any one description, but if the types as a whole lean one way, the average leans too. Type sampling reduces the holding-rate error on nine of ten questions, but the error for home ownership rises, and the error is larger for wealthy households. We have not run an experiment that separates these differences, so we cannot say which of them accounts for the contrast in results. The revised introduction adds this comparison.
Thank you for your response! I have decided to maintain my existing score.
Response to Reviewer pnBu (1/2): summary, W2
Thank you for reading the paper and for the review. We first summarize each answer.
W1. We agree that a diagram makes the method much clearer, and the revision adds one (Figure 1, on the second page).
W2. The method is meant for running experiments on simulated households and estimating effects from them, and the lottery experiment (Section 4) is one on a question without a ground-truth answer. Below we separate two kinds of such questions and say where the paper makes no claim.
W3. Beyond the PSID and the lottery experiment, the paper applies the method to household-finance surveys from China and Japan, to political and social attitudes in the General Social Survey, and to two published survey experiments (Section 5 and Appendix D).
Q1. Li et al. give each simulated person one generated persona, while we average over fifty for each survey record. Part of their warning applies to our results too.
W2 (questions without a ground-truth answer). We take this point to have three parts, whether the paper contains such simulations, whether an LLM can answer questions that are absent from its training data, and where the method is useful.
On the first, the paper does contain such an experiment, the lottery experiment of Section 4. No PSID household has a correct answer to it, because no survey records what these households would have done had they won a lottery. We describe each of 200 households in 2019 twice, once as it stands and once as having won, in 2009, a prize paying 150,000 dollars a year for twenty years, and the difference between its answers under the two descriptions is the simulated effect of the prize. Since there is no answer to check each household against, we compare the estimated effect with what the literature has estimated from real lottery winners (Imbens, Rubin and Sacerdote, 2001). In the LLM families that meet our three criteria most fully, the division of the prize among financial assets and housing is close to their estimates.
On the second, we think your concern is right, and it helped us state the boundary of what the paper claims. A question can lack a ground-truth answer in two ways. In the first, no data records the answer, but the behaviour behind it is contained in what the LLM has read. In the second, the situation itself is absent from anything the LLM has read, and there we agree with you that nothing guarantees the answers. The paper makes no claim about the second kind. The lottery experiment is of the first kind, since how households work and save after receiving money is something people have written about at length.
On the third, questions of the first kind are where we think the method is useful. Such a question is still not one the LLM can simply be asked. Type sampling is not meant as a prediction model that returns an answer. As in empirical economics, a quantity such as the effect of a prize on working hours has to be estimated from a treated and an untreated group, and with real households a clean experiment of this kind is rarely available. Running such experiments on simulated households is what the method is for.
Our approach is to start where the answer is known and move outward, because if simulated people cannot be shown to answer correctly where we can verify it, there is no reason to trust them where we cannot. Aher et al. (2023) and Ashokkumar et al. (2026) take another route. They simulate experiments directly and compare the simulated results with those of the human experiments, without first comparing answers respondent by respondent. We see building outward from a validated base as part of what the paper contributes.
For the lottery experiment, Section 4 sets criteria that can be tested without knowing the answer. The simulated winners must state that they won and how much they have received, their reported wealth must reflect the prize, and their other answers must agree with that wealth and with one another.
The shared note above also reports two new experiments, which test whether the households we describe as winners behave as a treated group.