LLMs as Research Participants: Households Built from Survey Records for Empirical Economics

Using large language models (LLMs) as research participants in economics requires each simulated person to decide as a real person with the same background would, but their answers have been shown to be biased relative to real people's, and neither the source of this bias nor a reliable correction is known. We propose type sampling, which does not correct the bias but instead unfolds the one answer that has to serve many people sharing the same characteristics into the answers of the people those characteristics could describe. For 500 households from the Panel Study of Income Dynamics and seven LLM families, mean AUC on ten asset-holding questions rises from 0.66 to 0.80, and for average amounts held by groups of similar households, mean log R-squared rises from −0.08 to 0.49 across the three LLMs tested. We then set out a procedure for experiments on simulated households, with a design that keeps the two conditions the same household and criteria that their responses must meet, and apply it to Imbens, Rubin, and Sacerdote's lottery study, which the simulation also extends to prizes of other sizes and payment timings that observed data cannot provide. In the families whose responses meet our three criteria most fully, working hours fall by less than in the original study, and the division of the prize among financial assets and housing is close to the original estimates.