- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether large language models can predict household economic behavior across different economic settings, including responses to policy-related changes. The authors introduce HouseholdBench, a benchmark combining six U.S. household surveys and 32 prediction tasks across five economic domains, covering numeric, categorical, and probabilistic outcomes. They evaluate 13 LLMs against a no-change baseline and XGBoost, finding that most LLMs outperform the no-change baseline on numeric and categorical tasks, while XGBoost achieves the best overall ranking. They further examine systematic prediction biases and show that fine-tuning and aggregating multiple predictions improve the performance of smaller open-weight models. The study contributes a benchmark for household economic prediction, empirical comparisons between LLMs and statistical models, and observations about prediction biases and adaptation methods.
Critical strengths
- Broad benchmark coverage: HouseholdBench integrates six household surveys and 32 tasks across five economic domains, enabling evaluation beyond individual datasets or economic outcomes.
- The study compares 13 LLMs against both no-change and XGBoost baselines using task-specific metrics, establishing reference points for assessing predictive performance.
- The paper examines systematic prediction biases, performance across knowledge cutoffs, and fine-tuning with prediction aggregation, providing insights beyond aggregate model rankings.
Critical weaknesses
- The policy-response tasks measure agreement with observed or stated household outcomes rather than causal treatment effects, limiting conclusions about the models' suitability for counterfactual policy simulation.
Questions for authors
see weaknesses
3 comments
Thank you again for your thoughtful review! As the discussion period ends on November 18, could you let us know whether our response has addressed your concerns? We are happy to clarify any remaining concerns.
Thank you for the response, which addresses my concern. I will keep my positive score.
We thank the reviewer for the review!
We agree. Our policy-response tasks compare predictions with survey-reported answers, which may differ from actual behavior. The revised conclusion now states this limitation: good performance on these tasks does not fully establish that a model can predict how households would respond to a policy in practice. We therefore do not claim to measure causal treatment effects, but predicting these answers is a necessary step toward policy simulation.