acceptodds
Under review as a conference paper at ICLR 2027

HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior

Abstract

Large language models (LLMs) have the potential to meet a key goal in economics: a quantitative model of household decision making, across a variety of settings. Yet existing evaluations cover few surveys and outcomes, and do not study how households adjust to changing economic conditions. We introduce a new evaluation, HouseholdBench, which unites 6 U.S. household surveys and 32 prediction tasks spanning numeric, categorical and probabilistic outcomes, related to consumption, income, labor, expectations, and housing. Using past behavior, demographics and macroeconomic conditions, the tasks test whether LLMs predict behavior, including how households adjust to changes in various policies. We evaluate 13 proprietary and open-weight LLMs against a no-change baseline and a gradient-boosted tree model. Most LLMs outperform the no-change baseline, including for policy response tasks—with the best model lowering error for numeric outcomes by 12.2%. Across most tasks, gradient-boosted trees rank first; leading proprietary LLMs approach their performance, but open-weight models lag. LLMs exhibit systematic over- and underprediction across different tasks. We identify methods that enable a 4 billion parameter open-weight model to match proprietary models' performance: fine-tuning and aggregating 16 predictions per observation. Improvements generalize to policy-response tasks, which are excluded from fine-tuning.

Primary area:
datasets and benchmarks
Keywords:
benchmark, large language models, household economic behavior

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mtpLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether large language models can predict household economic behavior across different economic settings, including responses to policy-related changes. The authors introduce HouseholdBench, a benchmark combining six U.S. household surveys and 32 prediction tasks across five economic domains, covering numeric, categorical, and probabilistic outcomes. They evaluate 13 LLMs against a no-change baseline and XGBoost, finding that most LLMs outperform the no-change baseline on numeric and categorical tasks, while XGBoost achieves the best overall ranking. They further examine systematic prediction biases and show that fine-tuning and aggregating multiple predictions improve the performance of smaller open-weight models. The study contributes a benchmark for household economic prediction, empirical comparisons between LLMs and statistical models, and observations about prediction biases and adaptation methods.

Critical strengths
  1. Broad benchmark coverage: HouseholdBench integrates six household surveys and 32 tasks across five economic domains, enabling evaluation beyond individual datasets or economic outcomes.
  2. The study compares 13 LLMs against both no-change and XGBoost baselines using task-specific metrics, establishing reference points for assessing predictive performance.
  3. The paper examines systematic prediction biases, performance across knowledge cutoffs, and fine-tuning with prediction aggregation, providing insights beyond aggregate model rankings.
Critical weaknesses
  1. The policy-response tasks measure agreement with observed or stated household outcomes rather than causal treatment effects, limiting conclusions about the models' suitability for counterfactual policy simulation.
Questions for authors

see weaknesses

3 comments

We thank the reviewer for the review!

"The policy-response tasks measure agreement with observed or stated household outcomes rather than causal treatment effects, limiting conclusions about the models’ suitability for counterfactual policy simulation."

We agree. Our policy-response tasks compare predictions with survey-reported answers, which may differ from actual behavior. The revised conclusion now states this limitation: good performance on these tasks does not fully establish that a model can predict how households would respond to a policy in practice. We therefore do not claim to measure causal treatment effects, but predicting these answers is a necessary step toward policy simulation.

Thank you again for your thoughtful review! As the discussion period ends on November 18, could you let us know whether our response has addressed your concerns? We are happy to clarify any remaining concerns.

Thank you for the response, which addresses my concern. I will keep my positive score.

Reviewer_VwDpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HouseholdBench, a benchmark constructed from six public U.S. household surveys, comprising 32 tasks across consumption, income, labor, macroeconomic expectations, and housing. The tasks cover numerical outcomes, categorical decisions, and reported subjective probabilities, and are organized into 18 baseline tasks and 14 policy-response tasks. The authors evaluate 13 LLMs alongside a no-change baseline and task-specific XGBoost models, and further investigate fine-tuning and prediction aggregation.

The main contribution is the curation and harmonization of established household surveys into a broad evaluation resource. The experiments reveal useful differences across models and task types, including difficulties with persistent beliefs and uneven benefits from fine-tuning. Overall, this is a valuable benchmark contribution.

Critical strengths
  1. Established data sources and substantial curation effort. The benchmark draws on six well-established household surveys with documented provenance and survey methodology. Appendices B and H describe identifier harmonization, longitudinal linking, missing-value handling, task-specific filtering, and prompt construction. These details provide a solid foundation for understanding and inspecting the dataset construction process.

  2. Broad and economically meaningful task coverage. The 32 tasks cover five important areas of household behavior and include both observed outcomes and reported beliefs. Bringing these heterogeneous sources into a common evaluation framework is a useful contribution in its own right, enabling comparisons that would otherwise require substantial survey-specific preparation.

  3. Useful reference evaluations and diagnostic findings. The comparison of 13 LLMs with persistence and XGBoost baselines helps users interpret benchmark performance. The analyses of majority-class predictions, belief persistence, and fine-tuning provide informative examples of the benchmark’s diagnostic value. The paper also reports limitations of LLM performance rather than presenting uniformly positive results.

Critical weaknesses
  1. The effects of data filtering on coverage are not sufficiently summarized. The appendices describe extensive quality controls, including missing-value exclusions, eligibility restrictions, and outlier trimming. These choices are reasonable, but readers have limited visibility into how they change the composition of the retained samples. A compact summary of retention rates and key demographic or outcome distributions would help establish which household populations the benchmark represents.

  2. The relationship between data splits across related tasks needs clarification. The quarter-based splitting procedure is described, but it is less clear whether splits are coordinated across tasks derived from the same survey and how repeated households are handled. Since some baseline and policy tasks use related records and outcomes, this information is important for interpreting transfer results and ensuring that future benchmark users apply the intended evaluation protocol.

  3. The scope of the policy-response category could be more precisely defined. This category combines realized outcomes following economic events, responses to hypothetical scenarios, and reported expectations. All are useful evaluation targets, but they measure different aspects of household behavior. More explicit distinctions would help users interpret aggregate scores and avoid treating conditional prediction accuracy as direct evidence of accurate policy-effect simulation.

Questions for authors
  1. Could the authors provide a compact summary of sample retention after the main filtering steps, together with a few key characteristics of the retained samples? It would also be helpful to clarify whether survey weights are used and whether the benchmark is intended to represent the retained sample or the broader U.S. household population.

  2. Are quarter assignments shared across tasks from the same survey, and can the same household or household-period record appear in both fine-tuning and policy-task evaluation? A brief explanation and overlap summary would help clarify the intended generalization setting.

  3. How should users interpret the different types of policy-response tasks? As a small supporting check, could the authors remove the policy-specific input on one or two representative tasks using a single leading model? This would help illustrate the contribution of policy information without requiring a broad new evaluation.

6 comments

We thank the reviewer for their thoughtful comments.

"Could the authors provide a compact summary of sample retention after the main filtering steps, together with a few key characteristics of the retained samples? It would also be helpful to clarify whether survey weights are used and whether the benchmark is intended to represent the retained sample or the broader U.S. household population."

Thank you for your comment. We summarize the sample retention process and key characteristics in the below tables. For each task, we start from the survey observations in the task's population and period. We then drop observations with missing targets or predictors and observations that fail validity checks. To remove outliers, in 21 of the 32 tasks we also trim the top and bottom 1% of monetary variables and other unbounded continuous variables, such as income, spending or expectations. We note that missing data accounts for most of the dropped observations.

SurveyTasksAfter dropping missing valuesAfter validity checksAfter trimming (final sample)
CEX556%48%46%
CPS657%57%56%
Census199%99%99%
Michigan272%72%64%
PSID557%46%41%
SCE1372%71%69%
All3265%62%59%

Table 1: Share of observations kept after each filtering step.

SurveyTasksMean ageFemaleMarriedHousehold sizeWhite
CEX55448%54%2.484%
CPS64345%55%2.981%
Census15054%––81%
Michigan24946%64%––
PSID54132%41%3.169%
SCE134948%67%2.6–
All324744%55%2.880%

Table 2: Characteristics of the final samples, averaged over the tasks that report each trait. – means that no task reports it.

We do not use survey weights because the benchmark scores predictions for individual households and each observation counts equally. All six source surveys are designed to be nationally representative, so HouseholdBench covers a broad cross-section of U.S. households. Because the filters in Table 1 remove part of each survey sample, we do not claim that the retained samples are fully representative of the U.S. population.

"Are quarter assignments shared across tasks from the same survey, and can the same household or household-period record appear in both fine-tuning and policy-task evaluation? A brief explanation and overlap summary would help clarify the intended generalization setting."

Thank you for your comment. We draw the quarter split separately for each task. We split each task separately because tasks from the same survey can cover very different periods, so enforcing the same quarter split could leave some tasks with no test data.

As a result, a quarter might fall into the training set in one task and into the evaluation set in a different task. And a household might appear once in the training set and once in the evaluation set for the same task. In the evaluation data, 46% of the baseline-task records and 40% of the policy-task records come from a household that also appears in the fine-tuning data, and 8% and 10% from the same household-period.

To test whether the overlap of households affects the evaluation results, we recompute the scores of the fine-tuned model on the households that do not appear in the fine-tuning data (Table 3). We find that the improvements from fine-tuning on these unseen households are similar to or larger than those on the full sample, which suggests that the improvements do not come from the overlap. We have added the full results as Table D1 in the revised paper.

TasksMetricFull sampleUnseen households
Qwen3.5-4BFine-tunedImprovementQwen3.5-4BFine-tunedImprovement
BaselineRelMAE ↓1.110.87+0.241.170.88+0.29
BaselineMacro-F1 ↑0.380.42+0.050.360.40+0.04
BaselineTV ↓0.240.15+0.090.240.16+0.08
Policy-responseRelMAE ↓1.311.13+0.181.521.25+0.27
Policy-responseMacro-F1 ↑0.240.31+0.060.250.31+0.06
Policy-responseTV ↓0.150.16-0.010.200.16+0.04

Table 3: Scores of Qwen3.5-4B before and after fine-tuning, with 16 predictions per observation.

"How should users interpret the different types of policy-response tasks?"

Thank you for your comment. The 14 policy-response tasks have three types of targets:

  • Realized household outcomes after an economic shock or policy intervention (8 tasks).

  • Stated responses to hypothetical scenarios (4 tasks).

  • Stated beliefs or expectations after a shock or intervention (2 tasks).

We interpret all three in the same way: whether LLMs can predict how households adjust when the economic environment changes. We pool the three types in the main evaluation because the last two have few tasks, but users can also score each type separately.

"As a small supporting check, could the authors remove the policy-specific input on one or two representative tasks using a single leading model? This would help illustrate the contribution of policy information without requiring a broad new evaluation."

Thank you for suggesting this check! We removed the policy information on five policy-response tasks and evaluated GPT-5.6 sol and Fable 5.1 on the test set (200 observations per task). We chose these five tasks because their prompts are still valid questions after the policy information is removed.

Modelcons_cex_stimulus08 (RelMAE ↓)cons_cex_rebate01 (RelMAE ↓)labor_cps_ui (macro-F1 ↑)labor_cps_displace (macro-F1 ↑)income_sce_policy (macro-F1 ↑)
No-change baseline1.0001.0000.2720.2550.125
GPT-5.6 solWith policy input0.959 ± 0.0450.861 ± 0.0510.295 ± 0.0220.313 ± 0.0320.300 ± 0.022
GPT-5.6 solWithout policy input0.931 ± 0.0410.871 ± 0.0490.291 ± 0.0210.283 ± 0.0240.201 ± 0.019
GPT-5.6 solRemoving policy information leads to a significant performance drop (p<0.05)✓
Fable 5.1With policy input0.944 ± 0.0520.875 ± 0.0540.320 ± 0.0270.419 ± 0.0380.320 ± 0.022
Fable 5.1Without policy input0.979 ± 0.0510.873 ± 0.0530.313 ± 0.0260.269 ± 0.0210.201 ± 0.018
Fable 5.1Removing policy information leads to a significant performance drop (p<0.05)✓✓✓

Table 4: Performance of GPT-5.6 sol and Fable 5.1 on five policy-response tasks, with and without the policy information. The best result in each column is in bold. ± is the standard error over 10,000 bootstrap resamples of the 200 test observations. Significance is from a two-sided paired bootstrap test (Koehn, 2004) on the same resamples.

The results show that removing the policy information leads to a significant performance drop on three of the five tasks for Fable 5.1 and on one task for GPT-5.6 sol. This indicates that, on these tasks, the policy information is critical for accurately predicting how people respond to a policy change, a shock, or a hypothetical scenario.

Reviewer_4NmEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HouseholdBench, a benchmark for whether LLMs can predict U.S. household economic behavior. It harmonizes six public surveys (CEX, CPS, Michigan Surveys of Consumers, SCE, PSID, and Census), yielding 21.7M observations, and defines 32 tasks across five topics (consumption/saving, income, labor/retirement, macro expectations, housing). Tasks cover numeric, categorical, and probabilistic targets, and are split into 18 baseline tasks and 14 policy-response tasks (real policy episodes, shocks, or hypothetical scenarios). The authors evaluate 13 LLMs (8 proprietary, 5 open-weight) against a no-change baseline and a task-by-task XGBoost reference, using RelMAE, macro-F1, and total-variation distance.

Critical strengths

1.Scope and harmonization. Six surveys, five topics, 32 tasks, and three target types is a substantial step beyond prior work that uses one or two surveys and a handful of outcomes. The comparison table (Table 2) makes the gap to prior work clear.

2.The paper does not hide that XGBoost wins and that open-weight models lag; the leaderboard is reported with bootstrap standard errors, and the bias analysis (Figure 3) is informative.

Critical weaknesses

1.No code or benchmark release, and no limitations section. For a benchmark paper, the value depends on the splits and evaluation harness being reusable.

2.The abstract overstates two results. First, “the best model lowering error for numeric outcomes by 12.2%” is the best LLM (Fable 5.1, RelMAE 0.878); the best model overall is XGBoost at 0.811, i.e., an 18.9% reduction. A reader of the abstract will likely misread 12.2% as the best overall result. Second, “methods that enable a 4 billion parameter open-weight model to match proprietary models’ performance” is true only on the eight baseline numeric tasks. On the full 18 numeric tasks, Table F1 gives 1.017 for fine-tuned Qwen3.5-4B with K=16 versus 0.878 for Fable 5.1; on held-out policy tasks, Table 4 gives 1.13 versus 0.86. The fine-tuned model does not match frontier models on the broader suite. The abstract should scope both claims explicitly.

Questions for authors

see weakness

3 comments

We thank the reviewer for the comments.

"No code or benchmark release, and no limitations section. For a benchmark paper, the value depends on the splits and evaluation harness being reusable."

We have uploaded the supplementary material. It includes the full evaluation codebase and the full validation and test data for 17 of the 32 tasks. The other 15 tasks are built from survey data whose terms do not allow redistribution, such as PSID and CPS, and for these tasks we provide step-by-step instructions to reproduce our evaluation data from the source surveys.

We have also added a second paragraph to the Conclusion that discusses two limitations of our work. First, the policy-response tasks compare predictions with survey-reported answers, which may differ from actual behavior. Second, all tasks are built from U.S. household surveys, so the results may not generalize to other populations.

"First, “the best model lowering error for numeric outcomes by 12.2%” is the best LLM (Fable 5.1, RelMAE 0.878); the best model overall is XGBoost at 0.811, i.e., an 18.9% reduction."

Thank you for the suggestion. We have changed that sentence to: "Most LLMs outperform the no-change baseline, including for policy response tasks—with the best LLM lowering error for numeric outcomes by 12.2%." in the revised paper. The changed text is in bold.

"Second, “methods that enable a 4 billion parameter open-weight model to match proprietary models’ performance” is true only on the eight baseline numeric tasks. ... The abstract should scope both claims explicitly."

Thank you. We have changed that sentence to: "We identify methods that enable a 4 billion parameter open-weight model to match proprietary models' performance on baseline tasks with numeric and probabilistic outcomes: fine-tuning and aggregating 16 predictions per observation." in the revised paper. The changed text is in bold.

Thank you again for your thoughtful review! As the discussion period ends on November 18, could you let us know whether our response has addressed your concerns? We are happy to clarify any remaining concerns.

Thank you for the feedback, I'll keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.