acceptodds
Withdrawn from ICLR 2027

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are increasingly used as synthetic users: stand-ins for human respondents whose simulated answers inform product, policy, and market decisions. We ask when this substitution is valid and when it fails. Using a unified evaluation protocol, we study four models spanning two model families and an 8B-to-frontier capability range across two independent domains of real human responses: U.S. social attitudes from the General Social Survey (GSS) and cross-cultural values from the World Values Survey (WVS). We benchmark each model against non-LLM baselines fit on held-out human data. Across both domains and model families, we find two consistent failures under the demographic prompting and survey-simulation protocols we test. First, at the individual level, no LLM outperforms the strongest non-LLM baseline. Under proper scoring rules (log-loss and Brier score), every model assigns less probability to the observed human response than a demographic baseline; this gap persists under distance-aware scoring, and on WVS, every model also falls well below the baseline in exact-match accuracy. Second, the models over-determine demographics: they treat demographic identity as more predictive of attitudes than it is among real human responses, most clearly for the Claude models and Llama-70B, though our small model sample overstates this gap. In the two within-family size comparisons we test, the larger model remedies neither failure.

Primary area:
datasets and benchmarks
Keywords:
large language models, synthetic users, survey simulation, LLM evaluation, demographic stereotyping, computational social science

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yMfHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks four LLMs as synthetic survey respondents under demographic prompting on GSS and WVS, against non-LLM baselines fit on held-out human data. Two reported failures: no LLM beats a demographic-lookup baseline at the individual level under proper scoring, and models over-determine demographics.

Critical strengths
  • Rigorous, baseline-anchored evaluation. Individual fidelity is scored with proper and distance-aware metrics against non-LLM baselines on the same rows, with paired-bootstrap CIs. The robustness checks are careful and transparent (FDR correction, respondent-clustered bootstrap, survey weights, invalid-output accounting, prompt perturbations).
  • Useful separation of evaluation levels. Showing that good aggregate fidelity can coexist with individual-level failure and subgroup distortion, consistently across two domains and two model families, is a practically relevant message, and it comes with a concrete validation checklist.
Critical weaknesses
  1. RQ1 is largely guaranteed by design. With only demographics in the prompt, the best any predictor can do is the Bayes-optimal conditional given those demographics — which the lookup/logistic baselines fit directly from thousands of labeled respondents. A zero-shot model cannot be expected to beat them, so "no LLM wins" is near-tautological; only the gap size is informative. More useful: (i) a learning curve (how many labeled respondents the baselines need to match each LLM); (ii) richer conditioning (few-shot, the respondent's other answers); (iii) a fine-tuned-LLM baseline as a fitted-vs-fitted check.

  2. RQ3 over-determination is not conclusive. The stereotyping index compares model η2\eta^2 from ~100 respondents against human η2\eta^2 from the full table, tested against zero; since η2\eta^2/Cramér's V are upward-biased at small n, magnitudes and significance are inflated. Fix: a same-n human null (subsample ~100 humans per question, recompute ηhuman2\eta^2_{\text{human}}, test against that). Also unclear whether Style C answers use argmax or sampling — if argmax, over-determination follows by construction. These must be separated before the "caricature" reading holds.

  3. Limited novelty and scope. The phenomena are known: caricature = individuation + exaggeration (CoMPosT, Cheng et al. 2023) and misportrayal/flattening (Wang et al. 2025); neither is cited, though the paper argues against "flattening" (the two can co-occur). The real contribution is the toolkit (Δη2\Delta\eta^2, ΔV\Delta V, decision diagnostics) and checklist, not the findings. Scope is also narrow for the title "When Synthetic Users Fail": 4 models/2 families (no GPT/Gemini/reasoning models), 26 questions, one elicitation family, attitudes only — and it conflicts with the paper's own RQ4 (validity is prompt-dependent, ~10% flip on option-order reversal). Narrow the claim, or broaden model and elicitation coverage.

Questions for authors

See weakness

Reviewer_QyDvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper evaluates how well LLMs can simulate human survey responses across two datasets covering social attitudes and cultural values. It finds that LLMs can match aggregate distributions reasonably well, but perform worse than simple demographic baselines at predicting individual responses and tend to exaggerate demographic differences.

Critical strengths
  • Evaluating across two survey datasets and multiple models makes the findings more convincing.
  • Comparing against simple statistical baselines trained on real human data is a good evaluation setup.
Critical weaknesses
  • The main findings that LLM synthetic users fail to capture individual variation and exaggerate demographic stereotypes have already been studied in prior work. The novelty beyond existing studies is somewhat limited.
  • The paper only uses demographic information to construct synthetic users. It does not evaluate richer persona descriptions, behavioral history, or other personal information, which limits how much the conclusions generalize to synthetic users in general.
  • The statistical baselines are trained on real human responses, while LLMs are evaluated zero-shot, making the comparision somewhat unfair.
  • The stereotyping analysis compares a small sample of synthetic users against a much larger human sample, which can inflate the measured differences.
  • The paper mostly identifies existing problems without proposing or evaluating methods to improve synthetic users.
Questions for authors
  • How do these findings differ from previous works such as Digital Twins Are Funhouse Mirrors and LLM Generated Persona Is a Promise with a Catch?
  • Would the conclusions still hold when the LLM is given richer personal information beyond demographics, such as previous survey responses or behavioral history?
  • How much of the stereotyping effect remains when comparing synthetic users against equally sized human samples?
  • Can the authors explore any methods to actually improve individual prediction or reduce demographic stereotpying?
Reviewer_nsvaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores the reliability of using large language models to simulate survey respondents and examines the levels at which they fail. The authors use data from the General Social Survey (GSS) and the World Values Survey (WVS) and evaluated four models: Claude Haiku 4.5, Claude Sonnet 4.6, Llama-3.1-8B, and Llama-3.3-70B. The experiments may be insufficient to support broader conclusions.

Critical strengths

None

Critical weaknesses

The experiments in the paper are seriously insufficient. For example, the authors tested only four LLMs from two model families, so the generalizability of their conclusions is not well supported.

The experimental design and the conclusions presented in the paper offer limited novelty.

Questions for authors

Do the authors believe that these findings also hold for other LLMs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.