- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper benchmarks four LLMs as synthetic survey respondents under demographic prompting on GSS and WVS, against non-LLM baselines fit on held-out human data. Two reported failures: no LLM beats a demographic-lookup baseline at the individual level under proper scoring, and models over-determine demographics.
Critical strengths
- Rigorous, baseline-anchored evaluation. Individual fidelity is scored with proper and distance-aware metrics against non-LLM baselines on the same rows, with paired-bootstrap CIs. The robustness checks are careful and transparent (FDR correction, respondent-clustered bootstrap, survey weights, invalid-output accounting, prompt perturbations).
- Useful separation of evaluation levels. Showing that good aggregate fidelity can coexist with individual-level failure and subgroup distortion, consistently across two domains and two model families, is a practically relevant message, and it comes with a concrete validation checklist.
Critical weaknesses
-
RQ1 is largely guaranteed by design. With only demographics in the prompt, the best any predictor can do is the Bayes-optimal conditional given those demographics — which the lookup/logistic baselines fit directly from thousands of labeled respondents. A zero-shot model cannot be expected to beat them, so "no LLM wins" is near-tautological; only the gap size is informative. More useful: (i) a learning curve (how many labeled respondents the baselines need to match each LLM); (ii) richer conditioning (few-shot, the respondent's other answers); (iii) a fine-tuned-LLM baseline as a fitted-vs-fitted check.
-
RQ3 over-determination is not conclusive. The stereotyping index compares model from ~100 respondents against human from the full table, tested against zero; since /Cramér's V are upward-biased at small n, magnitudes and significance are inflated. Fix: a same-n human null (subsample ~100 humans per question, recompute , test against that). Also unclear whether Style C answers use argmax or sampling — if argmax, over-determination follows by construction. These must be separated before the "caricature" reading holds.
-
Limited novelty and scope. The phenomena are known: caricature = individuation + exaggeration (CoMPosT, Cheng et al. 2023) and misportrayal/flattening (Wang et al. 2025); neither is cited, though the paper argues against "flattening" (the two can co-occur). The real contribution is the toolkit (, , decision diagnostics) and checklist, not the findings. Scope is also narrow for the title "When Synthetic Users Fail": 4 models/2 families (no GPT/Gemini/reasoning models), 26 questions, one elicitation family, attitudes only — and it conflicts with the paper's own RQ4 (validity is prompt-dependent, ~10% flip on option-order reversal). Narrow the claim, or broaden model and elicitation coverage.
Questions for authors
See weakness