- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper there is a framework presented, Population Fidelity, for evaluating how well LLM-generated survey responses represent different population groups. The authors measure the accuracy of each group’s responses, the variation between groups, and whether the same groups are more similar or different. This is all combined into a single population fidelity score (PFS). The evaluation uses four world values survey questions across 5 countries and 3 survey waves, reproduces an earlier study, and compares models before and after cultural fine-tuning. The results show that matching average survey responses does not necessarily recover group differences, and that cultural fine-tuning does not consistently improve population fidelity.
Critical strengths
S1. This is an interesting space and the authors present good experimental coverage within the chosen survey setting. The authors compare several models and fine-tuning conditions and examine a range of response methods and metric choices. Using the same retained groups for paired comparisons and repeating sampled model responses helps make the comparisons more reliable. The survey-average baseline also provides useful context.
S2. I think the core idea of separating different kinds of population mismatch is interesting. It makes sense that a model could match an average response while missing difference between groups, or produce variation between the wrong groups. Making these distinctions explicit could help researchers understand what an alignment method actually improves.
Critical weaknesses
W1. It is a bit unclear how the combined PFS should be used in practice. The structure measure captures group similarity and distance but it can give a perfect score even when the direction of differences is reversed. Accuracy can penalize this but the combined score can still remain quite high. It would be helpful to provide guidance or explain when PFS can be used for selecting a model / as a guardrail and which group-level errors matter most.
W2. There’s also some uncertainty about how reliable the human group differences are. For example survey cells can contain a small number of respondents. It could be helpful to separate the survey sampling variation from model mismatch when interpreting low structure scores for smaller subgroups.
Questions for authors
Q1. What size of PFS improvement would be meaningful when choosing between models?
Q2. Have the authors assessed how stable the human group structure is across survey samples or different minimum cell sizes? Any existing sensitivity analysis would help with establishing how much the low structure score reflect actual model behavior.
Q3. How should PFS be interpreted in applications that affect individuals, like in personalized recommendations? A model could match a demographic group’s broad response distribution while misrepresenting people within it leading to aggregation bias. Is there any risk of aggravating aggregation bias by selecting models for higher PFS and is there any additional validation that could be done around this?
6 comments
Response to W1 (Accuracy Metric Limitation)
We thank the reviewer for identifying this limitation. We have revised the Structure metric to account for the orientation of group differences.
Specifically, we introduce an orientation component that measures whether the model preserves the ordering of subgroup mean responses. The revised structure score takes the minimum of the pairwise-distance correlation (clipped at zero) and the orientation component, requiring agreement in both dimensions.
This addresses the reversal problem identified by the reviewer. In Figure 1, model M4 preserves all pairwise distances but reverses group ordering. Under the revised metric, it correctly receives a Structure score of zero, rather than one.
We have updated Section 4.2, Appendix B.2, and the corresponding experiments to incorporate this correction.
Response for Q1 "What size of PFS improvement would be meaningful when choosing between models?"
We thank the reviewer for this important question. We have added Appendix H, “Interpreting Differences in PFS between Models,” to provide practical guidance for model comparison and selection.
We argue that there is no universal threshold for a meaningful PFS difference. Its interpretation depends on sampling uncertainty, the underlying component scores, and the intended application.
To illustrate this, we introduce a synthetic example comparing three models and conduct an empirical comparison between Qwen3-VL-8B-Thinking and Gemma-4-31B-it across all four WVS questions and both elicitation modes. We use paired bootstrap resampling to estimate uncertainty in differences in PFS and its components.
Our examples also show that similar PFS values can conceal different strengths in accuracy, adaptability, and structure. We therefore recommend interpreting PFS differences alongside their uncertainty intervals and component scores, considering which dimensions of population representation are most important for the intended application.
Appendix H provides the methodological details, discusses the limitations of the resampling procedure, and summarizes our recommendations for model selection. We also reference this appendix when introducing PFS in Section 4.3.
Initial Comment
Thank you for these thoughtful comments. We have revised the manuscript to address your concerns, with the main changes highlighted in orange.
To facilitate your evaluation, we address each major point in a separate response.