acceptodds
Under review as a conference paper at ICLR 2027

Population Fidelity: Evaluating Population Representativeness in LLMs

Abstract

Large language models (LLMs) show considerable potential in simulating human attitudes and preferences. Prior work finds that LLM-generated responses can compress the range of attitudes found within populations and misrepresent particular subgroups in ways that vary across models and topics. We introduce Population Fidelity, an evaluation framework that distinguishes key conditions required for a set of LLM-generated responses to represent a population. It incorporates three dimensions: group-level accuracy, the amount of between-group variation, and the structure of that variation. We demonstrate the framework's utility in two ways. First, we reproduce a prior study of "machine bias" in LLM survey responses and apply the framework to its models and more recent ones, showing that poor representation reflects not only insufficient between-group variation but also variation assigned to the wrong groups. Second, we evaluate one proposed approach to improving models' population representativeness: cultural fine-tuning. We find that cultural fine-tuning can improve alignment with the survey center without improving the representation of within-population differences, a distinction that unidimensional evaluations overlook. We argue that representing a population requires models to reproduce several features of human attitudinal variation simultaneously. Our framework organizes these features and provides reusable code, data, and trained models for evaluating population fidelity across substantive domains and assessing proposed alignment methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLMs, bias, fairness, alignment, survey research, social, population, culture, representativeness

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_SVYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper there is a framework presented, Population Fidelity, for evaluating how well LLM-generated survey responses represent different population groups. The authors measure the accuracy of each group’s responses, the variation between groups, and whether the same groups are more similar or different. This is all combined into a single population fidelity score (PFS). The evaluation uses four world values survey questions across 5 countries and 3 survey waves, reproduces an earlier study, and compares models before and after cultural fine-tuning. The results show that matching average survey responses does not necessarily recover group differences, and that cultural fine-tuning does not consistently improve population fidelity.

Critical strengths

S1. This is an interesting space and the authors present good experimental coverage within the chosen survey setting. The authors compare several models and fine-tuning conditions and examine a range of response methods and metric choices. Using the same retained groups for paired comparisons and repeating sampled model responses helps make the comparisons more reliable. The survey-average baseline also provides useful context.

S2. I think the core idea of separating different kinds of population mismatch is interesting. It makes sense that a model could match an average response while missing difference between groups, or produce variation between the wrong groups. Making these distinctions explicit could help researchers understand what an alignment method actually improves.

Critical weaknesses

W1. It is a bit unclear how the combined PFS should be used in practice. The structure measure captures group similarity and distance but it can give a perfect score even when the direction of differences is reversed. Accuracy can penalize this but the combined score can still remain quite high. It would be helpful to provide guidance or explain when PFS can be used for selecting a model / as a guardrail and which group-level errors matter most.

W2. There’s also some uncertainty about how reliable the human group differences are. For example survey cells can contain a small number of respondents. It could be helpful to separate the survey sampling variation from model mismatch when interpreting low structure scores for smaller subgroups.

Questions for authors

Q1. What size of PFS improvement would be meaningful when choosing between models?

Q2. Have the authors assessed how stable the human group structure is across survey samples or different minimum cell sizes? Any existing sensitivity analysis would help with establishing how much the low structure score reflect actual model behavior.

Q3. How should PFS be interpreted in applications that affect individuals, like in personalized recommendations? A model could match a demographic group’s broad response distribution while misrepresenting people within it leading to aggregation bias. Is there any risk of aggravating aggregation bias by selecting models for higher PFS and is there any additional validation that could be done around this?

6 comments
AuthorsOpenReview ↗

Initial Comment

Thank you for these thoughtful comments. We have revised the manuscript to address your concerns, with the main changes highlighted in orange.

To facilitate your evaluation, we address each major point in a separate response.

AuthorsOpenReview ↗

Response to W1 (Accuracy Metric Limitation)

We thank the reviewer for identifying this limitation. We have revised the Structure metric to account for the orientation of group differences.

Specifically, we introduce an orientation component that measures whether the model preserves the ordering of subgroup mean responses. The revised structure score takes the minimum of the pairwise-distance correlation (clipped at zero) and the orientation component, requiring agreement in both dimensions.

This addresses the reversal problem identified by the reviewer. In Figure 1, model M4 preserves all pairwise distances but reverses group ordering. Under the revised metric, it correctly receives a Structure score of zero, rather than one.

We have updated Section 4.2, Appendix B.2, and the corresponding experiments to incorporate this correction.

AuthorsOpenReview ↗

Response for Q1 "What size of PFS improvement would be meaningful when choosing between models?"

We thank the reviewer for this important question. We have added Appendix H, “Interpreting Differences in PFS between Models,” to provide practical guidance for model comparison and selection.

We argue that there is no universal threshold for a meaningful PFS difference. Its interpretation depends on sampling uncertainty, the underlying component scores, and the intended application.

To illustrate this, we introduce a synthetic example comparing three models and conduct an empirical comparison between Qwen3-VL-8B-Thinking and Gemma-4-31B-it across all four WVS questions and both elicitation modes. We use paired bootstrap resampling to estimate uncertainty in differences in PFS and its components.

Our examples also show that similar PFS values can conceal different strengths in accuracy, adaptability, and structure. We therefore recommend interpreting PFS differences alongside their uncertainty intervals and component scores, considering which dimensions of population representation are most important for the intended application.

Appendix H provides the methodological details, discusses the limitations of the resampling procedure, and summarizes our recommendations for model selection. We also reference this appendix when introducing PFS in Section 4.3.

Reviewer_Wiv5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces “Population Fidelity”, evaluating whether LLM simulated survey responses reproduce subpopulation response distributions (distributional metric; authors call it ‘Accuracy’), the amount of between-subgroup variation (‘Adaptability’), and the structure of those distributional differences (‘Structure’). Experiments using World Values Survey data show that relatively high distributional accuracy can co-exist with poor recovery of group differences, and that CultureLLM style training does not necessarily improve the population fidelity.

Critical strengths
  1. Paper is easy to read.

  2. The paper clearly illustrates how seemingly-good average distributional accuracy can obscure failures to reproduce demographic differences, as shown in Figure 1. Separating accuracy, magnitude, and relational structure provides a good diagnostic perspective for evaluating synthetic survey respondents.

Critical weaknesses
  1. Limited incremental contribution of the metrics.

Two proposed metrics (which is a main contribution of the paper) Adaptability and Structure are summaries of the subgroup response distributions used in existing distributional evaluations. Although they distinguish failure modes of the LLM prediction obscured by averaging, they only provide incremental value over existing evaluation metrics and straightforward diagnostics.

  1. Limited informativeness of the main intervention.

CultureLLM fine-tuning trains toward a single aggregate answer per question; without supervising response diversity or subpopulations (e.g., demographic subpopulations). So its failure to recover subgroup variation is quite unsurprising and provides limited evidence about adaptation fine-tuning methods designed for population representation.

  1. Confounded comparison with SubPOP.

Related to Weakness point 2 – the supplementary SubPOP experiment results are confusing. The experiment changes multiple conditions at once: the survey source, population coverage, definitions of subgroups, and evaluation item overlap. This is not a good design of the experiment. The results are then just showing transfer under these simultaneous condition mismatches, rather than isolating the effectiveness of subpopulation-level training.

Questions for authors

Please refer to the weakness section.

6 comments
AuthorsOpenReview ↗

Initial comment

Thank you for your thoughtful comments. We have revised the manuscript to address your concerns, with the main changes highlighted in orange.

To facilitate your evaluation, we address each major point in a separate response.

AuthorsOpenReview ↗

Response for point "1. Limited incremental contribution of the metrics"

While our Adaptability and Structure metrics build on established approaches, neither one appears as described in a published peer-reviewed study or preprint, to the best of our knowledge. Moreover, no previous study systematically compares results for Accuracy, Adaptability, Structure, and Center Alignment for a given model/question, offers a single-value summary score across these dimensions, or provides code and data (including Python and R libraries, see https://anonymous.4open.science/r/popfidelity-AB4D ) for evaluating fidelity in new models and surveys. We see our synthesis of previous distributional evaluations into a single, portable framework as one of our paper's major contributions. As we outline in Section 2 (Related Work), previous works tend to focus on one, or at most two, specialized distributional metrics pertinent to population fidelity in a given study, often for the purpose of understanding the representation of specific groups, rather than providing a general framework for comparing model performance on surveys.

Our measurement approach builds on Boelaert et al. (2025), who compare LLM and human responses using Earth-Mover's Distance (EMD; i.e., 1-Wasserstein Distance). The metrics we introduce incorporate EMD because it is a standard statistical measure for comparing response distributions and unique among several alternatives (notably Jensen–Shannon Distance, used in some related work, e.g., Ma et al. 2025) in that it accounts for the ordered nature of categorical responses, essential to evaluating survey data (Boelaert et al. 2025, fn. 18).

Several studies use Cramér's V (e.g., Argyle et al. 2023; Lee et al. 2024; Ma et al. 2025) as an analogue to Adaptability, though it is less suited to survey data due to ignoring response ordering, being undefined when models produce no variation (see Lee et al. 2024: fig. 4 note), and being computed per demographic variable rather than across an arbitrary set of subgroups. Closest to our specific metric is that of Boelaert et al. (2025), who compute median pairwise nEMD distances on all subpopulations, as we do. However, they only report these values separately. Our Adaptability metric, by contrast, formalizes this comparison using a normalized ratio that is comparable across items and distinguishes response compression (A < 1) from amplification (A > 1).

Structure, which captures whether models reproduce which groups are similar to or different from one another, is not used in previous peer-reviewed studies to our knowledge. Williams et al. (2026) apply a related correlational approach, but to relationships between survey items, whereas we study differences between groups within an item. Two recent arXiv preprints use related measures, though neither corresponds to our Structure metric. Bhattacharyya et al. (2026) also correlate human and model distances between subgroup pairs, though they use Pearson correlation, average distances across items, and only compare a subset of high-divergence pairs. Qin et al. (2026), in a study of heterogeneity in climate opinions, instead represent survey and model subgroup distances as MDS maps and compare them using Procrustes distance. Neither incorporates the direction of group differences, which our revised Structure metric does, in response to Reviewer SVYs. We have revised Section 2 to discuss the Cramér's V-based comparisons and these preprints, and Section 3 to explain our choice of nEMD. We would welcome references to any specific evaluations the reviewer has in mind.

The considerable value of our Structure metric is borne out by the fact that relying on existing metrics and straightforward diagnostics can lead to different conclusions about models. First, a survey-center baseline that ignores demographics matches or exceeds model accuracy in 157 of 164 run–mode combinations (Section 6.1; Appendix D, Table 5). High accuracy can therefore suggest good performance even when a model poorly represents group differences. Structure reveals this failure (it is the limiting PFS component in 160 of 164). Second, GPT-5.6 Terra improves on the archived GPT-3 outputs in Structure in 112 of 120 demographic-level comparisons, but in center alignment in only 54 (Appendix D.2), so center alignment alone fails to detect most of this improvement.

References

  • Argyle et al. 2023. Out of One, Many. Political Analysis 31(3).
  • Bhattacharyya et al. 2026. How Well Do LLMs Capture Human Personality? arXiv:2606.18263.
  • Boelaert et al. 2025. Machine Bias. Sociological Methods & Research 54(3).
  • Lee et al. 2024. Can LLMs Estimate Public Opinion about Global Warming? PLOS Climate 3(8).
  • Ma et al. 2025. Algorithmic Fidelity of LLMs in Generating Synthetic German Public Opinions. ACL 2025.
  • Qin et al. 2026. Restoring Heterogeneity in LLM-based Social Simulation. arXiv:2604.06663.
  • Williams et al. 2026. Beyond Marginal Distributions. Findings of ACL 2026.
AuthorsOpenReview ↗

Response for point "2. Limited informativeness of the main intervention."

We agree that CultureLLM-style fine-tuning does not explicitly optimize subgroup variation and have clarified that its results should not be generalized to all population adaptation methods.

Nevertheless, CultureLLM aims to improve cultural alignment and reports gains across 60 culture-related datasets (Li et al., 2024). This motivates examining whether improvements in cultural alignment extend to population representativeness. Although limited subgroup recovery may be expected, the effects of fine-tuning on group-level accuracy, between-group variation, and their structure remain empirical questions that Population Fidelity can address.

To address this limitation, we substantially expanded our initial fine-tuning experiments, adding evaluations of two additional approaches: country-level distribution matching (SimLLCultureDist) and subgroup-level distribution matching (SubPOP).

On the WVS, SimLLCultureDist improves PFS in 33 of 48 comparisons (mean ΔPFS = +0.070), including improvements in structure (mean ΔSstruct = +0.042). On its original GSS benchmark, SubPOP achieves substantial improvements, increasing PFS in 698 of 798 comparisons (mean ΔPFS = +0.397), primarily through improvements in structure (mean ΔSstruct = +0.292).

These findings demonstrate that poor subgroup recovery is not an inevitable outcome of fine-tuning and that methods targeting richer response distributions can improve population fidelity. At the same time, the improvements vary substantially across evaluation settings.

Our contribution is therefore not simply documenting CultureLLM's expected limitations, but providing a common evaluation framework that distinguishes which dimensions of population representativeness different adaptation approaches recover. We have incorporated these additional experiments and clarified their interpretation in Section 6.3 and Appendices F and G.

References

  • Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. CultureLLM: Incorporating cultural differences into large language models. Advances in Neural Information Processing Systems, 37:84799–84838, 2024.

Comments on the paper

5 comments
Area_Chair_M4F4OpenReview ↗

Replacement for missing 3rd review

Summary This paper proposes Population Fidelity, a way to judge whether LLM-generated survey responses represent a population. It splits representativeness into accuracy, adaptability (how much the groups differ), and structure (whether the differences fall on the right groups), then combines them into one score (PFS). Using the Machine Bias setup of Boelaert et al. (2025) with WVS data, the authors find that models track the survey average well but rarely get the group structure right. They also report that CultureLLM-style fine-tuning does not help the score on average. Strengths The paper points at a real blind spot in current social simulation work: a model can match the overall distribution and still get the subgroups wrong. The framework is also clearly laid out, and can be a good addition to the research diversity in ML venues. Weaknesses

  1. The results also depend on metric choices. Clipping the structure correlation at zero hides reversed structure, and the geometric mean changes the picture a lot compared with an arithmetic mean. The authors should show that the findings survive these alternatives.
  2. The evidence is narrow: one survey, four items, two countries, one main fine-tuning method, and English prompts. But the abstract and conclusion read more broadly than that.
  3. Novelty is moderate. The framework builds directly on the Machine Bias design, and the relation to Williams et al. (2026) on attitude structure needs a clearer account. Questions for the authors
  4. How do the main results change with an unclipped structure score?
  5. (How) was overlap with WVS in the pretraining data checked?
  6. How does your structure measure differ from the analysis in Williams et al. (2026)? And what are your major contribuions. Presentation: 2 Informativeness: 2 Soundness: 3 Overall score: 3/4
AuthorsOpenReview ↗

Initial Comment

We thank the AC for the helpful comments, and we are glad that they find our paper to be engaging with an important and overlooked aspect of social simulation research.

To facilitate evaluation, we address each major point in a separate response.

AuthorsOpenReview ↗

Response to "How do the main results change with an unclipped structure score?"

We thank the reviewer for this suggestion. We added a sensitivity analysis using an orientation-aware, signed Structure score without zero clipping (Appendix E.3, Table 13). Under this specification, negative structural correspondence produces negative PFS values rather than being truncated to zero.

Across the 164 evaluations in our original WVS panel, mean PFS decreases from 0.212 to 0.147, while median PFS remains 0.220. Structure remains the limiting component in 161 of 164 evaluations (versus 160 under the default), and PFS rankings remain highly consistent (Spearman correlation = 0.987). Of the 49 cases with zero PFS under the default specification, 48 receive negative scores.

Thus, while removing clipping changes score magnitudes and reveals negative structural correspondence, it does not alter our central finding that models struggle to reproduce the observed structure of subgroup differences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.