acceptodds
Under review as a conference paper at ICLR 2027

The Need for Socially-Grounded Personas for User Simulation

Abstract

Synthetic personas are widely used to condition large language models (LLMs) for social simulation, yet most personas are still constructed from coarse sociodemographic attributes or summaries. We revisit persona creation by introducing SCOPE, a socially grounded framework for persona construction and evaluation, built from a 141-item, two-hour sociopsychological protocol collected from 124 U.S.-based participants. Across seven models, we find that demographic-only personas are a structural bottleneck: demographics explain only 1.5% of variance in human response similarity. Adding sociopsychological facets improves behavioral prediction and reduces over-accentuation, and non-demographic personas based on traits and identity achieve strong alignment with substantially lower bias. These trends generalize to SimBench (441 aligned questions), where SCOPE personas outperform default prompting and NVIDIA Nemotron personas, and SCOPE augmentation improves Nemotron-based personas. Our results indicate that persona quality depends on sociopsychological structure rather than demographic templates or summaries.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Synthetic personas, user simulation, persona modeling, demographic bias, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RW2UOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SCOPE, a framework for constructing and evaluating user personas based on social psychological information. The authors collect 141 questionnaire responses from 124 US participants, compare simulation performance across conditions including demographic information, personality traits, identity narratives, and model-generated summaries, and conduct experiments over seven language models and SimBench. The paper argues that socio-psychological information improves behavioral simulation and reduces over-amplification of demographic disparities.

Critical strengths
  1. The research question is practically meaningful. How to construct personas that reflect real behavioral differences is an important problem in user simulation and social computing. By jointly examining behavioral prediction and the over-amplification of demographic structures, the paper helps identify the limitations of building personas relying solely on demographic attributes.

  2. The real-participant data and multi-dimensional experimental design carry research value. The paper collects questionnaires and identity narratives from 124 participants, providing an empirical foundation for comparing model outputs against real human behaviors. Comparisons across seven models, multiple persona construction strategies, and an external benchmark also yield a useful analytical framework for investigating the effects of different persona information.

Critical weaknesses
  1. Core results contain internal inconsistencies, and discrepancies between figures and tables reverse comparative conclusions across methods.

Figure 2 and Table 16 report markedly different correlation coefficients for the same models under identical persona conditions.

ModelCase 7c: Figure 2 (approx.)Case 7c: Table 16Case 4: Table 16
DeepSeek R10.6420.6080.625
GPT-5.10.6400.6120.621
Qwen3 480b0.6410.5660.586

These discrepancies reverse the comparison between non-demographic personas and full conditioning: Figure 2 places Case 7c above Case 4 for all three models, whereas Table 16 places it below Case 4. This directly affects the claim in Section 4.1, point III (p. 6, lines 315–319), and Section 5 that non-demographic personas match or exceed full conditioning in correlation

  1. The partitioning between persona inputs and evaluation targets is self-contradictory, making it impossible to verify whether the core experiments truly employ held-out evaluation.

§3.1 explicitly lists “sociodemographic behaviors” as inputs for persona construction, and §3.3 states that Case 4 includes this information. Nevertheless, §4.1 incorporates items of this type into the 72 supposedly held-out evaluation questions without explaining whether item-level splitting was performed within this dimension. Similarly, §3.1 reserves “occupational identity” for evaluation, yet §3.3 explicitly states that Case 2 uses personal and occupational identity narratives. Appendix Table 14 raises further concerns: input examples for Case 3 and Case 4 include statements of values and goals, while §3.1 defines “values and motivations” as a held-out evaluation dimension.

These contradictions do not constitute definitive proof of answer leakage, but they prevent readers from verifying the input–target separation upon which the core experiments rely. If richer personas directly contain answers to some evaluation items, performance gains may stem from access to target information rather than superior behavioral generalization. To support its central claims, the paper must provide clear item-wise lists of inputs and evaluation items for each condition, including information accessible during summary generation, and report results only on verified non-overlapping targets.

Questions for authors
  • Why do Figure 2 and Table 16 report different correlations for Case 7c, reversing its comparison with Case 4 for DeepSeek R1, GPT-5.1, and Qwen3 480b? Which values are correct, and which conclusions remain supported after correction?

  • Could the authors provide the exact conditioning and evaluation item IDs for each persona case, including the inputs used for summary generation? Sections 3.1–4.1 give conflicting descriptions of whether sociodemographic behavior, professional identity, and values/goals are conditioning inputs or held-out targets. Do the reported gains persist after excluding any overlapping items?

  • Section 4.2 evaluates agreement with the most frequent response of a matching demographic group. How does this evaluation distinguish improved prediction of group-majority answers from faithful simulation of within-group behavioral variation? Do response-distribution comparisons support the claimed improvement in behavioral fidelity?

Reviewer_d2t4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SCOPE, a framework for constructing LLM personas using sociopsychological information. The authors collect a 141-item questionnaire from 124 participants and construct personas using different combinations of demographics, personality traits, and identity narratives. They evaluate these personas across seven LLMs using correlations with human responses, answer accuracy, and demographic bias measures. The results suggest that adding psychological and personal information improves behavioral alignment, while demographic-only personas tend to exaggerate demographic differences. Additional experiments are conducted on SimBench and NVIDIA Nemotron personas.

Critical strengths

The paper collects relatively detailed sociopsychological data from real participants, allowing the authors to compare different persona construction strategies against human responses. The evaluation also covers seven LLMs and includes an external benchmark, which strengthens the empirical results.

Critical weaknesses

The main finding is that adding personality traits and personal narratives improves behavioral prediction compared with using demographics alone. This is not particularly surprising, and similar ideas have already been explored in previous work on psychologically grounded personas and interview-based simulations. The paper does not clearly explain what SCOPE adds beyond combining these existing approaches.

Another concern is that the comparisons do not control for the amount of information provided to the model. The richer personas contain substantially more information about each participant, so it is difficult to determine whether the improvements come from the proposed sociopsychological framework or simply from having more individual information. The current experiments do not separate these explanations.

The evaluation is insufficient to support several of the main claims. This paper primarily evaluates whether model responses correlate with human responses across survey questions. However, this does not necessarily mean that the model has captured the characteristics of a particular individual. A model could achieve relatively high correlations by reproducing common response patterns without accurately representing individual differences. The paper provides limited evidence that its personas distinguish between individuals with similar backgrounds but different psychological characteristics.

The interpretation of demographic bias is also questionable. The proposed metric measures the association between demographic similarity and response similarity. Lower values do not necessarily mean less biased or more realistic behavior. Some non-demographic personas produce substantially weaker demographic associations than those observed in the human data. It is unclear why this should be interpreted as an improvement.

The SimBench evaluation has a related limitation. Accuracy is calculated against the majority response of a demographic group rather than the actual responses of the corresponding individual. This evaluation therefore does not directly test whether the constructed personas can simulate individuals more accurately.

Questions for authors

1.How does SCOPE differ substantively from previous psychologically grounded and interview-based persona approaches? The paper currently appears to combine several established ideas rather than introduce a new approach to persona construction.

2.Have the authors compared different persona strategies while controlling for input length and the amount of individual information? Without such controls, the improvements cannot be attributed specifically to the proposed sociopsychological structure.

3.Do the generated personas predict their corresponding participants better than other participants? The reported correlations do not establish whether the model captures individual-specific differences or only common response patterns.

4.Why are lower demographic associations interpreted as reduced bias? Some conditions underestimate demographic differences relative to the human baseline, which may also indicate poor simulation fidelity.

Reviewer_gGTKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCOPE, a framework for constructing and evaluating LLM personas using sociopsychological information beyond demographic attributes. It collects a 141-item questionnaire from 124 U.S.-based participants covering eight facets, then varies persona conditioning through human-provided profiles, model-written summaries, inferred facets, and representations without explicit demographics. Evaluation spans seven models and measures answer accuracy, human–model response correlation, and demographic accentuation relative to a human baseline, with additional analysis of open-ended responses. External evaluation with GPT-4o uses 441 aligned SimBench questions and compares SCOPE with default and Nemotron personas, including SCOPE-augmented Nemotron variants. The paper reports improved response alignment under richer conditioning and reduced demographic over-accentuation in several configurations.

Critical strengths
  1. A valuable human-grounded resource and controlled comparison of persona information. Collecting a two-hour, multidimensional questionnaire from 124 participants is a substantial contribution. The facet ablations address a practical question: which information improves a persona beyond demographics? Full conditioning improves the reported correlation over demographic-only conditioning across all seven models; for GPT-4o, correlation increases from 0.624 to 0.667 and accuracy from 35.07% to 39.67% (Tables 2, 15–16). Comparing human-provided facets, model-written summaries, inferred facets, and personas without explicit demographics makes this more informative than a comparison of unrelated prompting recipes. These results could guide what information practitioners collect for simulation.

  2. The framework considers demographic structure and external usefulness alongside answer agreement. Comparing demographic accentuation with an empirical human reference is a useful diagnostic, and Section 4.3 explicitly distinguishes it from normative unfairness. The external experiment is also encouraging: full SCOPE conditioning achieves 0.584 group-modal accuracy versus 0.518 for demographics alone, while augmenting Nemotron improves its score from 0.496 to 0.533 (Table 3). This suggests practical value beyond the internal questionnaire, although the interpretation requires the qualifications below. Appendix B's finding that richer conditioning need not improve creative-response similarity is a useful acknowledgment that persona quality depends on the task.

Critical weaknesses
  1. The conditioning/evaluation separation is internally inconsistent, preventing verification of the central generalization claim. Sections 3.1 and 3.3 and Figure 3 put sociodemographic behavior, traits, and identity narratives in conditioning, while holding out values, behavioral patterns, professional identity, and creativity. However, Section 4.1's 72 evaluation items include sociodemographic behavior, and Table 7 evaluates personality traits. Table 10 instead says its first four facets are conditioning inputs, which includes values and excludes identity narratives. Section 4.3 further uses 134 evaluation questions without explaining their relationship to the 141-item instrument or the conditioning inputs. Table 14 illustrates value/goal and professional information in prompts, but its examples are explicitly synthetic; they do not establish what was actually exposed. An item-level manifest and actual templates are needed to distinguish genuine prediction from reconstruction of supplied information. Until this is resolved, the main performance gains cannot reliably support held-out behavioral fidelity.

  2. The reported metrics and statistical evidence do not yet establish the claimed structural fidelity. Equation (1) correlates answers across questions, but the instrument mixes nominal categories and ordinal scales without specifying their numerical encoding. Pearson correlation can change under arbitrary relabeling of nominal options, and shared question-level response tendencies can generate correlation without recovering differences between people. The no-persona baseline already obtains 0.448 (Table 2), making question-only and participant-shuffled controls especially relevant. No uncertainty is provided for the comparatively small correlation improvements. Likewise, the pairwise-similarity analysis reports significance without explaining how dependence between pairs sharing participants is handled. Its bias reporting also contains a verifiable inconsistency: Table 16 lists GPT-5.1 Case 1 with Δr = 0.116 and Bias% = 1.082, whereas Equation (7), using the stated human baseline of 0.132, gives approximately 87.9%. This cannot be explained by rounding. Negative bias denotes under-accentuation, so increasingly negative values also cannot automatically establish improved fidelity. Finally, Section 4.2 measures agreement with the matching group's modal answer; this does not test fidelity to the full response distributions supplied by SimBench. Consequently, the external result supports a narrower claim than general preservation of human response structure.

Questions for authors

. Clarification: Can you provide an item-level conditioning/evaluation manifest for every persona case and metric, with the actual prompt templates? Please reconcile Sections 3.1/3.3, Figure 3, Tables 7/10, and the 72 versus 134 evaluation-item counts. Does Case 5's “full SCOPE profile” contain only conditioning information? The answer directly determines whether the reported evaluation measures held-out prediction.

  1. Validation of current claims: Do the Case 4 versus Case 1 gains persist with question-only and participant-shuffled baselines, facet-specific scoring appropriate to nominal/ordinal responses, and paired participant-level uncertainty estimates? These checks would determine how much of the reported correlation reflects meaningful persona information rather than shared item structure or sampling variability.

  2. Clarification and validation of current claims: What exact human baseline, raw AI correlation, and normalization produced each bias entry in Tables 15–16? Please reconcile the numerical inconsistency above and specify participant-aware inference for the pairwise analysis. Do the conclusions about closeness to human demographic structure survive the corrected calculation, counting both over- and under-accentuation as deviations?

  3. Clarification and validation of current claims: How were the 441 SimBench items and matching demographic groups selected, what does “within 3 turns” mean, and how were SCOPE facets assigned when augmenting Nemotron? Are all baselines evaluated under the same procedure? Does the claimed external fidelity improvement hold when comparing response distributions, or should it be restricted to group-modal answer accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.