- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents SCOPE, a framework for constructing and evaluating user personas based on social psychological information. The authors collect 141 questionnaire responses from 124 US participants, compare simulation performance across conditions including demographic information, personality traits, identity narratives, and model-generated summaries, and conduct experiments over seven language models and SimBench. The paper argues that socio-psychological information improves behavioral simulation and reduces over-amplification of demographic disparities.
Critical strengths
-
The research question is practically meaningful. How to construct personas that reflect real behavioral differences is an important problem in user simulation and social computing. By jointly examining behavioral prediction and the over-amplification of demographic structures, the paper helps identify the limitations of building personas relying solely on demographic attributes.
-
The real-participant data and multi-dimensional experimental design carry research value. The paper collects questionnaires and identity narratives from 124 participants, providing an empirical foundation for comparing model outputs against real human behaviors. Comparisons across seven models, multiple persona construction strategies, and an external benchmark also yield a useful analytical framework for investigating the effects of different persona information.
Critical weaknesses
- Core results contain internal inconsistencies, and discrepancies between figures and tables reverse comparative conclusions across methods.
Figure 2 and Table 16 report markedly different correlation coefficients for the same models under identical persona conditions.
| Model | Case 7c: Figure 2 (approx.) | Case 7c: Table 16 | Case 4: Table 16 |
|---|---|---|---|
| DeepSeek R1 | 0.642 | 0.608 | 0.625 |
| GPT-5.1 | 0.640 | 0.612 | 0.621 |
| Qwen3 480b | 0.641 | 0.566 | 0.586 |
These discrepancies reverse the comparison between non-demographic personas and full conditioning: Figure 2 places Case 7c above Case 4 for all three models, whereas Table 16 places it below Case 4. This directly affects the claim in Section 4.1, point III (p. 6, lines 315–319), and Section 5 that non-demographic personas match or exceed full conditioning in correlation
- The partitioning between persona inputs and evaluation targets is self-contradictory, making it impossible to verify whether the core experiments truly employ held-out evaluation.
§3.1 explicitly lists “sociodemographic behaviors” as inputs for persona construction, and §3.3 states that Case 4 includes this information. Nevertheless, §4.1 incorporates items of this type into the 72 supposedly held-out evaluation questions without explaining whether item-level splitting was performed within this dimension. Similarly, §3.1 reserves “occupational identity” for evaluation, yet §3.3 explicitly states that Case 2 uses personal and occupational identity narratives. Appendix Table 14 raises further concerns: input examples for Case 3 and Case 4 include statements of values and goals, while §3.1 defines “values and motivations” as a held-out evaluation dimension.
These contradictions do not constitute definitive proof of answer leakage, but they prevent readers from verifying the input–target separation upon which the core experiments rely. If richer personas directly contain answers to some evaluation items, performance gains may stem from access to target information rather than superior behavioral generalization. To support its central claims, the paper must provide clear item-wise lists of inputs and evaluation items for each condition, including information accessible during summary generation, and report results only on verified non-overlapping targets.
Questions for authors
-
Why do Figure 2 and Table 16 report different correlations for Case 7c, reversing its comparison with Case 4 for DeepSeek R1, GPT-5.1, and Qwen3 480b? Which values are correct, and which conclusions remain supported after correction?
-
Could the authors provide the exact conditioning and evaluation item IDs for each persona case, including the inputs used for summary generation? Sections 3.1–4.1 give conflicting descriptions of whether sociodemographic behavior, professional identity, and values/goals are conditioning inputs or held-out targets. Do the reported gains persist after excluding any overlapping items?
-
Section 4.2 evaluates agreement with the most frequent response of a matching demographic group. How does this evaluation distinguish improved prediction of group-majority answers from faithful simulation of within-group behavioral variation? Do response-distribution comparisons support the claimed improvement in behavioral fidelity?