acceptodds
Under review as a conference paper at ICLR 2027

S2Bench: A Survey-Grounded Benchmark for LLM-Based Social Simulation

Abstract

LLM-driven social simulations enable controlled and repeatable experiments that would be costly, slow, or unethical with real people. Evaluating whether these simulations faithfully reflect human behavior remains challenging; existing work largely relies on real-world survey responses as quantifiable records of human choices. Yet current survey-based benchmarks have three limitations: they evaluate only static settings with fixed respondents and questions, lack a unified evaluation protocol, and remain narrow in coverage and scale. To address these limitations, we introduce S²Bench, a large-scale benchmark for survey response prediction. It defines four settings by crossing seen and unseen respondents with seen and unseen questions. We also provide a public evaluation platform that scores submitted methods under identical conditions and ranks them on a live leaderboard, and release the benchmark construction pipeline. Applied to 15 real-world survey datasets across 234 waves, the pipeline yields nearly 0.4 billion respondent–question–answer triples. We evaluate prompting- and training-based methods across 13 LLMs and find that even the strongest models reach only 40–55% accuracy. Although scaling and reasoning modes improve performance, the gains diminish quickly, suggesting that the ability to simulate a person has improved far more slowly than general abilities such as knowledge and reasoning. Fine-tuning transfers well to new respondents, but much less to new questions.

Primary area:
datasets and benchmarks
Keywords:
LLM-based social simulation, survey response prediction, social simulation benchmarks, human behavior simulation, LLM evaluation, survey data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LMqDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for predicting real survey respinses from a person's background and available context. It combines 15 surbey datasets and defines 4 seeings with seen or unseen respondents and questions. Experiments compare prompting and training-based methods and show that ft has a better transfer to new respondents than to new questions.

Critical strengths

s1: the dataset comes from real human response instead of llm generated. And the multiple annotators have evaluated.

s2: The preocess of extracting and filtering from the codebooks seems legit.

s3: The author benchmarks a good number of commonly used models and scaling study within qwen family. though it could be better with more close source frontier models.

Critical weaknesses

W1: I was wondering how much of the performance comes from actually simulating the person, and how much just comes from the pretaining data distribution. For example in PRA Evaluation option 0 is chosen by 89.3% of the training respondents, so simply picking the most common answer would already score high.

W2: I noticed that rhe prompt for the zero-shot baseline is different from the others. Zero-shot asks the model to “select the most appropriate option”, which makes it answer as itself, while persona asks it to predict how a specific person would answer. In my view these are two different perspectives, so the gain from Zero-shot to persona mixes the effect of the profile information with the change of task framing.

W3: The authors already shuffle the options to reduce position bias, but some biases may still remain. Asking the model to answer with the full option text, and adding paraphrased versions of the questions, could help remove these biases.

W4: “the ability to simulate a person has improved far more slowly than general abilities” The claim seems too strong. The low accuracy doesn't not neccessarily indicate low capability on simulation. The background infos could be insufficient, the survey response of human could also be not stable, maybe lower the tone.

Questions for authors

see Weaknesses

4 comments
AuthorsOpenReview ↗

Supplementary Empirical Results: Unseen Life-History Survey, Cross-Family Scaling & Option Ranking

Dear Reviewer LMqD,

Building on our earlier responses, we would like to share four supplementary sets of experiments.

(1) Independently Collected Life-History Survey Beyond Pretraining Corpora

To test model simulation without public-survey pretraining exposure, we collected an independent life-history questionnaire (62 respondents with detailed biographical profiles, 50 multiple-choice questions, and 2,898 valid response pairs) and evaluated 12 LLMs under HQA (profile + 8 historical answers, excluding the target item):

ModelOriginal 15-Survey ACC (Rank)Independent Life-History ACC (Rank)
DeepSeek Flash45.44% (2)45.93% (1)
MiMo Pro45.58% (1)45.65% (2)
DeepSeek-v4-pro45.17% (3)44.44% (3)
LongCat-2.044.92% (4)43.51% (4)
Qwen2.5-72B41.66% (8)43.10% (5)
Llama-3.3-70B43.50% (6)42.89% (6)

Across all 12 models, rankings on this unseen survey closely match the 15 public surveys (Spearman r = 0.804), with the exact same top-4 model families. Moreover, 4-split SFT on Qwen2.5-7B and Qwen3-8B on this dataset again transfers better to unseen respondents (+8.93 pp on US) than to unseen questions (+5.32 pp on SU).

(2) Preserving Individual Disagreement & Leveraging Relevant History

  • Inter-individual heterogeneity (15 surveys × 13 LLMs, 37,646 responses): Rather than collapsing to the modal option for everyone, Persona retains 77.64% of human response disagreement, and adding personal history (HQA) increases retention to 80.78% while reducing modal-option concentration from 65.68% to 64.23% (improving heterogeneity in 11 of 13 models).
  • Relevant personal history: On WVS with Qwen3-14B (k = 8), selecting the most semantically similar historical responses boosts accuracy from 46.93% (random history) to 57.13% (+10.20 pp), vs. 45.63% for least-similar history—confirming that models actively use relevant personal evidence.

(3) Expanding Scaling Studies Beyond Qwen2.5

We extended our scaling analysis to 9 additional models across Qwen3.5 Dense (0.8B–27B) and Gemma 3 (1B–27B) on WVS, ATP, BSA, and GSS:

  • Mean HQA accuracy rises from 29.99% (0.8B) to 44.33% (27B) for Qwen3.5 Dense and from 27.48% (1B) to 40.96% (27B) for Gemma 3, showing consistent cross-family scaling benefits across Zero-shot, Persona, and HQA.

(4) Top-K Option Ranking Under Human Response Instability

Because a respondent's underlying attitude may make multiple adjacent options plausible across repeated surveys (W4), we evaluated option ranking across 5,135 WVS Test-SS responses:

  • For DeepSeek-Flash and MiMo-v2.6-Flash, Recall@2 reaches 73.18% and 70.65% (vs. 53.01% random expectation), and Recall@3 on >3-option items reaches 76.23% and 74.86% (vs. 58.63% random). Increasing personal history from 4 to 12 items steadily improves Recall@K, MRR@K, and NDCG@K for both models.

We hope our initial responses and these additional experiments thoroughly address your comments, and we remain happy to provide any clarification before the discussion period concludes. If our responses have resolved your concerns, we would sincerely appreciate your reconsideration of our work.

Best regards,
The Authors

AuthorsOpenReview ↗

A Brief Follow-Up Before the Rebuttal Period Closes ☺

Dear Reviewer LMqD,

With 20 hours remaining in the rebuttal period, we would welcome any questions or concerns you may still have. Our entire author team is available and will make every effort to address them before the deadline.

If our responses and additional experiments have addressed your concerns, it would mean a great deal to us if you would reconsider your assessment and score in light of these results.

Thank you again for your time and thoughtful review.

Best regards,
The Authors

I appreciate the authors’ extensive additional experiments, which have addressed most of my concerns. However, W1 remains only partially addressed. It is still unclear to what extent the benchmark scores reflect individual simulation rather than population-level patterns. IMO, the substantial additions to the paper is already a major revision, and the overall quality would need a full read on the revised manuscript instead of across separate rebuttal responses to assess. Given those, I will maintain my score.

AuthorsOpenReview ↗

Clarification on W1 and the Scope of Revisions

Dear Reviewer LMqD,

Thank you for your further feedback. Regarding W1, we would like to clarify that the current manuscript already describes the input design: in both the Persona and Persona + history questions settings, each individual’s information is provided independently to the LLM. The prompts contain no population-level statistics, aggregate response distributions, or other individuals’ responses. The corresponding performance gains therefore reflect improvements in individual response prediction from adding personal profiles and historical responses, rather than gains from explicitly supplying population-level statistics.

We also understand your preference for assessing the additions together in a revised manuscript. We would like to clarify that these experiments primarily explain and validate the existing methodology: the input design relevant to W1 is already described in the manuscript, the supplementary checks for W2 and W3 support the original conclusions, and the overly strong statement identified in W4 will be revised as you suggested. We therefore believe that these changes do not constitute a major revision of the methodology or central conclusions, and hope that the evidence already available in the manuscript and rebuttal can help you assess these specific concerns.

In light of these clarifications and additional evidence, we respectfully ask whether you would reconsider your assessment. Your feedback and reconsideration would mean a great deal to us as we improve this work and strive to contribute useful resources to an open and inclusive social simulation research community.

Thank you again for your time and thoughtful consideration.

Best regards, The Authors

Reviewer_aQfXOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark framework for systematically evaluating and comparing the ability of LLMs to simulate human survey respondents. Prior works have created static evaluations S2Bench is the first resource that continually updates and tests new models against a wide and growing set of survey questions. Beyond creating the infrastructure for this comparison, they provide results from 13 LLMs and show that payoffs to scale and reasoning plateau, and even the best models still fare poorly at respondent simulation.

[note: I am raising my rating from weak acceptance to clear acceptance during rebuttal]

Critical strengths

S2Bench meets an important need in the field of social simulation. Many individual studies have evaluated survey simulation capabilities but they have not created lasting frameworks that can be used for ongoing comparisons and benchmarking. The authors are also thoughtful about difficulties associated with question extraction, rewording, and normalization.

Critical weaknesses

The primary success metric presented is accuracy, which is operationalized as exacting matching the human's response. Survey responses among humans are highly stochastic, and are often modeled as draws from a latent probability distribution. Exact match is also a poor choice for accuracy because LLMs have a known tendency to exhibit much lower variance in survey responses than humans. Thus, they may score relatively high in exact match by always selecting the modal response, resulting in poor distributional simulation.

I encourage the authors to lean more heavily on measures of distributional similarity. KL divergence and Earth Mover’s distance are listed in the Appendix, but are not discussed in the text. Some further discussions and comparisons of these measures of distributional similarity would be appreciated, because without baseline comparisons it is difficult to interpret how good or bad an EMD of 0.330 is. Mean, median, and standard deviations would be valuable to compare in addition measures of distributional similarity like KL or EMD. It would also be useful to break down distributional similarities by socio-demographic subgroups to see if the heterogeneity of some groups is captured better than others.

Finally, it is hard to even assess how bad 40-55% accuracy is. Accuracy under exact match is strongly affected by the number of response categories. Also, test-retest reliability for the same respondent over a short time period should be considered a ceiling for accuracy. (Park et al. 2024 report that for questions from the General Social Survey repeated 2-weeks apart, test-retest reliability was 81.25%).

Park, J. S., Zou, C. Q., Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Willer, R., Liang, P., & Bernstein, M. S. (2024). Generative agent simulations of 1,000 people [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2411.10109.

Questions for authors

Under HQA, responses to other questions are provided in context. How are questions selected to be included in context? How informative one question’s response is for another would vary greatly between questions.

3 comments
AuthorsOpenReview ↗

Supplementary Empirical Results: Independent Life-History Survey, Calibration & Pipeline Audits

Dear Reviewer aQfX,

Thank you again for your positive assessment of S²Bench as an ongoing benchmarking framework. To complement our earlier distributional, subgroup, and option-ranking analyses, we would like to share three additional sets of results from our broader evaluation.

(1) Applying S²Bench to an Independently Collected Life-History Survey

To verify how S²Bench generalizes to newly collected, non-public questionnaires, we applied our pipeline and protocol to an independent life-history dataset (62 respondents with biographical profiles, 50 multiple-choice items, and 2,898 valid response pairs):

  • Consistent model rankings: Evaluating 12 LLMs under HQA yields accuracies of 38.34%–45.93% and a 0.804 Spearman correlation with rankings on the 15 public surveys (with DeepSeek Flash, MiMo Pro, DeepSeek-v4-pro, and LongCat-2.0 remaining the top four).
  • Consistent SFT & reasoning behavior: Fine-tuning Qwen2.5-7B and Qwen3-8B across the four visibility splits yields larger gains on unseen respondents (+8.93 pp on US) than on unseen questions (+5.32 pp on SU), while enabling thinking on Qwen3-32B improves HQA accuracy by +2.46 pp at a 31.63× runtime cost (consistent with +1.41 to +2.26 pp at 19.6×–29.7× runtime across ATP, BSA, and GSS).

(2) Probability Calibration (ECE, NLL, Brier) Under Population Distribution Shift

Alongside KL, EMD, and subgroup distributions, we evaluated top-choice probability calibration on WVS across five test-population shift levels (L0 to L4):

MetricL0 (JSD=0.0004)L1 (0.0765)L2 (0.1526)L3 (0.2285)L4 (0.3048)
Expected Calibration Error (ECE ↓)0.11070.22000.30290.36580.3932
Negative Log-Likelihood (NLL ↓)1.25541.43061.53321.59141.5496
Brier Score ↓0.63770.71760.79190.85130.8477

Calibration degrades substantially as target populations diverge from training distributions (ECE rising from 0.1107 to 0.3932).

(3) Subsampling Rank Stability & Multilingual Pipeline Audits

  • 500-trial subsampling stability: Across 500 random respondent-subsampling trials over the 15 surveys, 13-model rankings remain highly consistent—retaining 75% of respondents yields mean Spearman correlations of 0.986–0.995 across Persona, HQA, PRA, and DACA (and 0.963–0.975 even when retaining only 25%).
  • Multilingual manual audits (retained + excluded items): Three PhD researchers audited both retained items and exclusion decisions on Chinese CGSS 2023 (184 initial → 68 retained) and English Afrobarometer 2023 (373 initial → 212 retained), confirming a 100% pass rate with zero unjustified omissions.

We hope these supplementary results further support your confidence in S²Bench, and we would be glad to address any remaining questions during the discussion period.

Best regards,
The Authors

Thank you, I appreciate the extra analyses you have done, especially those that compare response heterogeneity between human and simulated respondents. I will rate my rating to "accept."

AuthorsOpenReview ↗

Thank You for Your Support (^_^)

Dear Reviewer aQfX,

Thank you for recognizing the value of our work. Your support means a great deal to us and encourages us to work with our colleagues to build a more inclusive and thriving social simulation research community.

Thank you again for your support and encouragement.

Best regards,
The Authors

Reviewer_5Ma6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces S²Bench, a large-scale benchmark for evaluating LLMs' ability to predict human survey responses. Its contributions include 381 million response triples from 15 survey families and 234 waves, four evaluation settings covering seen/unseen respondents and questions, an LLM-assisted survey-processing pipeline, and a public leaderboard.

Experiments across 13 LLMs and multiple prompting and fine-tuning strategies reveal 40–55% prediction accuracy, diminishing returns from scaling, modest reasoning benefits, and weaker generalization to unseen questions than respondents.

Critical strengths
  1. Scale and breadth: The aggregation of 381 million responses across 15 survey families, 234 waves, multiple countries, and diverse topics represents a substantial data-engineering contribution.
  2. Generalization framework: The four-way split distinguishing seen/unseen respondents and questions is interesting. It shows that models generalize substantially better to new respondents than to new questions.
  3. Reproducibility: The standardized processing pipeline, evaluation interface, and public leaderboard provide useful infrastructure for future research. 4.The evaluation covers several paradigms including: prompting, contextual conditioning, fine-tuning, model scaling, etc yielding several interesting findings,
Critical weaknesses
  1. Limited novelty of the generalization framework. The four evaluation settings are useful but represent a straightforward Cartesian product of seen/unseen respondents and questions, rather than a novel formulation of social simulation. The paper's primary contribution is therefore benchmark construction and standardization, and its methodological novelty should be positioned accordingly.

  2. Inadequate evaluation of social simulation. The benchmark primarily measures exact survey-response accuracy, which does not establish whether models reproduce individual preferences, subgroup disagreement, or population-level heterogeneity. For example, if 70% of respondents support a policy, a model predicting “support” for everyone achieves 70% accuracy while completely failing to capture disagreement between demographic groups. Conversely, a model reproducing the correct 70/30 population distribution could receive lower exact-match accuracy. The authors should evaluate subgroup-level fidelity, disagreement preservation, response correlations, and calibration.

  3. Missing statistical baselines. The absence of simple majority-class, demographic-conditioned, logistic regression, and collaborative-filtering baselines makes the reported 40–55% accuracy difficult to interpret. For example, the paper includes a question where 89.3% of respondents consider family “Very important.” Predicting this response for everyone would achieve 89.3% accuracy without any personalization. Per-question majority and demographic-conditioned baselines are particularly important for determining whether LLMs learn meaningful respondent characteristics.

  4. Benchmark scale is not reflected in experimental coverage. Despite containing 381 million responses across 234 waves, the main evaluation uses only the latest wave from each survey family and samples 500 respondents per wave. Consequently, the representativeness and robustness of the empirical conclusions remain uncertain.

  5. Insufficient validation of the automated construction pipeline. The reported 100% human validation is based on a single WVS wave, 251 retained questions, and 200 profiles. More importantly, auditing only retained questions cannot detect extraction omissions. For example, if the pipeline incorrectly discards a question containing complex skip logic, that error would never appear in an audit of successfully extracted questions. Validation should include raw source documents, rejected questions, multiple languages, and diverse survey families.

  6. Generalization settings do not reflect realistic deployment. Randomly holding out respondents and questions within the same survey wave primarily measures within-distribution generalization. For example, predicting responses from withheld participants in a 2024 survey does not establish whether a model trained on 2020 data can predict attitudes in 2025. Similarly, withholding a question about immigration may not test genuine question generalization if nearly identical immigration questions remain in training. Temporal, geographic, and semantic-cluster holdouts would provide stronger evidence.

  7. Survey responses are not necessarily stable across repeated administrations. For example, the same respondent might report being “somewhat satisfied” with democracy on one occasion and “not very satisfied” a week later, despite no meaningful change in underlying beliefs. A model predicting the respondent's underlying attitude could therefore be penalized for failing to reproduce a noisy realization. Without test–retest reliability or an empirical predictability ceiling, the reported 40–55% accuracy is difficult to interpret.

  8. Overgeneralized scaling and reasoning conclusions. The evidence does not fully support broad claims about scaling and reasoning in social simulation. For example, Qwen2.5 Persona accuracy increases from 27.6% at 0.5B to 40.7% at 14B, but reaches only 41.9% at 72B. This demonstrates diminishing returns within that configuration, not necessarily a general scaling law. Similarly, Qwen3-32B reasoning improves accuracy from 43.35% to 44.86% while increasing evaluation time approximately 27×, but this experiment is limited to WVS Test-SS.

  9. Training-data contamination. Public survey questionnaires and response distributions may have appeared in model pretraining data. For example, a model familiar with published World Values Survey statistics could exploit population-level response frequencies without learning to simulate individuals. Such exposure need not produce near-perfect accuracy to affect rankings by 1–3 percentage points. Hidden leaderboard labels do not address contamination originating from publicly available survey datasets.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Supplementary Empirical Results: Extended Statistical/IRT Baselines, Option-Stratified ACC & Prompt Controls

Dear Reviewer 5Ma6,

To provide a more complete empirical picture on statistical baselines (W3), the interpretation of 40–55% accuracy (W7), and evaluation robustness across prompt designs, we would like to share three additional sets of experimental results.

(1) Additional Psychometric, Matrix-Factorization & Tree Baselines Under Shift (W3)

Complementing POP, Group POP, User KNN, and NCF, we evaluated four additional baselines on WVS across the five distribution-shift levels (L0–L4): Group Sample (demographic-conditional sampling), Categorical MF (matrix factorization), Nominal IRT (item response theory), and XGBoost:

BaselineL0 (JSD=0.0004)L1 (0.0765)L2 (0.1526)L3 (0.2285)L4 (0.3048)
Group Sample37.96%33.92%30.11%27.45%27.55%
Categorical MF43.21%38.27%33.62%30.54%30.42%
Nominal IRT48.73%40.08%31.83%24.93%23.92%
XGBoost48.67%40.92%33.71%28.10%28.01%

Like POP (48.91% → 14.68%), Nominal IRT and XGBoost reach ~48.7% at L0 when test distributions mirror training statistics, but degrade sharply to 23.92%–28.01% under population shift, and cannot be applied directly when target questions are globally unseen (Test-SU / Test-UU).

(2) Stratifying 40–55% Accuracy by Option Cardinality (37,679 Responses; W7)

Stratifying 13-LLM Test-SS accuracy across all 15 surveys by option count contextualizes the 40–55% range against uniform chance:

Option CountValid PairsRandomZero-shotPersonaHQAPRADACA
25,76650.00%59.55%64.24%66.15%67.76%68.85%
35,25233.33%47.94%51.64%53.81%53.51%54.99%
412,37025.00%35.35%37.11%40.07%40.12%41.68%
59,35920.00%29.91%31.90%34.78%35.27%36.16%
6–104,15413.85%27.66%32.19%33.22%35.82%36.35%
11–127789.09%13.29%14.08%14.58%17.49%17.09%

(3) Prompt-Framing, Output-Format & Paraphrase Controls

Across 20 models × 15 surveys × 4 splits, aligning Zero-shot wording to third-person prediction confirms that adding respondent profiles (Persona) yields a statistically significant +1.64 pp gain under both respondent–question and dataset-level clustering (Holm-adjusted p < 0.001 and p = 0.0051). Likewise, testing 3 output formats (numeric, full-text, numeric+text; 7 LLMs) and 3 question paraphrases (6 LLMs) on WVS yields no significant accuracy shifts after Holm correction (all adjusted p = 1.0).

We hope our initial responses and these supplementary results help clarify the empirical scope and rigor of S²Bench, and we welcome any remaining questions before the discussion window closes. If our responses have addressed your concerns, we would sincerely appreciate your reconsideration of our work.

Best regards,
The Authors

AuthorsOpenReview ↗

A Brief Follow-Up Before the Rebuttal Period Closes ☺

Dear Reviewer 5Ma6,

Thank you again for your thoughtful review. We have made every effort to address the questions and concerns you raised.

With 20 hours remaining in the rebuttal period, please let us know if anything still needs clarification. Our entire author team is available and will do our utmost to respond before the deadline.

If our responses have addressed your concerns, it would mean a great deal to us if you would reconsider your assessment and score.

Best regards,
The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

📌 Quick Navigation Index to Reviewer Threads & Responses — Feedback Welcome

Dear Area Chair and Reviewers,

As our rebuttal discussion now spans over 35 comments with detailed experiments and analyses, we provide direct links below to help you locate each review thread and all corresponding responses. We would appreciate hearing whether these responses address your concerns and whether any questions remain.

  • 👉 Jump to Reviewer LMqD’s Thread: Point-by-point responses to W1–W4 covering personalization versus population-level response frequencies, matched Zero-shot/Persona prompt framing, output-format and question-paraphrase controls, and the scope of conclusions about simulation capability. Supplementary results: Evaluation on an independently collected life-history survey, preservation of interpersonal disagreement, relevant-history retrieval comparisons, cross-family scaling with Qwen3.5 and Gemma 3, and top-K option-ranking analyses.

  • 👉 Jump to Reviewer aQfX’s Thread: Point-by-point responses to W1–W3 & Q1 covering exact-match accuracy and response heterogeneity; distributional metrics and demographic-subgroup fidelity; interpretation of accuracy under varying option counts and human response instability; and historical-question selection in HQA. Supplementary results: An independently collected life-history survey, probability calibration under population shift (ECE, NLL, and Brier score), 500-trial subsampling rank-stability tests, and multilingual pipeline audits.

  • 👉 Jump to Reviewer 5Ma6’s Thread: Point-by-point responses to W1–W9 covering benchmark positioning; individual, subgroup, and distributional fidelity; statistical and collaborative-filtering baselines under population shift; evaluation coverage and subsampling stability; multilingual audits of retained and excluded items; deployment-oriented generalization; response instability; scaling and reasoning; and potential pretraining contamination. Supplementary results: Extended psychometric and statistical baselines (Nominal IRT, Categorical MF, XGBoost, and demographic-conditional sampling), option-count-stratified accuracy, and prompt-framing, output-format, and paraphrase controls.

With the discussion period nearing its close, we would be grateful for a brief follow-up on any remaining concerns so that we can address them while there is still time for discussion. We are happy to provide further clarification.

Thank you again for your time and thoughtful engagement!

Best regards,
The Authors

AuthorsOpenReview ↗

Final 24-Hour Rebuttal Summary: Addressed Concerns & Request for Final Feedback

Dear Area Chair and Reviewers,

With less than 24 hours remaining in the discussion period, we provide a concise summary table mapping every reviewer concern (LMqD, aQfX, 5Ma6) to our targeted empirical resolutions posted in this thread.

ReviewerCore Concern / QuestionSummary of Our Resolution & Key Quantitative Evidence
All Reviewers (5Ma6 W2/W3, aQfX W1/W2, LMqD W1)Distinguishing individual personalization from population priors & evaluating distributional/subgroup fidelity(1) Individual heterogeneity (37,646 responses, 15 surveys × 13 LLMs): Persona retains 77.64% of human response disagreement, and adding personal history (HQA) raises this to 80.78% (improving in 11/13 models) while reducing modal-option concentration. (2) Subgroup & distributional metrics: Reported KL, EMD, mean, median, and SD across gender, age, and education subgroups, plus calibration (ECE, NLL, Brier) across 5 distribution-shift levels (L0–L4). (3) 10 statistical/psychometric baselines: POP, Group POP, User KNN, NCF, Nominal IRT, and XGBoost reach ~48.7%–49.1% at L0 when test distributions mirror training data, but collapse to 14.68%–28.01% under shift (L4) and cannot handle globally unseen questions (Test-SU / Test-UU).
Reviewers 5Ma6 & aQfX (5Ma6 W7, aQfX W3)Contextualizing 40–55% accuracy under varying option counts & human response instability(1) Option-cardinality stratification (37,679 responses): Accuracy scales cleanly with option count (e.g., DACA reaches 68.85% on 2-option items vs. 50.00% chance, and 17.09% on 11–12 options vs. 9.09% chance). (2) Option-ranking evaluation: Evaluating top-K option ranking (Recall@K, MRR@K, NDCG@K) on 5,135 WVS items shows DeepSeek-Flash reaches 73.18% Recall@2 (vs. 53.01% random) and 76.23% Recall@3 on >3-option items (vs. 58.63% random), improving steadily as personal history increases from 4 to 12 items.
Reviewer aQfX (Q1)Impact of historical-question selection in HQAEvaluated most-similar-first, random, and least-similar-first retrieval across 11 history budgets (k = 1 to All) on WVS (Qwen3-14B): at k = 8, semantic retrieval boosts accuracy by +10.20 pp over random selection (57.13% vs. 46.93%; 45.63% for least-similar), confirming models actively leverage relevant personal evidence.
Reviewer 5Ma6 (W1, W4, W5, W6)Benchmark positioning, subsampling stability, multilingual pipeline audits, & deployment splits(1) Unified 4-split formulation: Clarified why globally holding out questions (Test-SU/Test-UU) prevents target-distribution leakage and exposes the SFT transfer gap (+8.93 pp on unseen respondents vs. +5.32 pp on unseen questions). (2) 500-trial subsampling stability: Mean Spearman rank correlations across the 13 LLMs reach 0.986–0.995 at 75% retention (and 0.963–0.975 at 25% retention). (3) Multilingual audits of retained & excluded items: Three PhD annotators audited 557 raw items across Chinese CGSS 2023 and English Afrobarometer 2023, confirming a 100% pass rate with zero unjustified skip-logic omissions. (4) Deployment splits: Demonstrated how temporal, geographic, and semantic-cluster splits instantiate directly within our visibility protocol.
Reviewers LMqD & 5Ma6 (LMqD W2/W3/W4, 5Ma6 W8/W9)Prompt/format/paraphrase controls, cross-family scaling, claim calibration, & pretraining contamination(1) Prompt, format & paraphrase controls: Aligning Zero-shot task wording confirms Persona adds a significant +1.64 pp gain (Holm-adjusted p<0.001p < 0.001 and p=0.0051p = 0.0051); testing 3 output formats and 3 paraphrases shows no significant accuracy drift (all Holm-adjusted p=1.0p = 1.0). (2) Cross-family scaling & reasoning: Evaluated 9 models across Qwen3.5 Dense (0.8B–27B) and Gemma 3 (1B–27B) plus multi-dataset Qwen3-32B thinking cost–benefit trade-offs, and tempered our comparative capability-growth claims. (3) Independent life-history survey: Collected an unseen dataset (62 respondents, 50 items, 2,898 pairs) across 12 LLMs, yielding a 0.804 Spearman rank correlation with the 15 public surveys and replicating our SFT and reasoning findings.

All detailed tables, statistical tests, and expanded analyses are posted in our thread responses above and will be fully incorporated into the revised manuscript.

As the discussion window closes in less than 24 hours, we would be sincerely grateful if Reviewers LMqD, aQfX, and 5Ma6 could briefly confirm whether these responses address your concerns and consider updating your assessments accordingly. We remain actively online and ready to answer any final questions immediately.

Thank you very much for your time, guidance, and thoughtful consideration!

Best regards,
The Authors

AuthorsOpenReview ↗

A Brief Follow-Up on Submission ☺

Dear Area Chair and Reviewers,

With less than 12 hours remaining in the rebuttal period, we would welcome any further questions. We have done our best to address the concerns raised, and our entire author team remains available to respond before the deadline.

We respectfully ask the Area Chair to take our responses into account when assessing the paper. If time permits, we would also be sincerely grateful if the Area Chair could send the reviewers a brief reminder to look over our replies while there is still time for discussion.

If our replies have addressed your concerns, we would be grateful if you would reconsider your assessments. Thank you again for your time and thoughtful review.

Best regards,
The Authors

AuthorsOpenReview ↗

Sincere Appreciation as the Discussion Period Concludes

Dear Area Chair and Reviewers,

As the discussion window comes to a close, we wish to express our sincere appreciation to all of you for investing your valuable time and thoughtful effort in evaluating our work.

First, we are deeply grateful to Reviewer aQfX for taking the time out of a busy schedule to review our additional analyses, recognizing the value of our work as an ongoing benchmarking infrastructure, and raising your rating to Clear Acceptance—your encouragement truly means a great deal to us. At the same time, we warmly thank Reviewer LMqD for providing follow-up feedback and acknowledging that our additional efforts have resolved most of your concerns; these targeted validations further reinforce our existing framework design and core findings, and we will make sure to smoothly and carefully integrate the corresponding clarifications and refinements into our revised manuscript as planned. Furthermore, we fully understand that Reviewer 5Ma6 has a packed schedule during this busy end-of-semester and deadline season and regrettably has not had the opportunity to engage further in the discussion thread; although many of the insightful questions you raised in your initial review are inherently challenging and very difficult to resolve within a short timeframe, we have genuinely tried our very best to explore and address every single point as thoroughly as possible to make this work more rigorous and solid.

Finally, we sincerely thank the Area Chair for carefully stewarding the entire review process, and we respectfully ask that you kindly take our supplementary empirical analyses and responses from the discussion period into account during your final assessment.

Thank you once again for your generous dedication to building an open and rigorous research community, and we wish you all the very best!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.