- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for predicting real survey respinses from a person's background and available context. It combines 15 surbey datasets and defines 4 seeings with seen or unseen respondents and questions. Experiments compare prompting and training-based methods and show that ft has a better transfer to new respondents than to new questions.
Critical strengths
s1: the dataset comes from real human response instead of llm generated. And the multiple annotators have evaluated.
s2: The preocess of extracting and filtering from the codebooks seems legit.
s3: The author benchmarks a good number of commonly used models and scaling study within qwen family. though it could be better with more close source frontier models.
Critical weaknesses
W1: I was wondering how much of the performance comes from actually simulating the person, and how much just comes from the pretaining data distribution. For example in PRA Evaluation option 0 is chosen by 89.3% of the training respondents, so simply picking the most common answer would already score high.
W2: I noticed that rhe prompt for the zero-shot baseline is different from the others. Zero-shot asks the model to “select the most appropriate option”, which makes it answer as itself, while persona asks it to predict how a specific person would answer. In my view these are two different perspectives, so the gain from Zero-shot to persona mixes the effect of the profile information with the change of task framing.
W3: The authors already shuffle the options to reduce position bias, but some biases may still remain. Asking the model to answer with the full option text, and adding paraphrased versions of the questions, could help remove these biases.
W4: “the ability to simulate a person has improved far more slowly than general abilities” The claim seems too strong. The low accuracy doesn't not neccessarily indicate low capability on simulation. The background infos could be insufficient, the survey response of human could also be not stable, maybe lower the tone.
Questions for authors
see Weaknesses
4 comments
A Brief Follow-Up Before the Rebuttal Period Closes ☺
Dear Reviewer LMqD,
With 20 hours remaining in the rebuttal period, we would welcome any questions or concerns you may still have. Our entire author team is available and will make every effort to address them before the deadline.
If our responses and additional experiments have addressed your concerns, it would mean a great deal to us if you would reconsider your assessment and score in light of these results.
Thank you again for your time and thoughtful review.
Best regards,
The Authors
I appreciate the authors’ extensive additional experiments, which have addressed most of my concerns. However, W1 remains only partially addressed. It is still unclear to what extent the benchmark scores reflect individual simulation rather than population-level patterns. IMO, the substantial additions to the paper is already a major revision, and the overall quality would need a full read on the revised manuscript instead of across separate rebuttal responses to assess. Given those, I will maintain my score.
Clarification on W1 and the Scope of Revisions
Dear Reviewer LMqD,
Thank you for your further feedback. Regarding W1, we would like to clarify that the current manuscript already describes the input design: in both the Persona and Persona + history questions settings, each individual’s information is provided independently to the LLM. The prompts contain no population-level statistics, aggregate response distributions, or other individuals’ responses. The corresponding performance gains therefore reflect improvements in individual response prediction from adding personal profiles and historical responses, rather than gains from explicitly supplying population-level statistics.
We also understand your preference for assessing the additions together in a revised manuscript. We would like to clarify that these experiments primarily explain and validate the existing methodology: the input design relevant to W1 is already described in the manuscript, the supplementary checks for W2 and W3 support the original conclusions, and the overly strong statement identified in W4 will be revised as you suggested. We therefore believe that these changes do not constitute a major revision of the methodology or central conclusions, and hope that the evidence already available in the manuscript and rebuttal can help you assess these specific concerns.
In light of these clarifications and additional evidence, we respectfully ask whether you would reconsider your assessment. Your feedback and reconsideration would mean a great deal to us as we improve this work and strive to contribute useful resources to an open and inclusive social simulation research community.
Thank you again for your time and thoughtful consideration.
Best regards, The Authors
Supplementary Empirical Results: Unseen Life-History Survey, Cross-Family Scaling & Option Ranking
Dear Reviewer LMqD,
Building on our earlier responses, we would like to share four supplementary sets of experiments.
(1) Independently Collected Life-History Survey Beyond Pretraining Corpora
To test model simulation without public-survey pretraining exposure, we collected an independent life-history questionnaire (62 respondents with detailed biographical profiles, 50 multiple-choice questions, and 2,898 valid response pairs) and evaluated 12 LLMs under HQA (profile + 8 historical answers, excluding the target item):
Across all 12 models, rankings on this unseen survey closely match the 15 public surveys (Spearman r = 0.804), with the exact same top-4 model families. Moreover, 4-split SFT on Qwen2.5-7B and Qwen3-8B on this dataset again transfers better to unseen respondents (+8.93 pp on US) than to unseen questions (+5.32 pp on SU).
(2) Preserving Individual Disagreement & Leveraging Relevant History
(3) Expanding Scaling Studies Beyond Qwen2.5
We extended our scaling analysis to 9 additional models across Qwen3.5 Dense (0.8B–27B) and Gemma 3 (1B–27B) on WVS, ATP, BSA, and GSS:
(4) Top-K Option Ranking Under Human Response Instability
Because a respondent's underlying attitude may make multiple adjacent options plausible across repeated surveys (W4), we evaluated option ranking across 5,135 WVS Test-SS responses:
We hope our initial responses and these additional experiments thoroughly address your comments, and we remain happy to provide any clarification before the discussion period concludes. If our responses have resolved your concerns, we would sincerely appreciate your reconsideration of our work.
Best regards,
The Authors