acceptodds
Under review as a conference paper at ICLR 2027

Learning to Share: Query-Efficient Reconstruction of LLM-Simulated Population Responses

Abstract

Simulating how heterogeneous populations respond to alternative policies with large language models (LLMs) requires evaluating a costly Cartesian product of personas and interventions. We introduce the new zero-history task of population-response reconstruction: given n personas, m policies, and a budget of B ≪ nm teacher queries, the goal is to reconstruct the response distribution for every persona–policy pair. We construct a hierarchical benchmark of 1,024 World Values Survey personas and 600 COFOG-based policies—614,400 pairs per teacher—spanning domains, families, and implementation variants. Diagnostics across four teacher models yield two findings: response sharing is compact but policy-conditioned, and informative queries derive much of their value from improving unobserved pairs. We introduce SCOPE (Shared-response Calibration for Online Population Emulation), a closed-loop framework that calibrates response sharing across personas and policies from queried answers, then uses the calibrated structure to select queries for population-wide reconstruction. Across four teachers and all evaluated query budgets, SCOPE achieves the lowest three-seed mean full-pool error at every reconstruction level. Code, benchmark data, and complete results are available at https://anonymous.4open.science/r/scope-F646/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based population simulation, active learning, query-efficient response reconstruction, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_mkksOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is related to active learning, and studies how a model can reconstruct an LLM’s responses across a large set of persona-policy pairs (i.e., LLM persona + candidate policy combinations, in the paper the example was 1,024 times 600) using a limited query budget. Its method, SCOPE, learns how responses transfer across personas and policies, then selects persona + policy queries expected to improve predictions across the full pool. Experiments use profiles drawn from the World Values Survey and LLM generated policy responses. SCOPE consistently outperforms the baselines.

Critical strengths
  1. The problem is clearly defined which helps understanding.
  2. Good flow: diagnostics motivate the design. Section 2 shows that main effects explain about 86 -- 95% of response variance and that ~ 80% of the gain comes from oracle selection lands on cells other than the queried. These findings are directly relating to the two components of the method.
Critical weaknesses
  1. The paper motivates through the practical problem of estimating population responses to a candidate policy, but evaluates only agreement with an LLM teacher. This, in my opinion, is a huge gap between what’s stated in Introduction and what’s actually carried out. Real survey profiles do not validate the teacher’s ‘simulated’ opinions. Without comparison against human responses (I think at least a limited set of matched policies is necessary) it remains unclear whether the reconstructed result captures meaningful population heterogeneity or just reproduces LLM artifacts. In other words, the ‘teacher’ should be validated at least to some extent, otherwise why would practitioners want a cheap reconstruction of it?

  2. Missing closely matched baselines. Comparisons with structured matrix completion or multi-task kernel methods would much help establish how much SCOPE adds beyond standard approaches that already uses person and policy structure. It is also unclear how the experiment will perform on the scenarios that practitioners would actually care about, and confounded by the weakness point 1.

Questions for authors

Please refer to the weakness section.

7 comments
AuthorsOpenReview ↗

Response to Weakness 1 (1/2): Matched-human validation and teacher agreement

Thank you for your constructive review and recognition of our clear problem formulation. We address your concerns point by point below.

Question 1: “Please refer to the weakness section.”

Addressed under Weaknesses 1–2 below.

Weakness 1: “Without comparison against human responses (I think at least a limited set of matched policies is necessary) it remains unclear whether the reconstructed result captures meaningful population heterogeneity or just reproduces LLM artifacts.”

We validate teachers against matched human survey questions and demographic differences.

Data and selection. We use US respondents from World Values Survey Wave 7 (WVS7) and Pew Research Center's American Trends Panel Wave 54 (Pew ATP W54), released through OpinionQA (Santurkar et al., ICML 2023). WVS data · OpinionQA paper · Data release.

From English-interviewed respondents in each survey, we sample 1,024 prediction personas without replacement, proportionally within sex × age strata. Before teacher collection, we fix complete batteries of policy, institutional, social and economic questions with verified ordered scales. We retain original wording and four-, five- or ten-category options. All 59 selected WVS and 60 selected W54 questions are evaluated, forming 60,416 and 61,440 pairs.

  • WVS7: institutional confidence, Q64–Q89 excluding Q71 (25 questions); justifiability, Q177–Q195 (19); democracy, Q241–Q249 (9); social/work attitudes, Q36–Q41 (6). Q71 is excluded because it is already a persona input.
  • OpinionQA W54: government priorities, GOVPRIORITYa–f (6); influences on economic opinions, ECON3_a–i (9); financial impact, ECON4_a–i plus INEQ6/INEQ7 (11); inequality causes, INEQ5_a–m (13); policy effectiveness, INEQ8_a–j (10); economic conditions' effects on groups, ECON5_a–k (11). All identifiers have the suffix _W54; wording and options follow the original Pew questionnaire.

We also test stronger, higher-query-cost API teachers, illustrating the value of query savings: gpt-6.1-terra and DeepSeek-V4.1-Flash generate JSON probability vectors, normalized by their positive sum.

Metrics. Lower TV and higher accuracy indicate better agreement. Accuracy averages whether the highest-probability option matches each valid respondent–question answer; ties use the first original option. The uniform reference assigns equal probabilities to legal options; its accuracy is the expected accuracy of uniform random choice, 1/K for K options.

Table Rmkks-1-1. Teacher–human agreement on WVS and OpinionQA.

TeacherWVS: TV ↓WVS: accuracy ↑OpinionQA: TV ↓OpinionQA: accuracy ↑
Qwen2.5-3B-Instruct0.433237.04%0.421130.48%
Qwen3.5-4B0.290236.59%0.280038.08%
Gemma-4-E4B0.411939.33%0.362640.63%
Qwen3.8-27B0.259742.00%0.196540.01%
gpt-6.1-terra0.183044.17%0.151341.99%
DeepSeek-V4.1-Flash0.186143.97%0.140943.13%
Uniform over legal options0.368817.36%0.261424.22%

Both API teachers and Qwen3.8-27B beat uniform in TV on both surveys; all six teachers beat it in accuracy. Terra and DeepSeek reduce TV by 42.12%–50.39% relative to uniform. Qwen3.8 has the lowest TV among local teachers. From Qwen2.5-3B to Qwen3.8-27B, WVS/OpinionQA accuracy rises from 37.04%/30.48% to 42.00%/40.01%; Terra reaches 44.17% on WVS and DeepSeek 43.13% on OpinionQA. These results support improved opinion simulation with the stronger teachers evaluated.

AuthorsOpenReview ↗

Response to Weakness 1 (2/2): Demographic heterogeneity and supporting research

Demographic heterogeneity: pairwise ordering accuracy and Spearman correlation. We compare teacher and human group means for age, education and income, using the same retained groups and each question's verified ordered scale. Category counts are 4/6/7 for WVS and 4/5/5 for OpinionQA. Within each question and attribute, all unordered group pairs score 1 for a correct direction, 0 for a reversal and 0.5 for a model tie; human ties are excluded. Differences in normalized means ≤10⁻¹⁰ count as ties; none occur here. Identical predictions for all groups score 50%.

We average pair scores within each question, then questions equally; the last column averages attributes equally. Spearman correlation uses the same group means and aggregation. Cells in Tables Rmkks-1-2 and Rmkks-1-3 report ordering accuracy / Spearman correlation; both are higher-is-better.

Table Rmkks-1-2. Demographic ordering on all 59 selected WVS7 questions.

TeacherAgeEducationIncomeThree-attribute mean
Qwen2.5-3B-Instruct58.5% / 0.18640.8% / -0.24051.3% / 0.02450.2% / -0.010
Qwen3.5-4B57.3% / 0.16361.6% / 0.30059.7% / 0.25159.6% / 0.238
Gemma-4-E4B59.3% / 0.19361.7% / 0.29762.8% / 0.34961.3% / 0.280
Qwen3.8-27B61.9% / 0.23767.0% / 0.42966.8% / 0.45565.2% / 0.373
gpt-6.1-terra69.2% / 0.38377.1% / 0.63671.1% / 0.52872.5% / 0.516
DeepSeek-V4.1-Flash70.9% / 0.42469.2% / 0.45970.0% / 0.51670.0% / 0.466

Table Rmkks-1-3. Demographic ordering on all 60 selected OpinionQA W54 questions.

TeacherAgeEducationIncomeThree-attribute mean
Qwen2.5-3B-Instruct47.8% / -0.07750.9% / 0.02246.8% / -0.10748.5% / -0.054
Qwen3.5-4B54.7% / 0.09749.9% / -0.01056.2% / 0.16753.6% / 0.084
Gemma-4-E4B49.4% / -0.03753.3% / 0.09053.8% / 0.10852.2% / 0.054
Qwen3.8-27B65.3% / 0.33760.1% / 0.23866.7% / 0.40864.0% / 0.328
gpt-6.1-terra72.8% / 0.50365.7% / 0.36870.8% / 0.49769.8% / 0.456
DeepSeek-V4.1-Flash75.8% / 0.59065.3% / 0.35368.8% / 0.41770.0% / 0.453

Tables Rmkks-1-1–1-3 show recovery of both human opinions and demographic differences. Under the same logits protocol, Qwen2.5-3B → Qwen3.5-4B → Qwen3.8-27B yields lower subgroup TV and higher mean ordering accuracy: 50.2% → 59.6% → 65.2% on WVS, and 48.5% → 53.6% → 64.0% on OpinionQA; Spearman correlation also improves. Terra JSON reaches 72.5%/69.8% ordering accuracy and DeepSeek JSON 70.0%/70.0%, alongside lower TV. Ordering measures demographic differences' direction; TV measures agreement with human response proportions.

Prior research demonstrates that LLMs can simulate human opinions and behavior. Argyle et al. (Political Analysis, 2023) show that GPT-3 conditioned on demographic and political profiles matches individual presidential votes in 85–89% of cases across three ANES election surveys. Park et al. (2026, v3) show that agents built from interviews and surveys for 1,052 people predict held-out General Social Survey answers with accuracy equal to 86% of participants' own test–retest consistency. EconAgent (ACL 2024) exhibits heterogeneous economic decisions: consumption propensity increases across age groups, broadly matching prior empirical patterns. Its economy also reproduces the negative Phillips-curve and Okun's-law relationships (r=−0.619 and −0.918). Survey-specific fine-tuning can improve agreement further:** Cao et al. (NAACL 2025) fine-tune Llama3-8B-Instruct on survey distributions, improving mean WVS 1−JSD from 0.613 to 0.823 and EMD from 0.136 to 0.067 on splits with unseen countries and/or questions. Our teachers receive no such fine-tuning.

This motivates our Introduction's explicit focus on “population-response reconstruction”: as LLM opinion simulation improves, how can a small query budget recover the complete persona–policy response table for a chosen teacher and prompting protocol?

AuthorsOpenReview ↗

Response to Weakness 2 (1/2): Structured baselines and iterative reconstruction

“Comparisons with structured matrix completion or multi-task kernel methods would much help establish how much SCOPE adds beyond standard approaches that already uses person and policy structure.”

We have added complete IMC and MTK comparisons across all four teachers and all three levels of the original paper’s benchmark. Both methods use the same public features and query budgets, with fitting and model selection restricted to paid teacher answers.

  • IMC fits persona main effects, policy main effects, and a rank-four feature-conditioned interaction in centered-log-ratio (CLR) response coordinates: z^ij,c=μc+xiTac+ejTbc+(xiTUc)(VcTej)\hat z_{ij,c}=\mu_c+x_i^Ta_c+e_j^Tb_c+(x_i^TU_c)(V_c^Te_j). A fixed-seed 75/25 split of paid answers selects L2 regularization from 0.01/0.1/1 using probability-space Brier loss; the model is then refit on all paid answers.
  • MTK fits CLR responses with a persona RBF × policy RBF product kernel and an intercept. Leave-one-out Brier loss on paid answers selects a single kernel from 27 bandwidth/regularization combinations. Both baselines query uniformly without replacement.

Complete comparison and an explicit strongest-baseline denominator. We use the original benchmark and budget settings. We first average over all tasks within each seed, then report the mean and sample standard deviation of full-pool TV across three seeds.

Table Rmkks-2-1. Full-pool TV across all 28 benchmark conditions. Results for Random–MLP, Random–XGBoost, BAIT and TypiClust appear in the paper's §4.2, Tables 1–2 (p. 8) and Table 3 (p. 9).

TeacherLevelBIMCMTKSCOPEGain over the strongest baseline (%)
Qwen2.5-3Bfamily160.3107±0.01630.2397±0.00800.2375±0.00440.9
Qwen2.5-3Bfamily320.2696±0.01600.1903±0.00860.1791±0.01155.9
Qwen2.5-3Bfamily640.2321±0.01200.1451±0.00680.1289±0.003111.2
Qwen2.5-3Bdomain1600.2085±0.00500.1409±0.00210.1249±0.001511.4
Qwen2.5-3Bdomain3200.1603±0.00360.1150±0.00110.1022±0.000511.2
Qwen2.5-3Bdomain6400.1474±0.00380.0968±0.00080.0856±0.000211.6
Qwen2.5-3Bglobal6400.1636±0.00190.1358±0.00290.1244±0.00208.4
Qwen3.5-4Bfamily160.1180±0.00350.1113±0.00350.1109±0.00300.4
Qwen3.5-4Bfamily320.0975±0.00540.0868±0.00580.0809±0.00126.8
Qwen3.5-4Bfamily640.0797±0.00510.0656±0.00110.0592±0.00069.8
Qwen3.5-4Bdomain1600.0854±0.00120.0727±0.00170.0632±0.000613.0
Qwen3.5-4Bdomain3200.0779±0.00100.0579±0.00120.0517±0.000310.8
Qwen3.5-4Bdomain6400.0747±0.00120.0481±0.00040.0436±0.00019.4
Qwen3.5-4Bglobal6400.0879±0.00110.0764±0.00070.0701±0.00088.3
Gemma-4-E4Bfamily160.3348±0.00420.2839±0.01560.2791±0.01330.04
Gemma-4-E4Bfamily320.3205±0.01090.2131±0.02390.1995±0.00166.4
Gemma-4-E4Bfamily640.2984±0.01180.1542±0.00870.1363±0.001111.6
Gemma-4-E4Bdomain1600.2866±0.00210.1668±0.00220.1520±0.00168.9
Gemma-4-E4Bdomain3200.2225±0.00380.1348±0.00290.1207±0.001810.4
Gemma-4-E4Bdomain6400.1786±0.00210.1090±0.00140.0993±0.00068.8
Gemma-4-E4Bglobal6400.2079±0.00100.1678±0.00150.1582±0.00485.7
Qwen3.8-27Bfamily160.2590±0.00810.2212±0.00780.2205±0.00970.3
Qwen3.8-27Bfamily320.2365±0.00210.1823±0.00800.1778±0.00312.2
Qwen3.8-27Bfamily640.2097±0.01350.1423±0.00340.1275±0.00048.2
Qwen3.8-27Bdomain1600.2082±0.01500.1491±0.00360.1342±0.00099.5
Qwen3.8-27Bdomain3200.1599±0.00370.1259±0.00200.1130±0.00048.1
Qwen3.8-27Bdomain6400.1466±0.00410.1059±0.00170.0959±0.00034.3
Qwen3.8-27Bglobal6400.1660±0.00470.1414±0.00200.1346±0.00294.8

As shown in Table Rmkks-2-1, SCOPE has the lowest mean TV in all 28 conditions. Its global improvements over the strongest of the six baselines are 8.43%, 8.25%, 5.75%, and 4.83% for Qwen2.5, Qwen3.5, Gemma, and Qwen3.8, respectively.

Iterative coupling of the two bottlenecks. Paid answers calibrate a mixture of persona, policy and interaction structures. Its probability-space covariance proxy combines within-structure uncertainty and between-structure disagreement. Queries are valued by their exact correction plus expected gains elsewhere in the pool; new answers recalibrate sharing: calibration → full-pool query value → new answers → recalibration. This matches our findings of policy-dependent sharing and cross-pair gains.

Why smaller gains at B=16? At family level, all 16 queries go to the shared initialization, leaving no budget for further active querying and recalibration. Gains over the strongest baseline in Table Rmkks-2-1 are 0.04%–0.94%, rising to 2.15%–6.82% at B=32 and 8.23%–11.61% at B=64 across the same four teachers. Together with the factorial ablation (Table 6, Appendix F.1, p. 21), this trend supports repeated coupling of sharing and query allocation.

Reviewer_6dkKOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper explores a practical problem when simulating policy responses with LLMs: with n personas, m policies, and a budget B≪nmB\ll nm, how to efficiently query a teacher (target) model, so that a student model can reconstruct the response distribution at large. With formal definition of this proposed task, this paper first introduces a benchmark. It consists of 1024 personas constructed from WVS responses and 600 policy interventions, forming a three-level hierarchy. It then presents the two findings from the offline diagnostics, which motivates the design of the SCOPE method. SCOPE alternatively predict the responses and allocate queries to calibrate the sharing structure on the fly. Lastly, this paper presents the experiment results where SCOPE shows consistent performance improvement over baselines across policy domains/families on four tested teacher models.

Critical strengths
  1. This paper clearly stated its research question and formally defined the proposed task. The writing is easy to follow and the illustrations are also informative.
  2. This paper also introduces a benchmark with a systematic design: it contains over one thousand personas constructed from WVS and 600 policies across multiple domains/families. The authors have pulled responses from four LLMs on the full persona*policy pairs, forming a large response distribution. This benchmark might also benefit other studies such as understanding how LLM response varies with the persona. The authors could consider elaborating some of the poential applications of this benchmark in the paper somewhere.
  3. The proposed SCOPE method is well motivated by the offline diagnostics with two findings. The alternate process in the algorithm is also effective.
  4. Experiment results show that SCOPE consistently improves over baselines across policy domains/families on all four tested teacher models.
  5. The authors have also conducted analyses on seperating the effects of the design choices in SCOPE (Sec. 4.3).
Critical weaknesses

Overall, this paper is well formulated and well-written. The method is carefully designed, and the experiments are also conducted rigorously.

My biggest concern is on the research problem or the proposed task itself. Why do we want to care about reconstructing LLM responses on persona * policy pairs under begets? For example, researchers could completely avoid this challenge by directly asking the LLM to predict the overall response distribution, instead of querying it n times and then aggregate it up into a distribution. Also, in real-world applications, do we really have such a large and meaningful persona and policy pool? I think rather than framing this method as merely a practical method for this constructed task, the authors might also want to consider what scientific understanding it could bring to us. For instance, can this SCOPE method be used to find the minimal number of personas that best summarize the population? Moreover, this paper excludes the simulation fidelity in its scope, while fidelity is a critical concern in this area. I wonder how the emphasis on simulation fidelity, that is how the reconstructed responses really mimic the human responses, will change the picture. This paper could benefit from providing some real-world application examples, to let the readers better see how the task is practically meaningful.

Another concern is on the generalizability over the persona formats. Though the benchmark contains 1024 personas, forming a diverse pool. But all of them are constructed with WVS using the same set of data fields. This leaves a question on how the method would generalize to other persona constructs.

The section of offline diagnostics (Sec. 2.2-2.3) is what I found very informative and could potentially be discussed more. For example in Sec. 2.2, how's the heterogeneity in the results from different teacher models, and what are the distributions of aia_i and bjb_j. Some descriptions are also not clear, like "top eight directions" (what does directions mean?), "persona permutation" (how personas are permutated?). I personally think this is the very scientific part that explores why the proposed task is challenging, and what are the intrinsic properties of this task. So a more detailed description on the diagnostics process and elaborated discussion on the results would be very helpful.

Lastly, some abbreviations are used in the introduction without expiation, such as APS, MLP, TV. Though what they stand for could be inferred form the context, it would be better explained out to avoid ambiguity.

Questions for authors

Please refer to the Critical Weaknesses section.

8 comments
AuthorsOpenReview ↗

Response to Weaknesses 1.1–1.2: Individual responses and real-world scale

Thank you for recognizing the scientific value of our diagnostics and for your constructive review. We respond to your concerns in turn below.

“Why do we want to care about reconstructing LLM responses on persona * policy pairs under begets?”

“For example, researchers could completely avoid this challenge by directly asking the LLM to predict the overall response distribution, instead of querying it n times and then aggregate it up into a distribution.”

Individual responses preserve population heterogeneity. They support subgroup intersections chosen after querying, alternative population weights and within-group policy comparisons. Agent-based state updates also need agent-conditional responses; here we evaluate a fixed-state response table. For example, two low-income people supporting a policy and two high-income people opposing it yield 50% overall support, but 100% versus 0% within groups. Swapping their answers leaves the overall average unchanged.

Four-teacher test. We reuse existing predictions: ten independently reconstructed domains per teacher, each with 1,024 personas × 60 policies and budget 640 (614,400 pairs; 6,400 queries per teacher–seed). From public profiles, all pairwise intersections of country, recorded sex, age band and income band yield 40 overlapping groups with ≥50 members.

Two retention rules: the population-only reference assigns the teacher's exact overall distribution to every group; SCOPE averages individual predictions within each group. In the example, the reference assigns 50% support to both income groups. Define cgj=pˉgj−pˉjc_{gj}=\bar p_{gj}-\bar p_j, where the bars denote group and population averages, computed separately for teacher and SCOPE. The reference predicts zero contrast. Group TV compares predicted and teacher distributions for the same group; contrast RMSE compares their group–population differences across 40 groups, 600 policies and five answer coordinates.

Table R6dkK-1-1. Subgroup recovery. Each cell is population-only / SCOPE. SCOPE reports mean±sample SD over seeds 719/720/721.

TeacherGroup TV ↓Contrast RMSE ↓
Qwen2.5-3B0.1062 / 0.0372±0.00010.0698 / 0.01584±0.00007
Qwen3.5-4B0.0433 / 0.0194±0.00030.0264 / 0.00750±0.00005
Gemma-4-E4B0.0587 / 0.0537±0.00040.0427 / 0.0244±0.0001
Qwen3.8-27B0.0733 / 0.0496±0.00060.0459 / 0.0190±0.00025

In table order, contrast RMSE falls 77.3%, 71.6%, 42.8%, 58.7%; contrast correlations are 0.9739±0.0003, 0.9591±0.0007, 0.8222±0.0021, 0.9109±0.0024. All four teachers support retaining individual responses for subgroup analysis.

Overall averages hide group differences; individual responses allow new subgroup analyses without more queries.

“Also, in real-world applications, do we really have such a large and meaningful persona and policy pool?”

Population–policy and survey-question pools of this scale are common in social science research, as illustrated below.

Table R6dkK-1-2. Real-world population–policy and survey-question pools.

Study/surveyRespondentsPolicies/questions
WVS797,220; 66 societies290 survey items.
New York Fed SCE~1,300 household heads/monthPublic Policy module: 20 policy instruments; five-point ratings of increases/decreases.
Bulut & Samuel, Communications Earth & Environment 20254,664; four countries180 carbon-tax/information configurations; 13,992 collected ratings.
Bechtel & Scheve, PNAS 20138,500; four countries2,880 candidate climate agreements; 68,000 collected ratings.
OpinionQA, ICML 20236,878 in the inequality survey116 questions in that survey; 1,498 across 15 surveys.
Michigan Surveys of Consumers~1,000/month~50 core questions on finances, economic conditions and expectations.

SCE repeatedly assesses policy changes in the same panel and reports income, education, age and gender breakdowns. The computational pool crosses people with candidate alternatives: all carbon-tax configurations for all respondents would require 4,664×180=839,5204,664\times180=839,520 responses, exceeding our 614,400 pairs. The study collected 13,992 ratings by sampled assignment. SCOPE supports this full-pool simulation workflow; our 6/60/600-policy settings vary its scale. Matched-human validation follows in our response to the fidelity concern.

AuthorsOpenReview ↗

Response to Weakness 1.3: Representative populations and scientific analyses

“I think rather than framing this method as merely a practical method for this constructed task, the authors might also want to consider what scientific understanding it could bring to us.”

“For instance, can this SCOPE method be used to find the minimal number of personas that best summarize the population?”

The reconstructed response table also supports two concrete scientific analyses: how compactly a population's simulated behavior can be summarized, and which policy contexts reveal its largest demographic differences. Both analyses below reuse the ten completed Qwen3.5 domain reconstructions.

1. A response-based population summary. We cluster each persona's predicted responses across a domain's 60 policies, choose an actual persona nearest each cluster center, and weight representatives by the numbers of people they represent. To estimate a subgroup, the weights count that subgroup's members in each cluster. Representatives and assignments are fixed from the student before consulting teacher answers for evaluation. We test 8, 16, 32, and 64 representatives. Compression error is mean TV between compressed and full-student distributions, averaged over policies and 12 predefined country, sex, age, and income groups; every group contains at least 50 people. Table R6dkK-1-3 reports all tested representative-set sizes and controls.

Table R6dkK-1-3. Representative population summaries and controls.

Representatives and assignmentNumber of representativesGroup TV to full studentGroup TV to teacher
Student-response clustering80.0107 ± 0.000250.0205 ± 0.0003
Student-response clustering160.0078 ± 0.00020.0189 ± 0.0004
Student-response clustering320.0064 ± 0.000150.0184 ± 0.0005
Student-response clustering640.0051 ± 0.00010.0179 ± 0.0004
Public-feature clustering640.0160 ± 0.00100.0245 ± 0.0011
Random representatives, public-feature assignment640.0186 ± 0.00520.0262 ± 0.0062
Full SCOPE population1,02400.0175 ± 0.0004

As shown in Table R6dkK-1-3, for an illustrative mean subgroup-compression tolerance of TV ≤ 0.01, 16 is the smallest successful size on the tested grid, a 64-fold reduction in the number of stored representative response vectors. Its teacher group TV is 0.0189, close to the full student's 0.01748. Both public-feature controls remain above this tolerance with 64 representatives. This yields a behavioral size–fidelity curve after reconstruction and allows us to check whether a compact population summary is adequate for the intended analysis. Each domain selects its own response representatives.

2. Identifying policies that reveal demographic differences. Using the 40 intersectional groups above, we rank the 600 policies by the root-mean-square magnitude of their teacher group contrasts, hj=[(5∣G∣)−1∑g,acgja2]1/2h_j=[(5|\mathcal G|)^{-1}\sum_{g,a}c_{gja}^{2}]^{1/2}, and compare that ranking with the student ranking. SCOPE attains Spearman correlation 0.9114 ± 0.0079 and recovers 45–51 of the teacher's 60 highest-heterogeneity policies across the three seeds. The reconstructed table therefore supports prioritizing policy contexts for detailed distributional analysis.

AuthorsOpenReview ↗

Response to Weakness 1.4 (1/2): Human validation and supporting evidence

“I wonder how the emphasis on simulation fidelity, that is how the reconstructed responses really mimic the human responses, will change the picture.”

Thank you for raising this concern. We validate teachers against matched human survey responses.

1. Direct validation against human responses.

Data and selection. From English-interviewed US respondents in WVS7 and Pew ATP W54, released through OpinionQA, we sample 1,024 personas per survey, without replacement and proportionally within sex × age strata. Before teacher collection, we select complete question batteries with verified ordered scales, retaining original wording and four-, five-, or ten-category options. All 59 selected WVS and 60 selected W54 questions are evaluated, forming 60,416 and 61,440 persona–question pairs.

  • WVS7: institutional confidence, Q64–Q89 excluding Q71; justifiability, Q177–Q195; democracy, Q241–Q249; social/work attitudes, Q36–Q41. Q71 is excluded because it is already a persona input.
  • OpinionQA W54: GOVPRIORITYa–f; ECON3_a–i; ECON4_a–i plus INEQ6/INEQ7; INEQ5_a–m; INEQ8_a–j; ECON5_a–k. All identifiers carry the suffix _W54.

We also evaluate stronger, higher-query-cost API teachers: gpt-6.1-terra and DeepSeek-V4.1-Flash generate JSON probability vectors, normalized by their positive sum.

Metrics. Lower TV and higher accuracy indicate better agreement. Accuracy averages whether the highest-probability option matches each valid respondent–question answer; ties use the first original option. The uniform reference uses equal probabilities and expected accuracy 1/K for K legal options.

Table R6dkK-1-4. Teacher–human agreement on WVS and OpinionQA.

TeacherWVS: TV ↓WVS: accuracy ↑OpinionQA: TV ↓OpinionQA: accuracy ↑
Qwen2.5-3B-Instruct0.433237.04%0.421130.48%
Qwen3.5-4B0.290236.59%0.280038.08%
Gemma-4-E4B0.411939.33%0.362640.63%
Qwen3.8-27B0.259742.00%0.196540.01%
gpt-6.1-terra0.183044.17%0.151341.99%
DeepSeek-V4.1-Flash0.186143.97%0.140943.13%
Uniform over legal options0.368817.36%0.261424.22%

Both API teachers and Qwen3.8 outperform the uniform reference in TV on both surveys; all six teachers exceed it in accuracy. Terra and DeepSeek reduce TV by 42.12%–50.39% relative to uniform. Qwen3.8 has the lowest TV among the local teachers. From Qwen2.5-3B to Qwen3.8-27B, WVS/OpinionQA accuracy rises from 37.04%/30.48% to 42.00%/40.01%; Terra reaches 44.17% on WVS and DeepSeek 43.13% on OpinionQA. These results show improved opinion simulation with the stronger teachers evaluated.

Prior research demonstrates that LLMs can simulate human opinions and behavior. Argyle et al. (2023) show that GPT-3 conditioned on demographic and political profiles matches individual presidential votes in 85–89% of cases across three ANES election surveys. Park et al. (2026) show that agents built from interviews and surveys for 1,052 people predict held-out General Social Survey answers with accuracy equal to 86% of participants' own test–retest consistency. EconAgent (ACL 2024) demonstrates heterogeneous economic decisions: consumption propensity increases across age groups, broadly matching prior empirical patterns. Survey-specific fine-tuning can further improve agreement: Cao et al. (NAACL 2025) fine-tune Llama3-8B-Instruct on survey distributions, increasing mean WVS 1−JSD from 0.613 to 0.823 and reducing EMD from 0.136 to 0.067 on splits with unseen countries and/or questions. Our teachers are evaluated without this adaptation.

This motivates our Introduction's focus on “population-response reconstruction”: once a teacher and prompting protocol are chosen, how can a small query budget recover the complete persona–policy response table?

Reviewer_JXosOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This is an interesting paper. The authors tackle a new task called 'population response reconstruction', aiming to predict how a population of synthetic individuals might respond to policy interventions. The underlying observation is that simulated responses are not independent: individuals with similar profiles, similar policy interventions etc. offer conditionality that can be exploited to reduce LLM calls within a budget.

The benchmark comprises >1000 real world personas, 600 policy interventions organised hierarchically, leading to >600k pesona policy combinations. Motivated by the finding of exploitable correlations the authors propoes a framework (SCOPE) for combining learning across personas/policies, and active selection of new persona/policy queries. SCOPE performs competitively with baselines, with largish improvements over the nearest competitor.

Critical strengths
  • Interesting and well motivated problem formulation that identifies a practical bottleneck in population simulation and tries to solve it. Nice mathematical treatment.
  • Convincing empirical results for the proposed method, e.g. use of ablations and reported diagnostics, hierarchical benchmark.
  • Clear presentation and reproducibility.
Critical weaknesses
  • Population average error might conceal some interesting heterogeneity - not a blocker but more a call to dig deeper into potential subgroup demographics, and consider worst-case errors.
  • Somewhat limited family of models used in constructing the benchmark. Not sure how persona-EQ-friendly Qwen's family is but it might be worth a motivating comment. Only Gemma is outside the Qwen family - any thoughts?
Questions for authors
  • Could you comment on the subgroup point raised above and perhaps say something about what you have found?
  • Also on the fit for task of Qwen?
2 comments
AuthorsOpenReview ↗

Weakness 1 / Question 1: Heterogeneity and tail errors

Thank you for your thoughtful and constructive review of our work. We address your comments and questions below.

Weakness 1 / Question 1: “Population average error might conceal some interesting heterogeneity - not a blocker but more a call to dig deeper into potential subgroup demographics, and consider worst-case errors.”

Thank you for this suggestion. We added Appendix F.6, “Demographic and policy-domain heterogeneity,” with Figures 5–6 (pp. 24–25 of the revised PDF), highlighted in light purple. Using the original four teachers’ global experiments (1,024 personas × 600 policies = 614,400 pairs; B=640; seeds 719/720/721), we analyze 55 single-attribute groups with at least 50 personas across 15 attributes, and ten policy domains. The heatmaps report SCOPE’s absolute TV and relative reduction against a fixed strongest original baseline.

First, socioeconomic disadvantage is associated with consistently harder reconstruction across teachers (Figure 5a, p. 24). For all four teachers, rural, low-income, primary-educated, and unemployed groups have higher TV than urban, high-income, bachelor-educated, and full-time-employed groups, respectively. For example, TV is 19.1%–65.8% higher for primary-educated than bachelor-educated personas, and 32.8%–65.5% higher for unemployed than full-time-employed personas, depending on the teacher.

A possible explanation is uneven demographic coverage in teacher training and preference tuning, which could affect the regularity of persona–policy responses and how readily they can be learned from sparse queries. Santurkar et al. (ICML 2023) found that some instruction-tuned models' opinions align more closely with higher-income and better-educated respondents.

Policy difficulty also varies (Figure 6a, p. 25): social protection is hardest for all three Qwen teachers, while defense is hardest for Gemma; public order and safety is easiest for Qwen3.5, Gemma, and Qwen3.8, and recreation, culture, and religion for Qwen2.5. Teacher-specific group patterns remain (Figure 5a, p. 24): high government confidence is associated with Gemma's easiest group (0.1037) and Qwen2.5's hardest (0.1835).

Second, SCOPE’s gains extend across all evaluated marginal slices (Figures 5b and 6b, pp. 24–25). All 220 group–teacher and 40 domain–teacher three-seed mean TVs improve, with relative reductions of 3.15%–24.68% and 3.12%–25.95%, respectively. Group results average errors over that group’s personas across all policies; domain results average errors over all personas within that domain. All 16 comparisons for the four socioeconomic groups highlighted above improve over the baseline, with TV reductions of 7.47%–21.83%. The relative benefit varies by teacher: rural versus urban gains are 21.83% versus 16.12% for Qwen3.5, and 8.74% versus 14.31% for Gemma. Thus, SCOPE improves reconstruction for the harder groups as well as the easier ones, while the heatmaps identify the remaining differences in difficulty.

Third, SCOPE also improves reconstruction of the highest-error cases (Table RJXos-1-1). We measure CVaR95: the average TV of each method’s worst-predicted 5% of pairs (30,720 pairs). These pairs are selected separately for each method and seed. We retain the same baseline selected by global mean TV as in Figures 5–6 (pp. 24–25). Entries in Table RJXos-1-1 report the mean ± sample SD over seeds 719/720/721.

Table RJXos-1-1. Average error on each method’s worst-predicted 5% of persona–policy pairs (lower is better).

TeacherBaselineBaseline: mean TV of its worst 5%SCOPE: mean TV of its worst 5%Relative reduction
Qwen3.5-4BRandom–MLP0.3029 ± 0.00390.2364 ± 0.002221.96%
Qwen2.5-3BRandom–MLP0.5510 ± 0.00530.4938 ± 0.016010.38%
Gemma-4-E4BRandom–XGBoost0.7851 ± 0.01930.7064 ± 0.011110.02%
Qwen3.8-27BRandom–XGBoost0.5490 ± 0.00910.4994 ± 0.01139.03%

For example, on Qwen3.5, the baseline’s worst-predicted 5% have average TV 0.3029, whereas SCOPE’s worst-predicted 5% have average TV 0.2364, a 21.96% reduction. Across all four teachers, SCOPE reduces this high-error-tail average by 9.03%–21.96%, with improvement in every seed. Together, Figures 5–6 (pp. 24–25) and Table RJXos-1-1 show that the gains extend to demographic groups, policy domains, and the highest-error 5% of cases.

AuthorsOpenReview ↗

Weakness 2 / Question 2: Cross-family results, API teacher and Qwen rationale

Weakness 2 / Question 2: “Somewhat limited family of models used in constructing the benchmark.” “Also on the fit for task of Qwen?”

Thank you for suggesting broader coverage and a clearer Qwen rationale. We added GLM-4-9B-Chat, Ministral-3-8B-Instruct-2512 and API teacher gpt-6.1-terra. Evaluation now covers six open-weight checkpoints from four families—Qwen, Gemma, GLM and Mistral—plus Terra.

For GLM/Ministral, we collected all 1,024 personas × 600 policies (614,400 pairs) with the original Chinese prompts and five-way probability scoring. Comparisons cover all 100 independent families, ten independent domains and the global task. Budgets are 16/32/64 per family, 160/320/640 per domain and 640 globally. Each teacher uses its own response reference and policy representations; SCOPE uses each task's budgeted answers for calibration.

All results report mean±sample SD over seeds 719/720/721. The strongest baseline is selected per setting by mean TV from Random–MLP, Random–XGBoost, BAIT–MLP and TypiClust–MLP. Table RJXos-2-1 aggregates task-level full-pool TV within each seed. SCOPE is best in all 14 settings.

Table RJXos-2-1. Full-pool TV on GLM and Ministral.

Additional teacherGranularityBudget per taskSCOPE TV ± SDStrongest baselineBaseline TV ± SDRelative reduction
GLM-4-9B-ChatFamily160.1431 ± 0.0155Random–MLP0.1490 ± 0.01054.0%
GLM-4-9B-ChatFamily320.1073 ± 0.0033TypiClust–MLP0.1198 ± 0.002210.5%
GLM-4-9B-ChatFamily640.0777 ± 0.0012Random–MLP0.0898 ± 0.004613.5%
GLM-4-9B-ChatDomain1600.0784 ± 0.0017Random–MLP0.0844 ± 0.00117.0%
GLM-4-9B-ChatDomain3200.0644 ± 0.0003Random–MLP0.0668 ± 0.00063.6%
GLM-4-9B-ChatDomain6400.0528 ± 0.0002Random–MLP0.0534 ± 0.00031.1%
GLM-4-9B-ChatGlobal6400.0780 ± 0.0007Random–MLP0.0861 ± 0.00189.4%
Ministral-3-8BFamily160.1728 ± 0.0199Random–MLP0.1757 ± 0.01131.6%
Ministral-3-8BFamily320.1157 ± 0.0032TypiClust–MLP0.1351 ± 0.001714.3%
Ministral-3-8BFamily640.0792 ± 0.0018Random–MLP0.1043 ± 0.006824.0%
Ministral-3-8BDomain1600.0821 ± 0.0024Random–MLP0.0977 ± 0.001715.9%
Ministral-3-8BDomain3200.0646 ± 0.0006Random–MLP0.0738 ± 0.001212.4%
Ministral-3-8BDomain6400.0515 ± 0.0003Random–MLP0.0592 ± 0.000512.9%
Ministral-3-8BGlobal6400.0842 ± 0.0007Random–MLP0.0996 ± 0.002815.5%

With 640 queries from 614,400 pairs, global TV falls 9.4% for GLM and 15.5% for Ministral.

Terra uses a balanced 61,440-pair subset (10%) spanning all 1,024 personas and 600 policies. We retain the original prompt structure, personas, policies and five response categories. SCOPE parses Terra's five JSON probabilities as targets for queried pairs. Public persona features and independently encoded BGE-small-zh-v1.5 policy features provide the inputs. Budgeted API answers calibrate response sharing and guide subsequent queries, without access to teacher weights, logits or hidden states. All four budgets use the same collected reference. Table RJXos-2-2 evaluates unqueried-pair TV; SCOPE is best at every budget.

Table RJXos-2-2. Unqueried-pair TV for Terra.

API teacherBudgetSCOPE TV ± SDStrongest baselineBaseline TV ± SDRelative reduction
GPT-6.1-terra160.1393 ± 0.0070Random–XGBoost0.1514 ± 0.01047.96%
GPT-6.1-terra320.1394 ± 0.0049BAIT–MLP0.1468 ± 0.00265.03%
GPT-6.1-terra640.1296 ± 0.0023Random–XGBoost0.1406 ± 0.00567.81%
GPT-6.1-terra1280.1264 ± 0.0028Random–XGBoost0.1334 ± 0.00285.25%

Family and API tasks start with 16 queries and add 16 per round. B=16 measures the initial fit; higher budgets include repeated selection and refitting. Terra's TV reductions of 5.03%–7.96% extend the gains to generated probability responses. These results demonstrate practical applicability to paid API teachers: SCOPE can use a small query budget without requiring access to the teacher's internals.

Why Qwen? Qwen was chosen for broad adoption and competitive human-response simulation. Hugging Face's 2026 report documents its downstream ecosystem. SimVBG, SocioVerse and SimBench show competitive simulations of survey responses, population opinions and behavioral choices. SocioVerse's Qwen2.5-72B election simulation correctly predicts 92.2% of state outcomes. These applications motivate persona-conditioned simulation; open checkpoints support reproducible probability extraction.

If you have any further questions, please let us know.

Reviewer_QHdnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to reconstruct an LLM teacher's responses for every persona–policy pair from a small budget of teacher queries, without access to pre-collected persona–policy response labels. The benchmark crosses 1,024 WVS personas with 600 COFOG-based policies (614,400 pairs per teacher), organized into family, domain, and global reconstruction levels; the target is each pair's five-way response distribution, and error is mean TV over the full pool.

Offline diagnostics motivate the method: persona and policy main effects explain about 86–95% of response variation, interactions are concentrated but policy-conditioned, and 75–81% of the benefit of an informative query falls on pairs that were not queried. The proposed method, SCOPE, alternates between calibrating a mixture of persona, policy, and interaction kernels from paid answers and selecting new queries by their predicted error reduction across the whole pool.

Experiments cover 4 teachers, against Random–MLP, Random–XGBoost, BAIT, and TypiClust. The paper reports that SCOPE achieves the lowest mean error in all 24 family/domain settings and in all 4 global settings.

Critical strengths

None

Critical weaknesses

1. The reported numbers for Table 3 appear to come from a single seed, not the three-seed means described in §4.1.

Every entry in Table 3 matches seed 719 exactly in the released per-task results (results/raw_task_metrics.csv), rather than the three-seed mean. The authors' own three-seed table (results/table3_replacement.csv) gives gains of 7.6–17.8%, not the 11.3–18.3% stated in §4.2.

2. The computational-cost motivation is not quantified.

The task is motivated by the cost of exhaustive teacher evaluation. Several aspects of the setup make it unclear how high that cost is in practice relative to SCOPE's own cost:

  • All four teachers are open-weight models with 3–27B parameters.
  • According to configs/prompt.json, each query is scored from the softmax of five answer-token logits with thinking disabled. A query is thus a single prefill pass with no generation.
  • SCOPE itself refits H = 81 kernel candidates per round (O(H|Q|³), Appendix E.1), computes pool-wide cross-kernels and covariance terms for acquisition, and produces full-pool predictions.

Eq. 29 decomposes SCOPE's runtime, but neither the paper nor the repository reports wall-clock time, hardware, or cost for any component.

3. Prompt language does not match the populations being simulated, and this choice is not discussed.

The entire teacher prompt is in Chinese:

However, the personas are respondents from South Korea, Germany, Canada and Zimbabwe, and the system prompt frames each policy as implemented in the respondent's own country.

Minor issues

  • Eq. 8 contains a LaTeX artifact: "qquad ". This command appears to have been escaped incorrectly.
  • §4.2 states that "the single-seed Qwen3.8 exception in the earlier draft reverses after replication."
Questions for authors
  1. Can you confirm whether Tables 3 and 6 report seed 719 only? If so, could you provide three-seed means with standard deviations?
  2. What are the GPU-hours and hardware used to compute each teacher's full response tensor?
  3. What is SCOPE's end-to-end wall-clock time and memory footprint at the family, domain and global levels, on the same hardware?
  4. At what teacher cost per query does SCOPE become cheaper overall than exhaustive evaluation? Are there realistic settings (larger teachers, generated rather than logit-scored answers, multiple samples per pair) where the savings would be substantial, and are those settings compatible with the method's need for logits and hidden states?
  5. Why were all prompts written in Chinese for non-Chinese personas? Please justify this choice and evaluate the robustness of the results to prompt language, for example by comparing Chinese prompts with English translations (and, where feasible, native-language prompts) on a subset of personas and policies.
7 comments
AuthorsOpenReview ↗

Weakness 1 / Question 1: Corrected three-seed results

Thank you for your constructive review of our work. We address each concern below.

Weakness 1: “The reported numbers for Table 3 appear to come from a single seed, not the three-seed means described in §4.1.”

Question 1: “Can you confirm whether Tables 3 and 6 report seed 719 only? If so, could you provide three-seed means with standard deviations?”

Thank you for carefully checking our released raw experimental results. The experiments for all three seeds (719, 720, and 721) had already been completed and their results were available in our repository. During manuscript preparation, we inadvertently left Tables 3 and 6 with the seed-719 results instead of updating them to the three-seed summaries. This also explains the absence of standard deviations in those two tables. We apologize for this reporting oversight.

We have now corrected both tables to report three-seed means and sample standard deviations, aggregating tasks within each seed before computing the across-seed statistics. We have also updated the corresponding global improvement range in §4.2 to 7.6–17.8%. Figure 4 (p. 9) and the ablation discussion in §4.3 already used the three-seed results; the corrected Table 6 is now consistent with them.

All substantive conclusions in the paper remain unchanged. SCOPE retains the lowest mean TV in all 24 family/domain settings and all four global settings. The factorial ablation continues to support the complementary contributions of response calibration and pool-value acquisition, with their combination achieving the lowest mean TV in all five settings with queries beyond initialization.

Family B=16 is an initialization-only special case. The ablation uses 16 initial queries, so the entire budget is spent on a single random selection at the outset. SCOPE fits and calibrates on those answers and then outputs predictions, with no remaining budget for the closed-loop sequence of calibrating response sharing, actively selecting new queries, and recalibrating from the newly acquired answers. The method therefore reduces to a single fit after random initialization in this setting; for each sharing structure, the random and active columns are identical. Fixed sharing has slightly lower mean TV in this condition. When the budget permits subsequent querying and updating, the joint method has the lowest mean TV in all five remaining ablation settings.

The corrected table entries, added standard-deviation descriptions, and revised improvement range are highlighted in yellow in the revised paper.

The corrected Tables 3 and 6 are provided in Table RQHdn-1-1 and Table RQHdn-1-2, respectively. Relative reductions use unrounded means and the lowest-mean baseline.

Table RQHdn-1-1. Corrected manuscript Table 3.

TeacherRandom–MLPRandom–XGBoostBAITTypiClustSCOPERelative reduction
Qwen3.5.0853 ± .0006.0876 ± .0018.0912 ± .0026.0974 ± .0019.0701 ± .000817.77%
Qwen2.5.1439 ± .0008.1562 ± .0024.1560 ± .0068.1606 ± .0111.1244 ± .002013.55%
Gemma.1857 ± .0026.1815 ± .0032.1988 ± .0090.2059 ± .0058.1582 ± .004812.85%
Qwen3.8.1530 ± .0044.1456 ± .0031.1550 ± .0019.1672 ± .0020.1346 ± .00297.61%

Table RQHdn-1-2. Corrected manuscript Table 6.

LevelBudgetFixed sharing / RandomFixed sharing / ActiveCalibrated sharing / RandomCalibrated sharing / Active
Family16.1088 ± .0040.1088 ± .0040.1109 ± .0030.1109 ± .0030
Family32.0882 ± .0055.0814 ± .0012.0867 ± .0060.0809 ± .0012
Family64.0687 ± .0018.0616 ± .0007.0654 ± .0006.0592 ± .0006
Domain160.0708 ± .0018.0660 ± .0004.0690 ± .0019.0632 ± .0006
Domain320.0566 ± .0010.0543 ± .0004.0550 ± .0011.0517 ± .0003
Domain640.0467 ± .0004.04519 ± .00004.0456 ± .0004.0436 ± .0001
AuthorsOpenReview ↗

Questions 2–3: Hardware, teacher costs, and SCOPE runtime overview

Weakness 2: “The computational-cost motivation is not quantified.”

(a) Teacher evaluation: hardware and GPU-hours — Question 2

What are the GPU-hours and hardware used to compute each teacher's full response tensor?

We report the cost of evaluating the complete 1,024-persona × 600-policy tensor (614,400 pairs) on one NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB VRAM), with an Intel Xeon Platinum 8470Q CPU, a 22-core CPU quota, and a 110 GiB host-memory limit. Timing processes use two CPU threads pinned to two cores; the GPU is exclusively allocated during measurement. Both backends use the original Chinese prompts, BF16 weights, disabled thinking, and the five answer-token probabilities.

vLLM configuration: version 0.30.0, tensor_parallel_size=1, max_model_len=2048, max_num_seqs=256, max_num_batched_tokens=8192, gpu_memory_utilization=0.85, enable_prefix_caching=True, enable_chunked_prefill=True, and enforce_eager=True; requests are submitted in chunks of at most 512. Prefix caches are reset before each timing panel, and persona-prefix priming is included in the timing.

Without vLLM, we use Hugging Face Transformers 5.17.0 and PyTorch 2.13.0, with batch size 32, left padding, use_cache=False, and logits_to_keep=1. This computes the same five-answer scoring task using batched forward passes, without shared-prefix reuse or generated reasoning.

Table RQHdn-2-1. Full-pool answer-evaluation costs.

TeachervLLM time (min)vLLM GPU-hTransformers time (min)Transformers GPU-h
Qwen2.5-3B24.460.40894.861.581
Qwen3.5-4B107.691.795251.994.200
Gemma-4-E4B41.980.700200.583.343
Qwen3.8-27B606.3710.1061021.8817.031

GPU-hours equal elapsed hours on the exclusively allocated GPU.

(b) SCOPE runtime and memory — Question 3

What is SCOPE's end-to-end wall-clock time and memory footprint at the family, domain and global levels, on the same hardware?

We completed all 28 predeclared teacher–granularity–budget conditions on the same exclusively allocated GPU and host as in Question 2. Timing starts at the runner’s first statement and ends after predictions are saved and resources released, covering fresh encoding of all 600 policies, model preparation, live teacher queries, student fitting/calibration, acquisition, full-pool prediction, and output. The complete teacher reference is opened only afterward for evaluation. Kernel linear algebra, acquisition, and prediction use the GPU; public PCA and mixture-weight optimization use the CPU. The runner uses four CPU threads and the vLLM teacher service two.

Time and GPU-hours (Table RQHdn-2-2). SCOPE takes 51.99–1,352.93 seconds (0.87–22.55 minutes) and 0.014–0.376 GPU-hours per complete setting. Relative to the full-teacher cost in Question 2, the speedup is 1.98–143.52×, reducing elapsed time and allocated GPU-hours by 49.47%–99.30%. Speedups are 13.07–111.64× for all 100 families, 1.98–143.52× for all ten domains, and 2.42–41.47× for the global task. GPU-hours include the entire allocation, including preparation, CPU computation, and teacher-idle periods. Why SCOPE saves time (Table RQHdn-2-3). Each setting queries only 0.10%–1.04% of the 614,400 pairs. Policy representations are computed once and shared across personas, while the student transfers information from queried answers to the remaining pairs. The reported times include student learning, repeated acquisition, and full-pool prediction. Table RQHdn-2-3 shows that full-pool prediction dominates global-task time.

Memory use (Table RQHdn-2-2). All 24 family/domain conditions have a lower complete-process GPU-memory peak than the corresponding Q2 teacher-service benchmark, by 20.22%–46.39%. Student/runner peaks are 0.78–3.89 GiB for family tasks, 2.31–6.16 GiB for domain tasks, and 29.84–34.42 GiB for global tasks, where pool-wide arrays are larger. Complete-process host RAM ranges from 6.08 to 52.00 GiB. Relative to the corresponding Q2 teacher-service GPU-memory peak, including its vLLM reservation, the student's independent peak is only 0.93%–7.35% in family/domain tasks and 36.27%–42.00% in global tasks. The reported runs keep the teacher resident during active querying and release it before final student fitting and prediction. A stage-separated schedule can avoid simultaneous teacher and student occupancy by releasing GPU memory between stages, with peak memory governed approximately by the largest stage.

AuthorsOpenReview ↗

Question 3: Complete runtime and memory results

Table RQHdn-2-2. Whole-setting time and resource footprint. Family-level rows aggregate all 100 families and domain-level rows all ten domains; every condition covers 614,400 pairs. The full-teacher reference adds measured initialization and warmup to Q2’s answer cost.

TeacherLevelB/taskPaid totalSCOPE s (live)Full s (Q2)RatioGPU-h (allocated)Q3 RAM peak GiBQ2 service VRAM peak GiBQ3 peak VRAM: teacher + student = total (GiB)Peak stageStudent independent peak GiBQ3 VRAM GiB-h
Qwen2.5-3Bfamily161,60051.991,483.2728.53×0.0146.8781.6943.26 + 0.54 = 43.79Teacher query0.780.18
Qwen2.5-3Bfamily323,20072.431,483.2720.48×0.0207.6481.6943.59 + 3.68 = 47.27Teacher query3.680.44
Qwen2.5-3Bfamily646,400113.471,483.2713.07×0.0327.6581.6943.62 + 3.89 = 47.50Acquisition3.890.97
Qwen2.5-3Bdomain1601,60085.211,483.2717.41×0.0246.0881.6943.49 + 2.31 = 45.80Acquisition2.310.66
Qwen2.5-3Bdomain3203,200203.851,483.277.28×0.0576.9681.6943.58 + 3.24 = 46.82Acquisition3.242.09
Qwen2.5-3Bdomain6406,400749.551,483.271.98×0.2089.1681.6943.65 + 5.27 = 48.92Acquisition5.278.95
Qwen2.5-3Bglobal640640613.331,483.272.42×0.17028.7181.6943.12 + 29.84 = 72.96Acquisition29.847.69
Qwen3.5-4Bfamily161,600216.916,538.5830.14×0.0609.5082.1249.33 + 0.54 = 49.87Teacher query0.780.79
Qwen3.5-4Bfamily323,200127.746,538.5851.19×0.0358.4382.1249.33 + 3.68 = 53.01Teacher query3.680.95
Qwen3.5-4Bfamily646,400190.626,538.5834.30×0.0538.5982.1249.25 + 3.89 = 53.13Acquisition3.891.87
Qwen3.5-4Bdomain1601,600128.936,538.5850.71×0.0367.1482.1248.78 + 2.46 = 51.24Acquisition2.461.01
Qwen3.5-4Bdomain3203,200261.296,538.5825.02×0.0738.4082.1248.71 + 3.14 = 51.85Acquisition3.142.80
Qwen3.5-4Bdomain6406,400764.326,538.588.55×0.2129.6382.1248.78 + 4.98 = 53.76Acquisition4.989.78
Qwen3.5-4Bglobal640640636.736,538.5810.27×0.17730.7082.1248.33 + 31.20 = 79.53Acquisition31.208.53
Gemma-4-E4Bfamily161,600171.192,710.4315.83×0.04817.5981.9643.68 + 0.54 = 44.22Teacher query0.780.72
Gemma-4-E4Bfamily323,20096.432,710.4328.11×0.02716.9981.9644.48 + 3.68 = 48.16Teacher query3.680.62
Gemma-4-E4Bfamily646,400141.882,710.4319.10×0.03915.8981.9644.48 + 3.89 = 48.37Acquisition3.891.23
Gemma-4-E4Bdomain1601,600114.172,710.4323.74×0.03216.0981.9644.54 + 2.46 = 47.00Acquisition2.460.90
Gemma-4-E4Bdomain3203,200239.352,710.4311.32×0.06616.2781.9644.39 + 3.07 = 47.47Acquisition3.072.43
Gemma-4-E4Bdomain6406,400736.912,710.433.68×0.20516.6481.9644.54 + 5.33 = 49.87Acquisition5.338.80
Gemma-4-E4Bglobal640640825.652,710.433.28×0.22934.9381.9643.38 + 34.42 = 77.80Acquisition34.4210.73
Qwen3.8-27Bfamily161,600763.1536,899.3048.35×0.21252.0083.8860.24 + 0.54 = 60.78Teacher query0.7810.63
Qwen3.8-27Bfamily323,200330.5336,899.30111.64×0.09251.1483.8860.24 + 3.68 = 63.92Teacher query3.684.82
Qwen3.8-27Bfamily646,400551.2636,899.3066.94×0.15349.9383.8863.03 + 3.89 = 66.92Acquisition3.898.96
Qwen3.8-27Bdomain1601,600257.1136,899.30143.52×0.07149.6583.8858.63 + 2.41 = 61.04Acquisition2.413.53
Qwen3.8-27Bdomain3203,200499.0236,899.3073.94×0.13949.9583.8859.12 + 3.27 = 62.39Acquisition3.277.45
Qwen3.8-27Bdomain6406,4001,352.9336,899.3027.27×0.37649.7283.8858.62 + 6.16 = 64.79Acquisition6.1621.65
Qwen3.8-27Bglobal640640889.8536,899.3041.47×0.24751.2583.8857.43 + 30.42 = 87.85Acquisition30.4214.45

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

Dear Area Chair,

Thank you for your time and consideration in evaluating our submission.

We introduce a new task, population-response reconstruction: given n personas, m policies, and a budget of B ≪ nm LLM teacher queries, the goal is to reconstruct the response for every persona–policy pair. Building on findings of compact, policy-conditioned response sharing and cross-pair query gains, SCOPE jointly optimizes response sharing and query allocation: paid answers calibrate how responses transfer across personas and policies, and the calibrated sharing model guides subsequent queries by their expected reduction in full-pool error. Our proposed new benchmark covers 1,024 personas and 600 policies across four teachers.

Reviewers recognized clear task formulation and empirical gains (JXos, 6dkK, mkks), a systematic hierarchical benchmark, convincing results, and ablations isolating design contributions (JXos, 6dkK), diagnostics that motivate the method and support logical flow (6dkK, mkks), and clear presentation (JXos, 6dkK). JXos also highlighted the interesting, well-motivated task formulation, mathematical treatment, and reproducibility. 6dkK emphasized the formal task definition, carefully designed method and effective alternating procedure, rigorous experiments, scientific insight from diagnostics, and the benchmark's reuse potential for persona-response research.

Following our rebuttal, 6dkK raised their score 3 → 4, and mkks 1 → 2. mkks confirmed their baseline concern was addressed and human validation partially addressed their first concern. QHdn and JXos have not responded further.

We summarize the added evidence and clarifications below.


Reviewer QHdn (Score: 2, Conf: 2):

  • Concern 1 (Three-Seed Reporting): Updated Tables 3 and 6 to three-seed means and sample standard deviations. Conclusions remain unchanged.
  • Concern 2 (Computational Savings and Compatibility): Measured 1.98–143.52× speedups and 63.07–99.49% lower peak VRAM × runtime (GiB·h) than exhaustive evaluation. DeepSeek-V4.1-Flash and gpt-6.1-terra validate larger models, reasoning-enabled inference, and JSON responses with independent embeddings, without teacher logits or hidden states.
  • Concern 3 (Prompt Language): Repeated the full English benchmark and German/Korean interview-language tests. SCOPE has the lowest mean TV in all 84 additional settings against the original baselines.

Reviewer JXos (Score: 4, Conf: 3):

  • Concern 1 (Heterogeneity and Tail Errors): Analyzed demographic groups and policy domains. SCOPE improves all 220 group/teacher and 40 domain/teacher comparisons and mean error on each method's worst-predicted 5% of pairs, against each teacher's strongest original baseline.
  • Concern 2 (Teacher Diversity): Added open-weight GLM and Ministral and closed-source gpt-6.1-terra. SCOPE achieves lower mean TV than all evaluated baselines in every added setting.

Reviewer 6dkK (Score: 3 → 4, Conf: 3):

  • Concern 1 (Purpose and Practical Relevance): Clarified that individual responses support subgroup analysis, reweighting, and policy comparisons. Added subgroup-recovery tests and real survey examples. Matched-human experiments show that reconstruction gains transfer to improved human-response prediction.
  • Concern 2 (Scientific Utility and Persona Generalization): Showed that reconstructed responses support compact representative populations and help identify policies revealing demographic differences. WVS and OpinionQA experiments support generalization across data sources and persona schemas.
  • Concern 3 (Diagnostic Clarity): Expanded analyses of teacher/subgroup differences and persona/policy effects, clarified singular directions and persona permutation, and explained their connection to the method. Defined abbreviations at first use.

Reviewer mkks (Score: 1 → 2, Conf: 3):

  • Concern 1 (Human Validity): Added WVS and OpinionQA validation. All six LLMs exceed uniform-random accuracy, with stronger tested models better recovering human responses and demographic differences, consistent with prior research. Baseline comparisons on these surveys show that SCOPE's reconstruction gains transfer to human simulation across all 18 settings. To further address the population mismatch, we repeated validation and reconstruction with the original benchmark personas, the rebuttal's WVS questions, and the same protocol using DeepSeek-V4.1-Flash, obtaining similar findings.
  • Concern 2 (Structured Baselines): Added IMC and MTK with matched features and query budgets, restricting fitting and model selection to paid answers. SCOPE retains the lowest mean TV in all 28 settings against all six baselines.

Thank you for considering the additional results and clarifications.

Sincerely,
The Authors of Submission 57682

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.