- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper is related to active learning, and studies how a model can reconstruct an LLM’s responses across a large set of persona-policy pairs (i.e., LLM persona + candidate policy combinations, in the paper the example was 1,024 times 600) using a limited query budget. Its method, SCOPE, learns how responses transfer across personas and policies, then selects persona + policy queries expected to improve predictions across the full pool. Experiments use profiles drawn from the World Values Survey and LLM generated policy responses. SCOPE consistently outperforms the baselines.
Critical strengths
- The problem is clearly defined which helps understanding.
- Good flow: diagnostics motivate the design. Section 2 shows that main effects explain about 86 -- 95% of response variance and that ~ 80% of the gain comes from oracle selection lands on cells other than the queried. These findings are directly relating to the two components of the method.
Critical weaknesses
-
The paper motivates through the practical problem of estimating population responses to a candidate policy, but evaluates only agreement with an LLM teacher. This, in my opinion, is a huge gap between what’s stated in Introduction and what’s actually carried out. Real survey profiles do not validate the teacher’s ‘simulated’ opinions. Without comparison against human responses (I think at least a limited set of matched policies is necessary) it remains unclear whether the reconstructed result captures meaningful population heterogeneity or just reproduces LLM artifacts. In other words, the ‘teacher’ should be validated at least to some extent, otherwise why would practitioners want a cheap reconstruction of it?
-
Missing closely matched baselines. Comparisons with structured matrix completion or multi-task kernel methods would much help establish how much SCOPE adds beyond standard approaches that already uses person and policy structure. It is also unclear how the experiment will perform on the scenarios that practitioners would actually care about, and confounded by the weakness point 1.
Questions for authors
Please refer to the weakness section.
7 comments
Response to Weakness 1 (2/2): Demographic heterogeneity and supporting research
Demographic heterogeneity: pairwise ordering accuracy and Spearman correlation. We compare teacher and human group means for age, education and income, using the same retained groups and each question's verified ordered scale. Category counts are 4/6/7 for WVS and 4/5/5 for OpinionQA. Within each question and attribute, all unordered group pairs score 1 for a correct direction, 0 for a reversal and 0.5 for a model tie; human ties are excluded. Differences in normalized means ≤10⁻¹⁰ count as ties; none occur here. Identical predictions for all groups score 50%.
We average pair scores within each question, then questions equally; the last column averages attributes equally. Spearman correlation uses the same group means and aggregation. Cells in Tables Rmkks-1-2 and Rmkks-1-3 report ordering accuracy / Spearman correlation; both are higher-is-better.
Table Rmkks-1-2. Demographic ordering on all 59 selected WVS7 questions.
| Teacher | Age | Education | Income | Three-attribute mean |
|---|---|---|---|---|
| Qwen2.5-3B-Instruct | 58.5% / 0.186 | 40.8% / -0.240 | 51.3% / 0.024 | 50.2% / -0.010 |
| Qwen3.5-4B | 57.3% / 0.163 | 61.6% / 0.300 | 59.7% / 0.251 | 59.6% / 0.238 |
| Gemma-4-E4B | 59.3% / 0.193 | 61.7% / 0.297 | 62.8% / 0.349 | 61.3% / 0.280 |
| Qwen3.8-27B | 61.9% / 0.237 | 67.0% / 0.429 | 66.8% / 0.455 | 65.2% / 0.373 |
| gpt-6.1-terra | 69.2% / 0.383 | 77.1% / 0.636 | 71.1% / 0.528 | 72.5% / 0.516 |
| DeepSeek-V4.1-Flash | 70.9% / 0.424 | 69.2% / 0.459 | 70.0% / 0.516 | 70.0% / 0.466 |
Table Rmkks-1-3. Demographic ordering on all 60 selected OpinionQA W54 questions.
| Teacher | Age | Education | Income | Three-attribute mean |
|---|---|---|---|---|
| Qwen2.5-3B-Instruct | 47.8% / -0.077 | 50.9% / 0.022 | 46.8% / -0.107 | 48.5% / -0.054 |
| Qwen3.5-4B | 54.7% / 0.097 | 49.9% / -0.010 | 56.2% / 0.167 | 53.6% / 0.084 |
| Gemma-4-E4B | 49.4% / -0.037 | 53.3% / 0.090 | 53.8% / 0.108 | 52.2% / 0.054 |
| Qwen3.8-27B | 65.3% / 0.337 | 60.1% / 0.238 | 66.7% / 0.408 | 64.0% / 0.328 |
| gpt-6.1-terra | 72.8% / 0.503 | 65.7% / 0.368 | 70.8% / 0.497 | 69.8% / 0.456 |
| DeepSeek-V4.1-Flash | 75.8% / 0.590 | 65.3% / 0.353 | 68.8% / 0.417 | 70.0% / 0.453 |
Tables Rmkks-1-1–1-3 show recovery of both human opinions and demographic differences. Under the same logits protocol, Qwen2.5-3B → Qwen3.5-4B → Qwen3.8-27B yields lower subgroup TV and higher mean ordering accuracy: 50.2% → 59.6% → 65.2% on WVS, and 48.5% → 53.6% → 64.0% on OpinionQA; Spearman correlation also improves. Terra JSON reaches 72.5%/69.8% ordering accuracy and DeepSeek JSON 70.0%/70.0%, alongside lower TV. Ordering measures demographic differences' direction; TV measures agreement with human response proportions.
Prior research demonstrates that LLMs can simulate human opinions and behavior. Argyle et al. (Political Analysis, 2023) show that GPT-3 conditioned on demographic and political profiles matches individual presidential votes in 85–89% of cases across three ANES election surveys. Park et al. (2026, v3) show that agents built from interviews and surveys for 1,052 people predict held-out General Social Survey answers with accuracy equal to 86% of participants' own test–retest consistency. EconAgent (ACL 2024) exhibits heterogeneous economic decisions: consumption propensity increases across age groups, broadly matching prior empirical patterns. Its economy also reproduces the negative Phillips-curve and Okun's-law relationships (r=−0.619 and −0.918). Survey-specific fine-tuning can improve agreement further:** Cao et al. (NAACL 2025) fine-tune Llama3-8B-Instruct on survey distributions, improving mean WVS 1−JSD from 0.613 to 0.823 and EMD from 0.136 to 0.067 on splits with unseen countries and/or questions. Our teachers receive no such fine-tuning.
This motivates our Introduction's explicit focus on “population-response reconstruction”: as LLM opinion simulation improves, how can a small query budget recover the complete persona–policy response table for a chosen teacher and prompting protocol?
Response to Weakness 2 (1/2): Structured baselines and iterative reconstruction
“Comparisons with structured matrix completion or multi-task kernel methods would much help establish how much SCOPE adds beyond standard approaches that already uses person and policy structure.”
We have added complete IMC and MTK comparisons across all four teachers and all three levels of the original paper’s benchmark. Both methods use the same public features and query budgets, with fitting and model selection restricted to paid teacher answers.
- IMC fits persona main effects, policy main effects, and a rank-four feature-conditioned interaction in centered-log-ratio (CLR) response coordinates: . A fixed-seed 75/25 split of paid answers selects L2 regularization from 0.01/0.1/1 using probability-space Brier loss; the model is then refit on all paid answers.
- MTK fits CLR responses with a persona RBF × policy RBF product kernel and an intercept. Leave-one-out Brier loss on paid answers selects a single kernel from 27 bandwidth/regularization combinations. Both baselines query uniformly without replacement.
Complete comparison and an explicit strongest-baseline denominator. We use the original benchmark and budget settings. We first average over all tasks within each seed, then report the mean and sample standard deviation of full-pool TV across three seeds.
Table Rmkks-2-1. Full-pool TV across all 28 benchmark conditions. Results for Random–MLP, Random–XGBoost, BAIT and TypiClust appear in the paper's §4.2, Tables 1–2 (p. 8) and Table 3 (p. 9).
| Teacher | Level | B | IMC | MTK | SCOPE | Gain over the strongest baseline (%) |
|---|---|---|---|---|---|---|
| Qwen2.5-3B | family | 16 | 0.3107±0.0163 | 0.2397±0.0080 | 0.2375±0.0044 | 0.9 |
| Qwen2.5-3B | family | 32 | 0.2696±0.0160 | 0.1903±0.0086 | 0.1791±0.0115 | 5.9 |
| Qwen2.5-3B | family | 64 | 0.2321±0.0120 | 0.1451±0.0068 | 0.1289±0.0031 | 11.2 |
| Qwen2.5-3B | domain | 160 | 0.2085±0.0050 | 0.1409±0.0021 | 0.1249±0.0015 | 11.4 |
| Qwen2.5-3B | domain | 320 | 0.1603±0.0036 | 0.1150±0.0011 | 0.1022±0.0005 | 11.2 |
| Qwen2.5-3B | domain | 640 | 0.1474±0.0038 | 0.0968±0.0008 | 0.0856±0.0002 | 11.6 |
| Qwen2.5-3B | global | 640 | 0.1636±0.0019 | 0.1358±0.0029 | 0.1244±0.0020 | 8.4 |
| Qwen3.5-4B | family | 16 | 0.1180±0.0035 | 0.1113±0.0035 | 0.1109±0.0030 | 0.4 |
| Qwen3.5-4B | family | 32 | 0.0975±0.0054 | 0.0868±0.0058 | 0.0809±0.0012 | 6.8 |
| Qwen3.5-4B | family | 64 | 0.0797±0.0051 | 0.0656±0.0011 | 0.0592±0.0006 | 9.8 |
| Qwen3.5-4B | domain | 160 | 0.0854±0.0012 | 0.0727±0.0017 | 0.0632±0.0006 | 13.0 |
| Qwen3.5-4B | domain | 320 | 0.0779±0.0010 | 0.0579±0.0012 | 0.0517±0.0003 | 10.8 |
| Qwen3.5-4B | domain | 640 | 0.0747±0.0012 | 0.0481±0.0004 | 0.0436±0.0001 | 9.4 |
| Qwen3.5-4B | global | 640 | 0.0879±0.0011 | 0.0764±0.0007 | 0.0701±0.0008 | 8.3 |
| Gemma-4-E4B | family | 16 | 0.3348±0.0042 | 0.2839±0.0156 | 0.2791±0.0133 | 0.04 |
| Gemma-4-E4B | family | 32 | 0.3205±0.0109 | 0.2131±0.0239 | 0.1995±0.0016 | 6.4 |
| Gemma-4-E4B | family | 64 | 0.2984±0.0118 | 0.1542±0.0087 | 0.1363±0.0011 | 11.6 |
| Gemma-4-E4B | domain | 160 | 0.2866±0.0021 | 0.1668±0.0022 | 0.1520±0.0016 | 8.9 |
| Gemma-4-E4B | domain | 320 | 0.2225±0.0038 | 0.1348±0.0029 | 0.1207±0.0018 | 10.4 |
| Gemma-4-E4B | domain | 640 | 0.1786±0.0021 | 0.1090±0.0014 | 0.0993±0.0006 | 8.8 |
| Gemma-4-E4B | global | 640 | 0.2079±0.0010 | 0.1678±0.0015 | 0.1582±0.0048 | 5.7 |
| Qwen3.8-27B | family | 16 | 0.2590±0.0081 | 0.2212±0.0078 | 0.2205±0.0097 | 0.3 |
| Qwen3.8-27B | family | 32 | 0.2365±0.0021 | 0.1823±0.0080 | 0.1778±0.0031 | 2.2 |
| Qwen3.8-27B | family | 64 | 0.2097±0.0135 | 0.1423±0.0034 | 0.1275±0.0004 | 8.2 |
| Qwen3.8-27B | domain | 160 | 0.2082±0.0150 | 0.1491±0.0036 | 0.1342±0.0009 | 9.5 |
| Qwen3.8-27B | domain | 320 | 0.1599±0.0037 | 0.1259±0.0020 | 0.1130±0.0004 | 8.1 |
| Qwen3.8-27B | domain | 640 | 0.1466±0.0041 | 0.1059±0.0017 | 0.0959±0.0003 | 4.3 |
| Qwen3.8-27B | global | 640 | 0.1660±0.0047 | 0.1414±0.0020 | 0.1346±0.0029 | 4.8 |
As shown in Table Rmkks-2-1, SCOPE has the lowest mean TV in all 28 conditions. Its global improvements over the strongest of the six baselines are 8.43%, 8.25%, 5.75%, and 4.83% for Qwen2.5, Qwen3.5, Gemma, and Qwen3.8, respectively.
Iterative coupling of the two bottlenecks. Paid answers calibrate a mixture of persona, policy and interaction structures. Its probability-space covariance proxy combines within-structure uncertainty and between-structure disagreement. Queries are valued by their exact correction plus expected gains elsewhere in the pool; new answers recalibrate sharing: calibration → full-pool query value → new answers → recalibration. This matches our findings of policy-dependent sharing and cross-pair gains.
Why smaller gains at B=16? At family level, all 16 queries go to the shared initialization, leaving no budget for further active querying and recalibration. Gains over the strongest baseline in Table Rmkks-2-1 are 0.04%–0.94%, rising to 2.15%–6.82% at B=32 and 8.23%–11.61% at B=64 across the same four teachers. Together with the factorial ablation (Table 6, Appendix F.1, p. 21), this trend supports repeated coupling of sharing and query allocation.
Response to Weakness 1 (1/2): Matched-human validation and teacher agreement
Thank you for your constructive review and recognition of our clear problem formulation. We address your concerns point by point below.
Addressed under Weaknesses 1–2 below.
We validate teachers against matched human survey questions and demographic differences.
Data and selection. We use US respondents from World Values Survey Wave 7 (WVS7) and Pew Research Center's American Trends Panel Wave 54 (Pew ATP W54), released through OpinionQA (Santurkar et al., ICML 2023). WVS data · OpinionQA paper · Data release.
From English-interviewed respondents in each survey, we sample 1,024 prediction personas without replacement, proportionally within sex × age strata. Before teacher collection, we fix complete batteries of policy, institutional, social and economic questions with verified ordered scales. We retain original wording and four-, five- or ten-category options. All 59 selected WVS and 60 selected W54 questions are evaluated, forming 60,416 and 61,440 pairs.
_W54; wording and options follow the original Pew questionnaire.We also test stronger, higher-query-cost API teachers, illustrating the value of query savings: gpt-6.1-terra and DeepSeek-V4.1-Flash generate JSON probability vectors, normalized by their positive sum.
Metrics. Lower TV and higher accuracy indicate better agreement. Accuracy averages whether the highest-probability option matches each valid respondent–question answer; ties use the first original option. The uniform reference assigns equal probabilities to legal options; its accuracy is the expected accuracy of uniform random choice, 1/K for K options.
Table Rmkks-1-1. Teacher–human agreement on WVS and OpinionQA.
Both API teachers and Qwen3.8-27B beat uniform in TV on both surveys; all six teachers beat it in accuracy. Terra and DeepSeek reduce TV by 42.12%–50.39% relative to uniform. Qwen3.8 has the lowest TV among local teachers. From Qwen2.5-3B to Qwen3.8-27B, WVS/OpinionQA accuracy rises from 37.04%/30.48% to 42.00%/40.01%; Terra reaches 44.17% on WVS and DeepSeek 43.13% on OpinionQA. These results support improved opinion simulation with the stronger teachers evaluated.