- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes the SCBO framework to address three limitations of the one-question-per-prompt paradigm in LLM-based social survey simulation: repeated encoding of shared context, the inability to share reference answers across questions, and independent answer prediction. SCBO batches questions through semantic clustering, constructs a shared reference bank, and orders questions and references from easy to hard, thereby reducing token costs and inference time while exploiting correlations among answers.
Critical strengths
1,The method is well targeted, addressing context amortization, reference-answer sharing, and dependencies among answers.
2,The experiments cover multiple survey datasets and LLMs, with a relatively large experimental scale.
Critical weaknesses
1, The Related Work section discusses batch prompting, semantic grouping, and Batch Permutation and Ensembling, but the experiments mainly compare SCBO with the one-question-per-prompt baseline.
2, SCBO changes question grouping, reference-bank size, reference visibility, question order, and reference order simultaneously. Therefore, even if the complete method outperforms the baseline, it is difficult to determine which factor is responsible for the gains.
3, The theoretical justification for semantic clustering and question ordering is relatively weak.
Questions for authors
How does the respondent-coverage filter affect the composition of the evaluation population, and how many respondents are excluded under each reference budget?
10 comments
Response for W1 (Part 2/2)
Execution time advantages. SCBO has a shorter recorded execution time than every baseline in all 16 model–dataset settings. For example, with GPT-6.1-sol(low) on WVS at batch size 32, SCBO takes 127.2 seconds, compared with 586.7 seconds for BPE and 712.6 seconds for BPE+SEAS. SCBO times on WVS/GSS include amortized preprocessing overhead, whereas recorded times on ANES/BSA exclude offline preprocessing. These results show that, under the evaluated workloads, SCBO achieves higher predictive accuracy with lower computational usage and shorter execution time.
Batch size = 32, 1-budget. Each cell reports weighted tokens (M) / execution time (seconds), where weighted tokens = input tokens + 4 × output tokens. Lower is better.
| Model | Method | WVS | GSS | ANES | BSA |
|---|---|---|---|---|---|
| DeepSeek Flash | BPE | 25.292 / 78.8 | 17.362 / 107.6 | 23.462 / 73.7 | 13.534 / 44.6 |
| DeepSeek Flash | BPE+SEAS | 18.175 / 59.6 | 12.664 / 70.8 | 16.807 / 56.0 | 9.427 / 33.0 |
| DeepSeek Flash | ADP | 13.894 / 88.8 | 8.730 / 59.4 | 12.539 / 77.6 | 7.168 / 43.4 |
| DeepSeek Flash | OBP | 7.559 / 101.6 | 5.222 / 78.5 | 7.812 / 90.5 | 4.392 / 57.7 |
| DeepSeek Flash | SCBO | 6.935 / 23.5 | 4.823 / 40.2 | 6.777 / 22.3 | 4.018 / 11.4 |
| MiMo-v2.6-Flash | BPE | 24.900 / 383.0 | 16.358 / 420.4 | 23.531 / 508.0 | 13.429 / 289.9 |
| MiMo-v2.6-Flash | BPE+SEAS | 17.764 / 280.1 | 12.516 / 251.6 | 17.087 / 324.1 | 9.594 / 181.4 |
| MiMo-v2.6-Flash | ADP | 13.359 / 453.1 | 8.433 / 355.6 | 11.959 / 465.7 | 6.928 / 256.4 |
| MiMo-v2.6-Flash | OBP | 7.997 / 738.7 | 5.248 / 312.1 | 7.649 / 398.6 | 4.631 / 270.4 |
| MiMo-v2.6-Flash | SCBO | 7.149 / 100.5 | 4.824 / 90.4 | 7.259 / 116.8 | 4.055 / 73.6 |
| DeepSeek-V4-Pro | BPE | 25.289 / 169.9 | 17.137 / 165.8 | 23.303 / 121.8 | 13.532 / 74.2 |
| DeepSeek-V4-Pro | BPE+SEAS | 17.727 / 117.0 | 11.767 / 101.8 | 16.155 / 91.5 | 9.292 / 58.1 |
| DeepSeek-V4-Pro | ADP | 13.893 / 238.2 | 8.732 / 155.3 | 12.538 / 144.1 | 7.167 / 80.3 |
| DeepSeek-V4-Pro | OBP | 7.922 / 270.2 | 5.488 / 198.3 | 7.956 / 182.8 | 4.574 / 105.7 |
| DeepSeek-V4-Pro | SCBO | 7.070 / 47.9 | 5.151 / 55.9 | 6.836 / 38.6 | 4.136 / 26.7 |
| GPT-6.1-sol(low) | BPE | 35.903 / 586.7 | 31.165 / 614.2 | 36.610 / 819.3 | 22.036 / 370.4 |
| GPT-6.1-sol(low) | BPE+SEAS | 36.676 / 712.6 | 36.412 / 695.2 | 36.786 / 1143.2 | 20.044 / 456.6 |
| GPT-6.1-sol(low) | ADP | 21.878 / 630.8 | 16.171 / 489.8 | 20.127 / 420.8 | 10.440 / 311.6 |
| GPT-6.1-sol(low) | OBP | 30.018 / 583.8 | 29.271 / 493.2 | 77.074 / 2080.5 | 24.281 / 1023.0 |
| GPT-6.1-sol(low) | SCBO | 8.999 / 127.2 | 7.197 / 163.1 | 10.062 / 218.4 | 5.676 / 109.0 |
Methodological Distinctions from General-Purpose Batch-Prompting Methods
-
Compared with BPE/SEAS: BPE mitigates sensitivity to ordering through multiple rounds of random permutations and ensembling, while SEAS reduces the number of ensemble rounds. SCBO retains target-specific relevant history, constructs a shared and deduplicated reference bank, and applies deterministic easy-to-hard question ordering and reference alignment. Under normal operation, SCBO requires only one call per batch, reducing the overhead of multi-round ensembling.
-
Compared with ADP: ADP uses previously generated question–answer pairs within a batch as demonstrations for subsequent predictions, requiring questions to be repeated in the output. SCBO instead retains target-specific relevant history, constructs a shared and deduplicated reference bank, orders questions according to historical support, and semantically aligns references with the question sequence. Both methods can typically operate with one call per batch. However, ADP’s question repetition increases output tokens, which are typically priced higher than input tokens, thereby increasing inference costs.
-
Compared with OBP: Beyond question grouping and demonstration selection, SCBO addresses survey-template noise, target-specific associations with personal history, and within-batch ordering. Question instantiation extracts topics, intents, and entities to support grouping and retrieval, while question ordering and reference alignment organize how evidence is used. Furthermore, in our adaptation to survey prediction, OBP leaves some target questions out during grouping, requiring additional single-question fallback calls and increasing both the number of calls and execution overhead.
Response for W2 (Part 1/2)
Thank you for highlighting this concern. We have supplemented the evaluation with an analysis of reference-bank size, a factorial experiment on question and reference ordering, separate ablations of batching and history selection, and prefix-answer interventions to better distinguish the roles of individual components.
(1) A larger shared reference bank does not guarantee higher accuracy
In the original manuscript, the shared reference-bank budget is defined as:
where is the per-question reference budget and is the batch size. For example, with Qwen3.7-Max under 1-budget, increasing the reference-bank budget from 12 to 96 references decreases smoothed Acc from 54.43% to 52.66%. Under 3-budget, increasing the budget from 12 to 144 references decreases smoothed Acc from 55.42% to 54.01%. These results show that expanding the shared reference bank does not necessarily improve prediction, and SCBO's gains cannot simply be attributed to a larger reference bank.
(2) Separating question ordering and reference ordering
We evaluate all six combinations of three question orders and two reference orders, keeping batch membership, reference content, and reference counts unchanged. The fixed reference order is inherited from full SCBO and is not realigned when the question order changes, allowing the two ordering factors to be examined separately.
Easy-to-hard ordering with fixed references achieves the highest Acc on all three datasets. With easy-to-hard question ordering held fixed, randomizing reference order decreases Acc by 1.99, 1.00, and 1.20 percentage points on WVS, GSS, and ANES, respectively. With reference order held fixed, switching to hard-to-easy ordering (inverse curriculum) decreases Acc by 1.05, 1.18, and 1.96 percentage points, respectively. These results indicate that both question ordering and reference ordering affect predictive performance.
DeepSeek Flash, batch size = 32; Acc (%).
| Question order | Reference order | WVS | GSS | ANES |
|---|---|---|---|---|
| Easy-to-hard | Fixed (full SCBO) | 52.25 | 52.08 | 53.68 |
| Easy-to-hard | Random | 50.26 | 51.08 | 52.48 |
| Hard-to-easy (inverse curriculum) | Fixed | 51.20 | 50.90 | 51.72 |
| Hard-to-easy (inverse curriculum) | Random | 50.52 | 50.97 | 51.93 |
| Random | Fixed | 50.36 | 50.49 | 52.01 |
| Random | Random | 51.25 | 49.90 | 52.14 |
Response for W1 (Part 1/2)
Thank you for pointing out this gap between the related-work discussion and the experimental comparisons. We have now added direct comparisons with BPE, BPE+SEAS, ADP, and OBP across four models and four survey datasets, using batch size 32 and a 1-budget setting. We report predictive performance, weighted token usage, and execution time, and will incorporate these comparisons into the revised main paper.
Empirical Advantages over General-Purpose Batch-Prompting Methods
Supplementary experimental setup. As some model versions used in the original manuscript have been updated, we conducted supplementary comparisons during the rebuttal phase using DeepSeek Flash (DeepSeek-V4.1-Flash-1017), DeepSeek Pro, MiMo-V2.6-Flash, and GPT-6.1-sol(low). The experiments cover four datasets—WVS, GSS, ANES, and BSA—with batch size 32 and a 1-budget setting, comparing SCBO against BPE, BPE+SEAS, ADP, and OBP.
Predictive performance. Using Acc over valid outputs, SCBO achieves the highest accuracy in all 16 model–dataset settings, outperforming the strongest baseline in each setting by 1.25–7.60 percentage points. For example, on WVS, DeepSeek Pro improves from 49.84% for the strongest baseline to 54.32%, while GPT-6.1-sol(low) improves from 50.58% to 58.18%. SCBO also achieves the lowest MAE and highest F1 in all 16 settings. These results demonstrate SCBO’s predictive advantages in direct comparisons with existing batch-prompting methods.
Each cell reports Acc↑ / MAE↓ / F1↑, with Acc and F1 expressed as percentages.
Computational usage advantages. SCBO achieves higher accuracy while consuming fewer weighted tokens. For example, with GPT-6.1-sol(low) on WVS at batch size 32, SCBO achieves an Acc of 58.18%, compared with 49.43% for BPE, while reducing weighted token usage by approximately 75%. Across comparisons with complete usage records, SCBO reduces weighted token usage by 5.1%–86.9% relative to the baselines. This metric is defined as input tokens + 4 × output tokens to compare input and output overhead.