acceptodds
Under review as a conference paper at ICLR 2027

Adaptive resource allocation for effective and efficient LLM social survey simulation

Abstract

Large Language Models (LLMs) enable scalable social survey simulation, yet existing pipelines typically assign the same strong general-purpose model and a fixed, often large, respondent history to every respondent–question request. This uniform design overlooks three properties of survey simulation. First, a stronger general-purpose model may rely more on its own knowledge and deviate from respondent-specific evidence, while also costing more. Second, more history can help when evidence is insufficient, but irrelevant responses may introduce noise and lengthen the input. Third, changing the history budget can change the preferred model, while changing the model can change the preferred history budget. These considerations call for joint request-level allocation under an explicit accuracy–cost trade-off. We propose E2Sim, an adaptive resource allocation framework that treats each model–history configuration as a joint request-level decision. Conditioned on the respondent persona, ranked response history, and target question, a lightweight policy predicts the accuracy–cost score of every configuration and selects the most suitable one. To reduce sensitivity to incomplete or variable histories, we apply respondent-history drop-and-swap augmentation; to handle differences in allocation ambiguity, we use a margin-based curriculum that progresses from clearly separated configurations to harder cases. Experiments on four real-world social-survey datasets, multiple model pools, and different history-budget spaces demonstrate improvements over oracle best-fixed configurations, with gains of up to 6.3 percentage points in Accuracy and reductions of up to 55.3% in cost. Our code is available at https://anonymous.4open.science/r/E2Sim-DE66.

Primary area:
generative models
Keywords:
large language models, social simulation, survey simulation, adaptive resource allocation, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZMjyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of jointly choosing an LLM mm and the history context length kk for simulating responses to surveys. It defines a scalarized Accuracy-Cost Trade-off (ACT) objective ACT=Accuracy−α⋅Cost\mathrm{ACT}=\mathrm{Accuracy}-\alpha\cdot\mathrm{Cost}, and proposes a supervised learning approach that uses existing surveys with real human responses to train a predictor g^θ\widehat{g}_{\theta} that predicts the ACT score for each model-history pair (m,k)(m,k). For a new survey simulation request, it chooses a model-history pair with the highest predicted ACT score. Experiments are conducted on four social-survey datasets, with three LLMs and four history budgets.

Critical strengths

The experiments show that across the four datasets, the proposed approach consistently improves the simulation accuracy and cost compared with the best fixed model-history configuration in hindsight.

Critical weaknesses

My main concern is the limited methodological and practical contributions. The proposed approach of scalarizing accruacy and cost, supervised learning of ACT scores, and choosing a model-history pair to maximize the predicted ACT is quite straightfoward. While jointly choosing the model and history length sounds reasonable, the paper does not demonstrate its advantages and benefits over existing approaches for adaptive inference, such as LLM routing.

Moreover, the proposed method has limited flexibility. First, it requires fixing in advance a set of LLMs and history budgets. As acknowledged in Appendix F, "Introducing a new model or history budget requires evaluating the added configurations and updating the allocation policy". This also seems to be the case for the parameter α\alpha in ACT. The paper does not offer guidance on how one should choose α\alpha in practice, either. Second, the offline supervision stage requires generating responses from all LLM-history pairs for all surveys in the training set, which can incur a high computational cost, especially when there is a large number of LLMs and history budgets. While the paper gives an offline-supervision amortization cost analysis, it is based on the ACT score, instead of the inference cost alone.

Another concern is that the paper does not demonstrate whether the learned allocation policy generalizes to new survey questions and respondents, which is an important consideration and motivation in LLM-based survey simulation. In the experiments, the training and testing sets are randomly splitted over question-respondent pairs, instead of holding out new questions or respondents. Thus, it is very likely that the survey questions and respondents in the testing set already appear in the training set.

Questions for authors
  1. In practice, how should one choose α\alpha in the ACT?
  2. How does the proposed approach account for the randomness in the LLM-simulated responses? How sensitive are its results (e.g., the learned allocation policy and the improvement in accuracy and cost) to the randomness?
7 comments
AuthorsOpenReview ↗

Response for W1

Thank you for raising this concern. We agree that scalarizing accuracy and cost and regressing configuration utilities are not, by themselves, new learning techniques. Our contribution lies in the empirical motivation, joint model–history allocation, and task-specific training design for survey simulation.

1. Empirical Findings: Stronger Models and More History Do Not Necessarily Improve Simulation

To the best of our knowledge, we are the first to systematically identify and quantify three phenomena in social survey simulation: stronger general-purpose models are not necessarily more accurate, more history is not necessarily more useful, and the preferred model and history budget are interdependent.

The first two findings motivate request-level model selection and history allocation, while the third supports joint decision-making. These findings challenge the practice of uniformly using a strong model with extensive history and constitute an empirical contribution of this work.

2. Joint Optimization and Comparisons with Routing Baselines

Utility regression is the implementation mechanism; joint configuration selection is the methodological focus. Models differ in their ability to extract useful information from respondent history, and changes in historical evidence can alter which model is preferable. We therefore learn the accuracy–cost utility of each model–history configuration and select jointly over the complete configuration space.

E²Sim selects both the model and the amount of historical information. To evaluate this design, we compared model-only routing, history-budget-only routing, and a two-stage procedure that selects the model before selecting the history budget. The following accuracies use the same validation requests and seed=42:

MethodANESBSAGSSWVS
Model-only routing, fixed k=0k=045.88%39.33%46.80%43.05%
History-budget-only routing, fixed 8B65.80%46.98%53.20%53.93%
Two-stage selection63.75%49.75%54.83%54.55%
E²Sim joint selection68.80%50.45%55.28%56.25%

Joint selection achieves higher accuracy in all these settings. Its SoftAcc-based utility gains over two-stage selection are also positive on all four datasets. These results support jointly accounting for model–history interactions rather than relying on a single allocation dimension or sequential selection.

3. Task-Specific Design: Handling History Variation and Allocation Ambiguity

History augmentation models variation in the evidence available at deployment. For example, when predicting attitudes toward public healthcare spending, training respondent A may have answered questions about healthcare, taxation, and government responsibility, whereas test respondent B may lack the taxation response. Small retrieval-score perturbations may also change the ordering of historical responses. Random dropping and swapping model these two variations, reducing dependence on a fixed history composition and order.

Curriculum learning addresses differences in allocation difficulty across requests. For healthcare policy, one respondent may express consistent historical attitudes, while another may support better services but oppose higher taxes. We use the ACT margin between the best and second-best configurations to measure allocation ambiguity, learning clearly distinguishable cases before introducing near-tied cases. These social examples illustrate the motivation; the actual curriculum is determined by observed configuration performance.

Together, these findings and comparisons support the practical value of jointly selecting models and history budgets, beyond extending the action space alone.

AuthorsOpenReview ↗

Response for W2 & Q1 (Part 1/2)

Thank you for your comments. We agree that the original manuscript did not sufficiently discuss configuration updates, the choice of α\alpha, and offline supervision costs. We address these concerns through additional experiments on incremental adaptation, α\alpha-conditioned routing, training-data requirements, and inference-cost break-even analysis.

1. New models and history budgets can be incorporated through incremental adaptation

Although the policy operates over a predefined configuration space, adding configurations does not require regenerating responses for existing configurations or retraining the policy from scratch. We evaluate two types of expansion:

  • Adding a model: expanding from 8B to 8B + 32B, with history budgets 0,1,3,5\\{0,1,3,5\\}.
  • Adding a history budget: expanding the budgets from 0,1,3\\{0,1,3\\} to 0,1,3,5\\{0,1,3,5\\} for the 8B + 32B model pool.

During adaptation, we reuse the trained policy parameters and output weights for existing configurations, initialize only the additional outputs, and fine-tune using different amounts of supervision. The following table reports the accuracy–cost utility Usoft=SoftAcc−0.05CU_{\mathrm{soft}}=\mathrm{SoftAcc}-0.05C; comparisons should be made within each row.

ExpansionDatasetOriginal policy20% adaptation40% adaptationFull-data training from scratch
New modelANES0.790690.792030.792700.80686
New modelBSA0.760840.763690.772030.77539
New modelGSS0.740240.736860.734750.74544
New modelWVS0.748450.752070.755770.76183
New budgetANES0.820380.815370.820150.82198
New budgetBSA0.787730.786220.788720.78570
New budgetGSS0.758840.755970.758640.75670
New budgetWVS0.758020.759460.763820.76766

These results demonstrate the feasibility of incremental adaptation. With 20% of the training data, corresponding to 3,200 requests, adding the model requires evaluating only the four new 32B configurations, totaling 12,800 additional training calls. Adding the budget requires evaluating only two new configurations, totaling 6,400 calls. Existing supervision is reused, and validation costs for the new configurations are counted separately. Adaptation uses 10 epochs, whereas the full-data reference uses 20 epochs.

2. Conditioning on α\alpha reduces the need for retraining when preferences change

To address whether changing α\alpha requires retraining, we introduce an α\alpha-conditioned policy. We encode α\alpha in the policy input and train a single router using utility supervision under multiple α\alpha values, allowing configuration selection to respond to different cost preferences.

We train with α∈0.01,0.05,0.10,0.20,0.30,0.40,0.50\alpha \in \\{0.01,0.05,0.10,0.20,0.30,0.40,0.50\\} and evaluate interpolation at unseen α\alpha values. All entries below are measured validation results without smoothing. Each cell reports Accuracy (%) / Normalized Cost.

α\alpha values used during training:

α\alphaANESBSAGSSWVS
0.0167.80 / 0.318651.45 / 0.432154.50 / 0.408255.75 / 0.3242
0.0569.13 / 0.391551.38 / 0.378354.15 / 0.294655.98 / 0.2965
0.1068.53 / 0.312051.30 / 0.331853.95 / 0.269955.75 / 0.2728
0.2067.90 / 0.273650.20 / 0.278453.70 / 0.231355.33 / 0.2413
0.3067.38 / 0.259949.80 / 0.252153.15 / 0.216754.88 / 0.2180
0.4067.35 / 0.250549.18 / 0.236053.08 / 0.208454.48 / 0.2031
0.5067.33 / 0.240848.40 / 0.224952.88 / 0.201853.85 / 0.1920

Selected unseen α\alpha values for interpolation:

α\alphaANESBSAGSSWVS
0.0369.10 / 0.375250.80 / 0.336754.23 / 0.282655.95 / 0.2948
0.07568.85 / 0.324950.40 / 0.313153.63 / 0.250255.25 / 0.2425
0.1567.70 / 0.279449.93 / 0.258053.30 / 0.222855.05 / 0.2288
0.2567.65 / 0.267249.70 / 0.247853.38 / 0.214855.08 / 0.2145
AuthorsOpenReview ↗

Response for W2 & Q1 (Part 2/2)

At the training α\alpha values, the policy generally selects less expensive configurations as the cost penalty increases, showing that separate policies are not required for each α\alpha. The displayed unseen values also produce different accuracy–cost trade-offs, although this does not establish reliable interpolation for arbitrary α\alpha.

In practice, we recommend choosing α\alpha according to deployment requirements:

  • Given a cost budget: Evaluate a grid of α\alpha values on the validation set, record the corresponding accuracy and average inference cost, and select the α\alpha achieving the highest accuracy within the budget.
  • Given a minimum accuracy requirement: Select the α\alpha yielding the lowest inference cost among settings that satisfy the accuracy requirement.
  • Without an explicit constraint: Examine the validation accuracy–cost curve and select a point on its Pareto frontier according to application preferences. A knee point can serve as a starting point, where further cost reductions begin to require larger accuracy sacrifices.

For the objective U=A−αCU=A-\alpha C, α\alpha specifies how much accuracy the user is willing to exchange for a unit reduction in normalized cost. For example, when AA is expressed on a 0–1 scale, α=0.05\alpha=0.05 means that a cost reduction of 0.10 offsets an accuracy decrease of 0.005, or 0.5 percentage points, in the objective.

Because the learned policy may not respond monotonically to α\alpha, the final setting should be selected using its measured validation accuracy and cost, rather than assuming that increasing α\alpha always satisfies a tighter budget. Once selected, α\alpha is supplied to the conditioned router at deployment without additional policy training.

3. Effective routing does not require supervision over the entire training pool

We conduct additional training using 20%, 40%, 60%, and 80% of the full training pool while keeping the original validation set unchanged. The full training pool contains 16,000 requests.

Training fractionANES AccuracyBSA AccuracyGSS AccuracyWVS Accuracy
20%66.55%48.45%54.50%54.93%
40%66.65%50.20%56.03%55.73%
60%67.65%50.63%55.63%56.43%
80%68.55%50.38%56.53%55.98%
100% reference68.80%50.45%55.28%56.25%

Smaller supervision subsets already support effective allocation policies. For example, BSA with 40% of the data and ANES with 80% achieve accuracy close to their full-data references.

4. Additional break-even analysis based solely on inference costs

We agree that amortizing ACT gains does not replace an analysis of inference-cost savings alone. We therefore calculate how many deployment requests are needed for savings relative to the best fixed configuration (BF) to cover the additional offline supervision cost.

Under the assumption that BF responses for the same training requests would already be generated and can be reused, the additional supervision cost is:

Cofflineextra=Cofflineall−CofflineBF.C_{\mathrm{offline}}^{\mathrm{extra}} = C_{\mathrm{offline}}^{\mathrm{all}} - C_{\mathrm{offline}}^{\mathrm{BF}}.

The corresponding break-even request count is:

NBE=⌈CofflineextraC‾BF−C‾E2Sim⌉.N_{\mathrm{BE}} = \left\lceil \frac{C_{\mathrm{offline}}^{\mathrm{extra}}} {\overline C_{\mathrm{BF}}-\overline C_{\mathrm{E^2Sim}}} \right\rceil.

Using full-data supervision and the corresponding policies, we obtain:

DatasetInference-cost break-even requests
ANES190,964
BSA180,149
GSS716,158
WVS93,023

These additional results show that, although offline supervision entails an initial investment, this investment can be reduced through fewer supervised requests, reuse of existing configuration evaluations, and incremental policy updates.

Reviewer_vKyAOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary
  • Task: LLM-based survey simulation, i.e., predicting a respondent's answer from their persona, past answers, and a new question.
  • Method (E²Sim): A ModernBERT scorer predicts the accuracy–cost score (ACT) of each (model, history-budget) configuration and picks the best one for each request. It is trained on offline evaluations of all configurations, with history drop/swap augmentation and a margin-based curriculum.
  • Results: On four surveys (ANES, BSA, GSS, WVS), E²Sim beats the oracle best fixed configuration, with up to +6.3 pp accuracy and up to 55.3% lower cost.
Critical strengths
  • Clear empirical motivation: Fig. 1 shows that the best model and the best history budget vary by request and interact.
  • Simple, practical method: One lightweight scorer jointly picks model and history budget, and the selector's own cost is counted.
  • Broad evaluation: Four surveys, two model pools, varied history-budget spaces, α sweeps, and ablations.
Critical weaknesses
  1. Limited significance: LLM inference cost is negligible compared with human survey cost. The more pressing problem is bias in LLM-simulated responses, addressed by debiasing methods such as design-based supervised learning (Egami et al., 2023) and prediction-powered inference (Angelopoulos et al., 2023). Saving tokens across LLM configurations is a secondary concern.
  2. Limited methodological novelty: The method is essentially LLM routing with the action space extended to (model, history budget). Per-action utility regression, input augmentation, and margin curricula are standard, which puts the contribution below the bar for ICLR.
  3. Untested retrieval assumption: Retrieval assumes that textually similar questions are the most predictive, but textual similarity does not imply predictive power. Diverse items may better capture a respondent's overall profile. The accuracy drop at larger kk in Fig. 1(b) may reflect redundant or poor retrieval rather than a real need for adaptive budgets.
  4. Mismatched evaluation and loss: Exact-match accuracy ignores the ordinal structure of Likert-type items: predicting 4 when the answer is 5 counts as wrong just as much as predicting 1. Separately, Smooth-L1 regression is applied to near-binary, single-run labels, even though inference uses only the argmax.
Questions for authors
  1. Can the authors verify that textual similarity implies predictive power? For example, compare top-similarity retrieval with random selection and bottom-similarity retrieval.
  2. How does E2SimE^2Sim compare with a ModernBERT fine-tuned to predict answers directly on the same training data?
7 comments
AuthorsOpenReview ↗

Response for W1

Thank you for highlighting the importance of bias correction in LLM-based survey simulation. We agree that reducing bias and ensuring reliable downstream inference are essential. However, improving the efficiency of large-scale simulation remains a meaningful and complementary research objective.

Large-Scale Simulation Still Incurs Substantial Costs

Even when individual LLM calls are inexpensive, simulating millions of respondent–question pairs can cost tens of thousands of dollars. Improving inference efficiency therefore remains practically important.

The table below estimates the token usage and API costs required to simulate one round of responses for all valid respondent–question pairs in each dataset.

DatasetRespondentsQuestionsValid Respondent–Question PairsAvg. Context TokensAvg. Question TokensInput Tokens (M)Output Tokens (M)GPT-6 (USD)Fable-5 (USD)Gemini-3.8 (USD)
WVS97,22028525,038,918491.170.614,064125$11,126+$58,552+$11,017+
ANES5,5219523,541,672551.462.92,17618$1,714+$9,026+$1,698+

For WVS alone, one simulation round is estimated to cost $11,017–$58,552, excluding additional tokens used for chain-of-thought reasoning.

Bias Correction and Resource Allocation Are Orthogonal

Bias correction addresses the reliability of simulated responses or downstream inference, whereas E²Sim determines which model and how much historical information to use for each request. These objectives are orthogonal and complementary.

Bias-corrected LLMs can be incorporated into E²Sim’s candidate pool. After evaluating the added configurations and updating the allocation policy, E²Sim can optimize their use. Advances in bias correction thus provide better simulators without eliminating the value of resource allocation.

The Framework Can Accommodate Better Future Simulators

E²Sim is not tied to a particular LLM. More reliable future simulators can be incorporated by evaluating their quality, cost, and historical-information requirements and updating the allocation policy accordingly.

As long as candidate configurations differ in quality or cost, adaptive selection can remain beneficial.

AuthorsOpenReview ↗

Response for W2

Thank you for your assessment of the methodological novelty. Our main contribution is to identify resource allocation challenges through empirical analysis of survey simulation, formulate joint model–history selection, and develop training mechanisms tailored to this task.

1. Empirical Findings: Stronger Models and More History Do Not Necessarily Improve Simulation

To the best of our knowledge, we are the first to systematically identify and quantify three phenomena in social survey simulation: stronger general-purpose models are not necessarily more accurate, more history is not necessarily more useful, and the preferred model and history budget are interdependent.

The first two findings motivate request-level model selection and history allocation, while the third supports joint decision-making. These findings challenge the practice of uniformly using a strong model with extensive history and constitute an empirical contribution of this work.

2. Joint Optimization: Accounting for Model–History Interactions

Utility regression is the implementation mechanism; joint configuration selection is the methodological focus. Models differ in their ability to extract useful information from respondent history, and changes in historical evidence can alter which model is preferable. We therefore learn the accuracy–cost utility of each model–history configuration and select jointly over the complete configuration space.

3. Task-Specific Design: Handling History Variation and Allocation Ambiguity

History augmentation models variation in the evidence available at deployment. For example, when predicting attitudes toward public healthcare spending, training respondent A may have answered questions about healthcare, taxation, and government responsibility, whereas test respondent B may lack the taxation response. Small retrieval-score perturbations may also change the ordering of historical responses. Random dropping and swapping model these two variations, reducing dependence on a fixed history composition and order.

Curriculum learning addresses differences in allocation difficulty across requests. For healthcare policy, one respondent may express consistent historical attitudes, while another may support better services but oppose higher taxes. We use the ACT margin between the best and second-best configurations to measure allocation ambiguity, learning clearly distinguishable cases before introducing near-tied cases. These social examples illustrate the motivation; the actual curriculum is determined by observed configuration performance.

4. Ablation Analysis: Evaluating the Role of Each Design

The paper’s ablation figure reports the following results. “No Joint Selection” replaces joint allocation with a two-stage procedure that first selects the model and then the history budget.

DatasetMethodAccuracy ↑Normalized Cost ↓
ANESFull Method0.6880.361
ANESNo Curriculum0.6740.382
ANESNo History Augmentation0.6840.377
ANESNo Joint Selection0.6600.377
GSSFull Method0.5540.287
GSSNo Curriculum0.5420.373
GSSNo History Augmentation0.5450.324
GSSNo Joint Selection0.5470.292
WVSFull Method0.5690.270
WVSNo Curriculum0.5520.313
WVSNo History Augmentation0.5670.319
WVSNo Joint Selection0.5500.322

Removing each component corresponds to lower accuracy and higher cost. We therefore hope that the contribution will be assessed through the combination of our empirical findings, joint formulation, and overall framework design.

AuthorsOpenReview ↗

Response for W3 & Q1

Thank you for this suggestion. We agree that textual similarity does not necessarily imply predictive value. We therefore added a controlled comparison of retrieval strategies and clarify how retrieval quality relates to adaptive history allocation.

1. Additional Experiments: Top-Similarity Retrieval Provides Predictive Value

We compared top-similarity retrieval, random selection, and bottom-similarity retrieval across four datasets and four models. For each dataset, all settings used the same 1,000 requests, with k=3k=3 and temperature=0, while keeping the prompt structure and decoding settings consistent. All strategies excluded the target question, and similarity scores were not shown in the prompts.

DatasetModelTop SimilarityRandom SelectionBottom Similarity
ANESQwen3-4B46.70%31.30%32.80%
ANESQwen3-8B59.50%43.50%43.00%
ANESQwen3-14B61.90%44.90%45.50%
ANESQwen3-32B64.80%45.20%43.70%
BSAQwen3-4B40.50%29.10%28.70%
BSAQwen3-8B44.20%37.00%32.60%
BSAQwen3-14B47.70%39.60%37.90%
BSAQwen3-32B49.20%40.20%36.80%
GSSQwen3-4B42.20%32.40%34.10%
GSSQwen3-8B50.30%41.90%41.20%
GSSQwen3-14B52.20%44.00%43.50%
GSSQwen3-32B52.10%44.80%44.30%
WVSQwen3-4B48.10%35.00%35.60%
WVSQwen3-8B50.70%36.40%39.30%
WVSQwen3-14B51.90%41.90%40.50%
WVSQwen3-32B51.80%42.70%41.60%
Macro-average—50.86%39.37%38.82%

Top-similarity retrieval achieved the highest accuracy in all 16 settings, outperforming random selection and bottom-similarity retrieval by 11.49 and 12.04 percentage points on average, respectively.

2. Retrieval Quality and Adaptive Budgets Are Complementary

We agree that diverse historical responses can provide complementary information. For example, when predicting attitudes toward public healthcare spending, healthcare-related responses provide direct evidence, while responses about taxation and trust in government may reveal additional considerations. As the retrieval scope expands, lower-similarity responses may naturally broaden coverage across topics and attitude dimensions, increasing diversity.

The reviewer’s explanation involving redundancy or retrieval quality is therefore compatible with the motivation for adaptive budgets. Even with an effective retrieval strategy, the appropriate amount of history can vary across requests: a few informative responses may be sufficient, whereas additional history may offer limited benefits, introduce interference, and increase cost. Even when accuracy does not decrease, diminishing returns and growing computational costs still create a budget-allocation trade-off.

Retrieval determines which evidence to select; adaptive allocation determines how much evidence to use and which model to pair it with. The two approaches can therefore complement each other.

Reviewer_1hoSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces E²Sim, an adaptive resource allocation framework for large language model (LLM)-based social survey simulations. The key idea is that instead of using a single powerful LLM and a fixed amount of respondent history for every simulation task, E²Sim dynamically selects the most appropriate combination of LLM and history budget for each specific respondent-question pair. This joint selection is designed to optimize the trade-off between prediction accuracy and computational cost. The authors identify three main issues with the conventional fixed-resource approach: Model Capability: The strongest model is not always the best, as it may introduce biases not aligned with the respondent's specific views.

Respondent History: More historical data is not always better; irrelevant history can introduce noise and increase costs.

Model-History Interaction: The optimal model choice can depend on the amount of history provided, and vice versa.

To address these, E²Sim trains a lightweight policy model to predict the accuracy-cost score for every possible combination of model and history size. It then selects the combination with the best-predicted score to perform the simulation. The framework incorporates two main techniques to improve the robustness of the policy: Respondent-History Augmentation: The policy is trained on perturbed histories (dropping and swapping responses) to make it less sensitive to the exact composition and order of historical data.

ACT-Margin Curriculum: The policy is trained progressively, starting with easy-to-decide cases (where one configuration is clearly better) and gradually moving to more ambiguous ones.

Experiments conducted on four social survey datasets (ANES, BSA, GSS, and WVS) show that E²Sim consistently outperforms baselines, including an oracle that selects the best-performing fixed configuration. The results demonstrate significant improvements in both accuracy and cost-efficiency.

Critical strengths

The problem of optimizing resource allocation in LLM-based simulations is both practical and important. The paper clearly articulates the limitations of a "one-size-fits-all" approach.

The core idea of jointly selecting the model and history budget at the request level is novel and well-motivated.

The proposed E²Sim framework, including the curriculum learning and history augmentation techniques, is technically sound and well-designed to address the core problem.

The experimental evaluation is comprehensive, covering multiple datasets, model pools, and ablation studies that effectively demonstrate the contribution of each component of the framework.

Critical weaknesses

The cost of the offline supervision phase, which requires running every model-history configuration for all training data, is substantial. While the paper provides a break-even analysis, this high initial cost could be a practical barrier to adoption, especially if the set of models or the survey data changes frequently.

The study relies on relative, parameter-based model costs. In a real-world deployment, costs are determined by API pricing and latency, which can be non-linear and subject to change. The paper acknowledges this limitation, but it raises questions about how the reported performance gains would translate to practical, real-world savings.

Questions for authors

The offline supervision process seems computationally intensive. Could you elaborate on the potential of using a smaller, sampled subset of training data or a more efficient method (e.g., distillation) to create the supervision signals, and how that might affect the policy's performance?

Your cost model is based on token counts and relative model sizes. Have you considered how E²Sim would adapt to real-world API pricing, where, for instance, a more powerful model might be significantly more expensive but also faster (lower latency), adding another dimension to the cost-benefit trade-off?

The framework operates on a predefined set of models. How sensitive is the policy to the composition of the model pool? For example, how would its performance change if the pool consisted of models with very similar capabilities and costs versus models with vastly different profiles?

4 comments
AuthorsOpenReview ↗

Response for W1 & Q1

Thank you for your suggestions. We agree that evaluating all training requests across all candidate configurations entails a substantial initial cost. We therefore conduct additional experiments on training-data subsampling and incremental adaptation, and discuss more efficient approaches to generating supervision.

1. Smaller training subsets can still support effective policies

We train on 20%, 40%, 60%, and 80% of the full training pool across four datasets, keeping the original validation sets unchanged. The full training pool contains 16,000 requests, each associated with eight model–history-budget configurations.

Training fractionANES AccuracyBSA AccuracyGSS AccuracyWVS Accuracy
20%66.55%48.45%54.50%54.93%
40%66.65%50.20%56.03%55.73%
60%67.65%50.63%55.63%56.43%
80%68.55%50.38%56.53%55.98%
100% reference68.80%50.45%55.28%56.25%

These results show that effective allocation policies can be learned without supervising the entire training pool. For example, BSA with 40% of the data and ANES with 80% achieve accuracy close to their full-data references. Using 20% of the data reduces the required training supervision calls from 128,000 to 25,600, an 80% reduction.

2. Configuration changes allow supervision reuse and incremental adaptation

When adding a model or history budget, responses for existing configurations can be reused; only the new configurations require evaluation before updating the policy. We conduct two expansion experiments:

  • Adding a model: 8B → 8B + 32B, keeping history budgets fixed at {0,1,3,5}.
  • Adding a budget: expanding history budgets from {0,1,3} to {0,1,3,5} for the 8B + 32B model pool.

During adaptation, we reuse the trained policy parameters and output weights for existing configurations, initializing only the additional outputs. The table reports Usoft=SoftAcc−0.05CU_{\mathrm{soft}}=\mathrm{SoftAcc}-0.05C; comparisons should be made within each row.

ExpansionDatasetOriginal policy20% adaptation40% adaptationFull-data training from scratch
New modelANES0.790690.792030.792700.80686
New modelBSA0.760840.763690.772030.77539
New modelGSS0.740240.736860.734750.74544
New modelWVS0.748450.752070.755770.76183
New budgetANES0.820380.815370.820150.82198
New budgetBSA0.787730.786220.788720.78570
New budgetGSS0.758840.755970.758640.75670
New budgetWVS0.758020.759460.763820.76766

With 20% of the training data, corresponding to 3,200 requests, adding the model requires evaluating four new configurations, totaling 12,800 additional training calls. Adding the budget requires evaluating two new configurations, totaling 6,400 calls. Validation costs for the new configurations are counted separately. Adaptation uses 10 epochs, whereas the full-data reference uses 20 epochs.

3. Distillation offers a direction for further reducing supervision costs

We agree that a teacher model could be trained on a small set of complete configuration evaluations and then generate approximate utility labels for other requests, reducing direct calls to candidate LLMs. However, approximate labels may alter the ranking of configurations with similar utilities and consequently affect routing quality.

Our existing LLM-as-a-Router baseline does not match the learned policy’s overall accuracy–cost trade-off, suggesting that directly relying on LLM judgments remains challenging. This comparison is not a distillation experiment. Our current experiments validate training-data subsampling and incremental adaptation; generating reliable distilled supervision from a small number of actual configuration evaluations remains a direction for future work.

AuthorsOpenReview ↗

Response for (W2 & Q2) & Q3

W2 & Q2

Thank you for your suggestion. We agree that model size does not fully reflect actual API prices or response latency. E²Sim does not require cost to be proportional to model size. Relative model size serves as a cost proxy in our original experiments; in deployment, it can be replaced with the provider’s API pricing, while latency can be incorporated as a separate objective.

1. Replacing the cost proxy with actual API pricing

For model mm and history budget kk, the API charge can be expressed as:

CAPI(m,k)=pminTm,kin+pmoutTm,kout,C_{\mathrm{API}}(m,k) = p_m^{\mathrm{in}}T_{m,k}^{\mathrm{in}} + p_m^{\mathrm{out}}T_{m,k}^{\mathrm{out}},

where pminp_m^{\mathrm{in}} and pmoutp_m^{\mathrm{out}} denote the prices per input and output token, respectively. Cached input or reasoning tokens can also be accounted for using their corresponding rates.

Thus, cost ratios between candidate models need not follow those used in our original experiments; they can instead reflect actual charges. When sufficient billing information is available in the offline records, we can recompute configuration costs and utility labels, then update the allocation policy, without regenerating all responses solely because prices have changed.

2. Incorporating latency as a separate optimization objective

A more expensive model may have lower latency, so monetary cost and latency should be measured separately. We extend the objective to:

U(m,k)=SoftAcc(m,k)−αC(m,k)−βL(m,k),U(m,k) = \mathrm{SoftAcc}(m,k) -\alpha C(m,k) -\beta L(m,k),

where CC and LL denote normalized cost and latency, respectively. The parameters α\alpha and β\beta control preferences for monetary cost and latency.

Public API latency depends on the provider, request timing, concurrent load, and cache state. To obtain more controlled measurements, we conduct an additional self-hosted experiment on WVS. Using the same 16,000 training requests and 4,000 validation requests as in router training and evaluation, we measure all combinations of the 8B and 32B models with four history budgets. Each request–configuration combination is measured once, with prefix caching enabled.

Keeping α=0.05\alpha=0.05 fixed and varying β\beta yields:

β\betaAccuracyNormalized CostMean response time of selected configurations
057.23%0.2896173.49 ms
0.0556.15%0.2745133.36 ms
0.1055.93%0.2324111.74 ms
0.2054.63%0.205293.40 ms

As β\beta increases, the policy selects configurations with lower average response times, yielding different accuracy–cost trade-offs.

These results demonstrate that E²Sim can incorporate monetary cost and latency as separate dimensions of resource allocation, supporting the feasibility of extending the framework to latency-aware routing.

Q3

Thank you for raising this question. We agree that the composition of the model pool affects policy performance and its gains over fixed configurations. Beyond differences in average capability or cost, an important factor is whether candidate models offer complementary strengths across respondent–question requests.

1. Similar capabilities and costs: gains depend on request-level complementarity

If candidate models have similar average performance and make highly consistent predictions on the same requests, the additional benefit of model selection may be limited. However, similar average accuracy does not imply that models are interchangeable: different models may perform better on different questions or respondents, leaving room for request-level routing. Moreover, even when differences between models are limited, E²Sim can still optimize resource allocation by adjusting history budgets.

2. Different model profiles can provide richer trade-offs

When models differ in capability and cost, the policy can select inexpensive models for some requests and models with greater predictive advantages for others. However, larger differences do not guarantee greater routing gains: if one configuration consistently achieves the highest utility across all requests, the preferred policy may approach a fixed selection.

3. Experimental results with different model pools

We compare three model pools, each using history budgets 0,1,3,5\\{0,1,3,5\\}. The following table reports validation accuracy with seed=42 and α=0.05\alpha=0.05:

Model poolANESBSAGSSWVS
Qwen3-8B65.80%46.98%53.20%53.93%
Qwen3-8B + Qwen3-32B68.80%50.45%55.28%56.25%
Qwen3-8B + Qwen3-32B + DeepSeek-V4-Flash70.53%52.63%57.18%58.43%

In these experiments, expanding the model pool improves prediction accuracy across all four datasets, showing that the framework can exploit the predictive capabilities offered by additional candidate models.

AuthorsOpenReview ↗

Supplementary Empirical Results: α-Conditioned Routing, Inference Break-Even & Unseen Generalization (˶ᵔ ᵕ ᵔ˶)

Dear Reviewer 1hoS,

Thank you again for your positive assessment and insightful feedback on E²Sim! As the discussion period nears its close, we have further conducted and compiled three additional sets of experiments on pure inference-cost amortization, dynamic cost-preference routing without retraining, and generalization to unseen respondents and questions.

(1) Inference-Cost-Only Break-Even Analysis

In our original paper, break-even analysis accounted for ACT utility gains. Measuring pure inference-cost savings alone relative to the best fixed configuration (BF)—reusing BF evaluations on the training requests—shows that offline supervision pays for itself well within standard large-scale survey workloads (e.g., 25M respondent–question pairs in WVS and 3.5M in ANES):

  • Break-even deployment requests (full supervision): 93,023 on WVS, 180,149 on BSA, 190,964 on ANES, and 716,158 on GSS (and up to 5× fewer when combined with our 20% training-subset subsampling).

(2) Single α-Conditioned Policy for Dynamic Cost/Pricing Preferences

Because real-world budgets and API pricing shift over time, we trained a single α-conditioned router (encoding α directly as an input across α ∈ {0.01, 0.05, 0.10, 0.20, 0.30, 0.40, 0.50}) and evaluated zero-retraining interpolation at unseen α values (Accuracy % / Normalized Cost):

Unseen α (Zero Retraining)ANESBSAGSSWVS
α = 0.0369.10 / 0.375250.80 / 0.336754.23 / 0.282655.95 / 0.2948
α = 0.07568.85 / 0.324950.40 / 0.313153.63 / 0.250255.25 / 0.2425
α = 0.1567.70 / 0.279449.93 / 0.258053.30 / 0.222855.05 / 0.2288
α = 0.2567.65 / 0.267249.70 / 0.247853.38 / 0.214855.08 / 0.2145

Practitioners can adjust accuracy–cost preferences smoothly at deployment via a single router without retraining.

(3) Generalization to Unseen Respondents & Unseen Questions + Multi-Run Stability

To verify practical robustness beyond random pair splits, we evaluated E²Sim under strict respondent-disjoint and question-disjoint holdouts against the oracle best-fixed configuration (BF):

Split SettingMetric (BF → E²Sim)ANESBSAGSSWVS
Unseen RespondentsAccuracy64.88% → 69.36%47.88% → 51.28%52.67% → 54.84%54.71% → 56.55%
Unseen RespondentsACT Gain over BF+0.0557+0.0440+0.0258+0.0146
Unseen QuestionsAccuracy69.13% → 68.24%49.25% → 51.59%52.05% → 56.27%54.33% → 57.08%
Unseen QuestionsACT Gain over BF+0.0035+0.0295+0.0350+0.0430

E²Sim achieves positive ACT gains across all 8 unseen-entity evaluations (and reduces cost from 0.5739 to 0.3252 even on ANES unseen questions). Furthermore, across 3 independent generation rounds (1,000 requests per dataset), E²Sim's exact-match gains over BF remain highly stable (+3.40 ± 0.00 pp on GSS; +2.03 ± 0.06 pp on WVS), and E²Sim outperforms direct ModernBERT answer fine-tuning across all four surveys by +1.45 to +17.08 pp.


LLM-based social simulation is still in an early, formative stage, where foundational questions—such as how model capability interacts with personal history and how large-scale synthetic surveys can be deployed cost-effectively—are only beginning to be explored. We hope our empirical findings and allocation framework offer a practical, meaningful step forward. We would be very grateful to hear whether our responses and supplementary experiments have fully addressed your questions; your continued support and championing of our work during the discussion phase would mean a great deal to both our team and the broader social simulation community!

Thank you once again for your time and thoughtful review.

Best regards,
The Authors

AuthorsOpenReview ↗

A Brief Follow-Up Before the Rebuttal Period Closes ☺

Dear Reviewer 1hoS,

Thank you again for your careful review. My co-authors and I have made every effort to address the questions and concerns you raised. With 20 hours remaining in the rebuttal period, please let us know if anything still needs clarification. We will do our utmost to give you a full response before the deadline.

If our replies have addressed your concerns, it would mean a great deal to us if you would reconsider your assessment and score.

Best regards, The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

📌 Quick Navigation Index to Reviewer Threads & Responses

Dear Area Chair and Reviewers,

As our rebuttal discussion now spans over 25 detailed comments and tables, we provide direct jump links below so that you can easily locate each review thread and all corresponding responses:

  • 👉 Jump to Reviewer 1hoS's Thread: Point-by-point responses to W1–W2 & Q1–Q3 (20%–80% training subsampling, incremental adaptation, real API pricing & latency-aware routing over 20,000 requests, and model-pool scaling) + Supplementary Results on Inference Break-Even, α-Conditioned Routing, and Unseen Generalization.
  • 👉 Jump to Reviewer vKyA's Thread: Point-by-point responses to W1–W4 & Q1–Q2 (large-scale cost analysis, joint routing ablations, 16-setting similarity retrieval comparisons, SoftAcc & loss comparisons, 3-round stability, and direct ModernBERT fine-tuning) + Follow-up Closed-Source Commercial API Experiments (GPT-6.1-sol & DeepSeek-Flash).
  • 👉 Jump to Reviewer ZMjy's Thread: Point-by-point responses to W1–W3 & Q1–Q2 (comparisons against single-dimension & two-stage routing, incremental adaptation, α-conditioned routing, inference-only break-even, and unseen respondent/question generalization) + Follow-up Clarification on Setting α and the Configuration Space.

Thank you again for your time and thoughtful engagement!

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of Rebuttal Updates, New Experiments, and Revisions for Final Consideration

Dear Area Chair and Reviewers,

As the discussion period enters its final two days, we would like to express our sincere gratitude to the Area Chair and all reviewers (1hoS, vKyA, and ZMjy) for your thoughtful, constructive feedback and active engagement throughout the rebuttal phase. We are especially encouraged that Reviewers vKyA and ZMjy have recognized our additional efforts and raised their scores.

To facilitate your final assessment, we summarize below all major concerns addressed and the comprehensive new experiments we conducted during the rebuttal—spanning real commercial API pricing, latency-aware routing, unseen respondent/question generalization, training cost reduction, and method ablations—all of which will be incorporated into the revised manuscript:

Concern / ThemeRaised ByActions Taken & New Empirical Evidence AddedKey Quantitative Results
1. Real Commercial API Pricing & Closed-Source Models1hoS (W2, Q2); vKyA (W1, Follow-up)Evaluated E²Sim on BSA using published commercial API rates (GPT-6.1-sol, DeepSeek-V4-Flash, Qwen3-8B/32B), including a 100% pure-API pool (zero open-weight models) and steeper 25× / 50× API price gaps.Pure API Pool (Flash + GPT-6.1): Matches frontier accuracy (54.18%, +3.18 pp over Flash) while cutting API spend by 21.94%. Hybrid Pool: Matches GPT-6.1 (54.25%) with 10.34% lower cost. 25× Price Gap: Cuts API cost by 65.4% vs. Best Fixed (BF).
2. Latency-Aware Multi-Objective Routing1hoS (W2, Q2)Extended the objective to U=Acc−αCmoney−βClatencyU = \text{Acc} - \alpha C_{\text{money}} - \beta C_{\text{latency}} and measured 20,000 requests on WVS with prefix caching enabled.Increasing β\beta (0→0.200 \rightarrow 0.20) smoothly reduces mean response latency from 173.49 ms to 93.40 ms (−46.2%) alongside normalized cost (−29.1%).
3. Offline Supervision Cost & Break-Even Amortization1hoS (W1, Q1); ZMjy (W2)Tested training-subset subsampling (20%–80%) and calculated inference-cost-only break-even request counts (reusing BF training evaluations).20% Subsampling: Cuts supervision calls by 80% (128,000 → 25,600) while preserving routing effectiveness. Inference-Only Break-Even: 93K–191K requests on WVS/BSA/ANES (up to 5× fewer with 20% subsampling), well below million-pair survey scales.
4. Generalization to Unseen Respondents & QuestionsZMjy (W3)Conducted strict respondent-disjoint and question-disjoint holdout evaluations across all 4 datasets (8 new splits).Achieves positive ACT gains over oracle BF across all 8 unseen-entity settings (e.g., +4.48 pp Acc on ANES unseen respondents; +4.22 pp Acc on GSS unseen questions).
5. Dynamic Preference (α\alpha) & Incremental ExpansionZMjy (W2, Q1); 1hoS (Q3)Trained a single α\alpha-conditioned router (zero-retraining interpolation at unseen α\alpha), provided a 2-step practical α\alpha selection rule, and tested incremental adaptation for new models/budgets + 3 model pools.Unseen α∈{0.03,0.075,0.15,0.25}\alpha \in \{0.03, 0.075, 0.15, 0.25\}: Smoothly trades off accuracy and cost without retraining. Incremental Adaptation: Reuses existing supervision; adapts to new models/budgets using only 20%–40% data.
6. Joint Routing vs. Baselines & Direct Fine-TuningZMjy (W1); vKyA (W2, Q2)Benchmarked E²Sim against model-only, budget-only, two-stage routing, and direct ModernBERT answer fine-tuning across all 4 datasets.Outperforms two-stage routing across all datasets (up to +5.05 pp Acc on ANES) and beats direct ModernBERT fine-tuning by +1.45 to +17.08 pp Acc.
7. Retrieval Strategy, Ordinal Metrics & StabilityvKyA (W3, W4, Q1); ZMjy (Q2)Compared top-similarity vs. random/bottom retrieval (16 settings), evaluated SoftAcc & 3 losses (Smooth-L1, MSE, Pairwise), and ran 3-round generation stability tests.Top-Similarity Retrieval: Beats random and bottom retrieval by +11.49 pp and +12.04 pp on average. Stability: Consistent gains across 3 fresh generation rounds (+3.40 ± 0.00 pp on GSS; +2.03 ± 0.06 pp on WVS).

We have already committed to integrating all of these empirical results, ablations, and practical guidelines into the camera-ready version of our paper. Thank you once again for your time, insightful suggestions, and dedication to improving this work.

Warm regards,
The Authors of Submission 17549

AuthorsOpenReview ↗

Sincere Thanks as the Discussion Period Concludes

Dear Area Chair and Reviewers,

As the discussion period draws to a close, we would like to extend our heartfelt gratitude to all of you for dedicating your valuable time and thoughtful effort to reviewing our work.

First, we would especially like to thank Reviewer vKyA and Reviewer ZMjy for actively engaging in the discussion amidst your busy schedules, recognizing our additional efforts, and raising your scores. The follow-up points you raised further motivated us to complete our final round of targeted responses and clarifications without leaving any loose ends, and we will make sure to thoroughly incorporate all new findings and refinements from the discussion period into our revised manuscript as planned. At the same time, we completely understand that Reviewer 1hoS has a tight schedule during this busy end-of-semester and deadline season and regrettably has not had the chance to engage further in the discussion thread; nonetheless, your encouraging initial assessment and constructive questions have greatly strengthened the depth and practical value of our work.

Finally, we sincerely thank the Area Chair for carefully overseeing the entire review process—and we are especially grateful for your kind assistance earlier in reminding the reviewers to join the discussion. We respectfully ask that you kindly take our supplementary empirical results and responses from the discussion period into consideration during your final assessment.

Thank you once again for your generous service to the research community, and we wish you all the very best!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.