- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of jointly choosing an LLM and the history context length for simulating responses to surveys. It defines a scalarized Accuracy-Cost Trade-off (ACT) objective , and proposes a supervised learning approach that uses existing surveys with real human responses to train a predictor that predicts the ACT score for each model-history pair . For a new survey simulation request, it chooses a model-history pair with the highest predicted ACT score. Experiments are conducted on four social-survey datasets, with three LLMs and four history budgets.
Critical strengths
The experiments show that across the four datasets, the proposed approach consistently improves the simulation accuracy and cost compared with the best fixed model-history configuration in hindsight.
Critical weaknesses
My main concern is the limited methodological and practical contributions. The proposed approach of scalarizing accruacy and cost, supervised learning of ACT scores, and choosing a model-history pair to maximize the predicted ACT is quite straightfoward. While jointly choosing the model and history length sounds reasonable, the paper does not demonstrate its advantages and benefits over existing approaches for adaptive inference, such as LLM routing.
Moreover, the proposed method has limited flexibility. First, it requires fixing in advance a set of LLMs and history budgets. As acknowledged in Appendix F, "Introducing a new model or history budget requires evaluating the added configurations and updating the allocation policy". This also seems to be the case for the parameter in ACT. The paper does not offer guidance on how one should choose in practice, either. Second, the offline supervision stage requires generating responses from all LLM-history pairs for all surveys in the training set, which can incur a high computational cost, especially when there is a large number of LLMs and history budgets. While the paper gives an offline-supervision amortization cost analysis, it is based on the ACT score, instead of the inference cost alone.
Another concern is that the paper does not demonstrate whether the learned allocation policy generalizes to new survey questions and respondents, which is an important consideration and motivation in LLM-based survey simulation. In the experiments, the training and testing sets are randomly splitted over question-respondent pairs, instead of holding out new questions or respondents. Thus, it is very likely that the survey questions and respondents in the testing set already appear in the training set.
Questions for authors
- In practice, how should one choose in the ACT?
- How does the proposed approach account for the randomness in the LLM-simulated responses? How sensitive are its results (e.g., the learned allocation policy and the improvement in accuracy and cost) to the randomness?
7 comments
Response for W2 & Q1 (Part 1/2)
Thank you for your comments. We agree that the original manuscript did not sufficiently discuss configuration updates, the choice of , and offline supervision costs. We address these concerns through additional experiments on incremental adaptation, -conditioned routing, training-data requirements, and inference-cost break-even analysis.
1. New models and history budgets can be incorporated through incremental adaptation
Although the policy operates over a predefined configuration space, adding configurations does not require regenerating responses for existing configurations or retraining the policy from scratch. We evaluate two types of expansion:
- Adding a model: expanding from 8B to 8B + 32B, with history budgets .
- Adding a history budget: expanding the budgets from to for the 8B + 32B model pool.
During adaptation, we reuse the trained policy parameters and output weights for existing configurations, initialize only the additional outputs, and fine-tune using different amounts of supervision. The following table reports the accuracy–cost utility ; comparisons should be made within each row.
| Expansion | Dataset | Original policy | 20% adaptation | 40% adaptation | Full-data training from scratch |
|---|---|---|---|---|---|
| New model | ANES | 0.79069 | 0.79203 | 0.79270 | 0.80686 |
| New model | BSA | 0.76084 | 0.76369 | 0.77203 | 0.77539 |
| New model | GSS | 0.74024 | 0.73686 | 0.73475 | 0.74544 |
| New model | WVS | 0.74845 | 0.75207 | 0.75577 | 0.76183 |
| New budget | ANES | 0.82038 | 0.81537 | 0.82015 | 0.82198 |
| New budget | BSA | 0.78773 | 0.78622 | 0.78872 | 0.78570 |
| New budget | GSS | 0.75884 | 0.75597 | 0.75864 | 0.75670 |
| New budget | WVS | 0.75802 | 0.75946 | 0.76382 | 0.76766 |
These results demonstrate the feasibility of incremental adaptation. With 20% of the training data, corresponding to 3,200 requests, adding the model requires evaluating only the four new 32B configurations, totaling 12,800 additional training calls. Adding the budget requires evaluating only two new configurations, totaling 6,400 calls. Existing supervision is reused, and validation costs for the new configurations are counted separately. Adaptation uses 10 epochs, whereas the full-data reference uses 20 epochs.
2. Conditioning on reduces the need for retraining when preferences change
To address whether changing requires retraining, we introduce an -conditioned policy. We encode in the policy input and train a single router using utility supervision under multiple values, allowing configuration selection to respond to different cost preferences.
We train with and evaluate interpolation at unseen values. All entries below are measured validation results without smoothing. Each cell reports Accuracy (%) / Normalized Cost.
values used during training:
| ANES | BSA | GSS | WVS | |
|---|---|---|---|---|
| 0.01 | 67.80 / 0.3186 | 51.45 / 0.4321 | 54.50 / 0.4082 | 55.75 / 0.3242 |
| 0.05 | 69.13 / 0.3915 | 51.38 / 0.3783 | 54.15 / 0.2946 | 55.98 / 0.2965 |
| 0.10 | 68.53 / 0.3120 | 51.30 / 0.3318 | 53.95 / 0.2699 | 55.75 / 0.2728 |
| 0.20 | 67.90 / 0.2736 | 50.20 / 0.2784 | 53.70 / 0.2313 | 55.33 / 0.2413 |
| 0.30 | 67.38 / 0.2599 | 49.80 / 0.2521 | 53.15 / 0.2167 | 54.88 / 0.2180 |
| 0.40 | 67.35 / 0.2505 | 49.18 / 0.2360 | 53.08 / 0.2084 | 54.48 / 0.2031 |
| 0.50 | 67.33 / 0.2408 | 48.40 / 0.2249 | 52.88 / 0.2018 | 53.85 / 0.1920 |
Selected unseen values for interpolation:
| ANES | BSA | GSS | WVS | |
|---|---|---|---|---|
| 0.03 | 69.10 / 0.3752 | 50.80 / 0.3367 | 54.23 / 0.2826 | 55.95 / 0.2948 |
| 0.075 | 68.85 / 0.3249 | 50.40 / 0.3131 | 53.63 / 0.2502 | 55.25 / 0.2425 |
| 0.15 | 67.70 / 0.2794 | 49.93 / 0.2580 | 53.30 / 0.2228 | 55.05 / 0.2288 |
| 0.25 | 67.65 / 0.2672 | 49.70 / 0.2478 | 53.38 / 0.2148 | 55.08 / 0.2145 |
Response for W2 & Q1 (Part 2/2)
At the training values, the policy generally selects less expensive configurations as the cost penalty increases, showing that separate policies are not required for each . The displayed unseen values also produce different accuracy–cost trade-offs, although this does not establish reliable interpolation for arbitrary .
In practice, we recommend choosing according to deployment requirements:
- Given a cost budget: Evaluate a grid of values on the validation set, record the corresponding accuracy and average inference cost, and select the achieving the highest accuracy within the budget.
- Given a minimum accuracy requirement: Select the yielding the lowest inference cost among settings that satisfy the accuracy requirement.
- Without an explicit constraint: Examine the validation accuracy–cost curve and select a point on its Pareto frontier according to application preferences. A knee point can serve as a starting point, where further cost reductions begin to require larger accuracy sacrifices.
For the objective , specifies how much accuracy the user is willing to exchange for a unit reduction in normalized cost. For example, when is expressed on a 0–1 scale, means that a cost reduction of 0.10 offsets an accuracy decrease of 0.005, or 0.5 percentage points, in the objective.
Because the learned policy may not respond monotonically to , the final setting should be selected using its measured validation accuracy and cost, rather than assuming that increasing always satisfies a tighter budget. Once selected, is supplied to the conditioned router at deployment without additional policy training.
3. Effective routing does not require supervision over the entire training pool
We conduct additional training using 20%, 40%, 60%, and 80% of the full training pool while keeping the original validation set unchanged. The full training pool contains 16,000 requests.
| Training fraction | ANES Accuracy | BSA Accuracy | GSS Accuracy | WVS Accuracy |
|---|---|---|---|---|
| 20% | 66.55% | 48.45% | 54.50% | 54.93% |
| 40% | 66.65% | 50.20% | 56.03% | 55.73% |
| 60% | 67.65% | 50.63% | 55.63% | 56.43% |
| 80% | 68.55% | 50.38% | 56.53% | 55.98% |
| 100% reference | 68.80% | 50.45% | 55.28% | 56.25% |
Smaller supervision subsets already support effective allocation policies. For example, BSA with 40% of the data and ANES with 80% achieve accuracy close to their full-data references.
4. Additional break-even analysis based solely on inference costs
We agree that amortizing ACT gains does not replace an analysis of inference-cost savings alone. We therefore calculate how many deployment requests are needed for savings relative to the best fixed configuration (BF) to cover the additional offline supervision cost.
Under the assumption that BF responses for the same training requests would already be generated and can be reused, the additional supervision cost is:
The corresponding break-even request count is:
Using full-data supervision and the corresponding policies, we obtain:
| Dataset | Inference-cost break-even requests |
|---|---|
| ANES | 190,964 |
| BSA | 180,149 |
| GSS | 716,158 |
| WVS | 93,023 |
These additional results show that, although offline supervision entails an initial investment, this investment can be reduced through fewer supervised requests, reuse of existing configuration evaluations, and incremental policy updates.
Response for W1
Thank you for raising this concern. We agree that scalarizing accuracy and cost and regressing configuration utilities are not, by themselves, new learning techniques. Our contribution lies in the empirical motivation, joint model–history allocation, and task-specific training design for survey simulation.
1. Empirical Findings: Stronger Models and More History Do Not Necessarily Improve Simulation
To the best of our knowledge, we are the first to systematically identify and quantify three phenomena in social survey simulation: stronger general-purpose models are not necessarily more accurate, more history is not necessarily more useful, and the preferred model and history budget are interdependent.
The first two findings motivate request-level model selection and history allocation, while the third supports joint decision-making. These findings challenge the practice of uniformly using a strong model with extensive history and constitute an empirical contribution of this work.
2. Joint Optimization and Comparisons with Routing Baselines
Utility regression is the implementation mechanism; joint configuration selection is the methodological focus. Models differ in their ability to extract useful information from respondent history, and changes in historical evidence can alter which model is preferable. We therefore learn the accuracy–cost utility of each model–history configuration and select jointly over the complete configuration space.
E²Sim selects both the model and the amount of historical information. To evaluate this design, we compared model-only routing, history-budget-only routing, and a two-stage procedure that selects the model before selecting the history budget. The following accuracies use the same validation requests and seed=42:
Joint selection achieves higher accuracy in all these settings. Its SoftAcc-based utility gains over two-stage selection are also positive on all four datasets. These results support jointly accounting for model–history interactions rather than relying on a single allocation dimension or sequential selection.
3. Task-Specific Design: Handling History Variation and Allocation Ambiguity
History augmentation models variation in the evidence available at deployment. For example, when predicting attitudes toward public healthcare spending, training respondent A may have answered questions about healthcare, taxation, and government responsibility, whereas test respondent B may lack the taxation response. Small retrieval-score perturbations may also change the ordering of historical responses. Random dropping and swapping model these two variations, reducing dependence on a fixed history composition and order.
Curriculum learning addresses differences in allocation difficulty across requests. For healthcare policy, one respondent may express consistent historical attitudes, while another may support better services but oppose higher taxes. We use the ACT margin between the best and second-best configurations to measure allocation ambiguity, learning clearly distinguishable cases before introducing near-tied cases. These social examples illustrate the motivation; the actual curriculum is determined by observed configuration performance.
Together, these findings and comparisons support the practical value of jointly selecting models and history budgets, beyond extending the action space alone.