- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed EIPG to calibrate persona generations using the difference between the simulated behavior and the reference. The structured search summarizes the behavior of synthetic populations and adjusts the persona distribution through an optimization loop. The natural-language method modifies prompts through constrained preference adjustments. The framework was evaluated on synthetic environments and real world datasets across several different LLMs. The results show that more intervention coverage improves counterfactual prediction. However, calibration can also lead to worse performance on unseen interventions.
Critical strengths
- The paper attempts to address an important question in the domain of simulating economic behaviors using LLMs. The paper provides a meaningful failure mode where calibration can lead to worse counterfactual predictions.
- The paper provides insightful reasons and supporting experiments to study why calibration can make counterfactual predictions worse.
Critical weaknesses
- My primary concern is that it is unclear from the paper how the M target is constructed. Including more details of this part can make the paper easier to read.
- The experiments need more ablation and comparative studies to show the effectiveness of the framework. For example, why is the MNL needed in the middle of structured method? And some previous work [1, 2] also attempts to construct personas or use customer preference data for outcome calibration.
[1] Bohan Zhang, Jiaxuan Li, Ali Hortacsu, Xiaoyang Ye, Victor Chernozhukov, Anqi Ni, and Edward W Huang. 2026. Agentic Economic Modeling. [2] Mengxin Wang, DennisJZhang, andHengZhang.2026. Large language models for market research: A dataaugmentation approach. Marketing Science
Questions for authors
- Regarding the first weakness, how is the target matrix constructed?
- The MNL model fitted directly to human data substantially outperforms EIPG in one of the experiments. While I understand the proposed framework aims to construct reusable synthetic populations, what additional practical benefit does the calibrated population provide over the directly fitted MNL?
2 comments
I thank the authors for clarifying the construction of M and look forward to seeing these additions in the revised paper. I will maintain my weak accept recommendation.
We thank the reviewer for the constructive feedback. We agree that the construction of Mtar needs to be explained more clearly, as its source differs across experiments. In the controlled benchmark, reference moments are computed from choices generated by a predefined synthetic population, including choice shares, chosen-attribute means, and responses to interventions. In the natural-language experiment, we instead calculate the targets from a fixed synthetic reference model. We specify MNL preference coefficients for three segments (budget-sensitive, quality-oriented, and sustainability-oriented) and use them to calculate choice probabilities over predefined product contexts.
The resulting target vector contains 19 moments. Fifteen are segment-specific chosen-attribute means: for each segment and attribute, we multiply each product's attribute value by its reference choice probability and average across contexts. The remaining four capture intervention responses. For each intervention, we modify one attribute of a product while holding the others fixed, calculate the change in its choice probability, and average that change across reference segments and paired contexts.
For example, the budget-sensitive segment has a target mean chosen price of 1.4960. When we increase the intervened product's price by 25%, the aggregate target change in its choice probability is −0.0988. These values are fixed before calibration. The persona optimizer receives the discrepancies between its simulated moments and these targets, but not the underlying reference coefficients or individual reference probabilities.
We will add the full target vector, reference coefficients, and intervention specifications to the appendix. These experiments test calibration against known synthetic behavior, not human behavioral fidelity. In Nutri2Cycle, by contrast, the reference statistics are calculated from observed human choices.
Regarding the rol of MNL, we consider MNL also plays a specific, rather than essential, role in the framework. In the structured implementation, we fit MNL to choices generated by each candidate population and compare its implied moments with the reference. It provides a common, interpretable summary of candidate behavior,, but also introduces a modeling approximation. The natural-language implementation computes moments directly from LLM choices, without an intermediate MNL. We will clarify this distinction in the method section.
the comparison with direct-human MNL is particularly important. In the submitted Nutri2Cycle experiment, direct MNL predicts held-out choices substantially better than structured EIPG (share L2 of 0.0758 versus 0.4491). we do not claim that producing synthetic personas is inherently preferable to estimating a choice model directly. The potential advantage we want to investigate is whether pretrained behavioral knowledge can reduce the human data needed to calibrate a useful population. That motivated two new experiments:
We therefore run a new experiment and evaluated both approaches across human-label budgets. At 48 respondents (144 labeled choices), the calibrated LLM-persona mixture achieved held-out L2 of 0.12, compared with 0.18 for regularized direct MNL. The advantage was concentrated at smaller budgets; direct MNL became more accurate as additional observations were available. This would identify a specific data-scarcity use case for pretrained personas, rather than general predictive superiority. A convincing comparison also requires selecting MNL regularization using training data alone, which we would document. The result would not establish lower total computational cost or prove that pretraining itself causes the advantage.
We also calibrated a persona population on Nutri2Cycle Tasks 1–2, froze its descriptions and mixture weights, and evaluated it on Tasks 3–4 without further calibration or human labels. Its mean choice-share L2 was 0.08, compared with 0.11 for MNL trained on the same source observations, and it better reproduced the change in choices between the two price scenarios. MNL can also transfer without refitting, so the claim is limited to better predictive transfer in these particular within-domain scenarios. Task 4 had previously been examined in our submitted evaluation, making this an exploratory result rather than an independent confirmation.
Finally, regarding related work, we agree that using personas and human evidence to model economic behavior is not new. Zhang et al. also investigate synthetic agents for economic modeling, while Wang et al. develop a data-augmentation approach for statistical inference. Our focus is on calibrating the persona-generating mechanism against explicit behavioral discrepancies, including intervention responses. this produces an explicit population of agents rather than only corrected predictions or parameter estimates. We will clarify both the overlap and this methodological distinction.