- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLM-driven social simulations can predict how real populations respond to interventions. The question studied here is very important, but the empirical basis is only eight events, and the below-chance accuracies are never checked against a constant predictor.
Critical strengths
The question raised is timely, the paper is easy to read, and the protocol is carefully specified: a shared information cutoff, prespecified assets and horizon, three seeds per event, and token consumption reported as a first-class metric.
Critical weaknesses
Weakness 1 One of my major concerns is that the effective sample size is four events per setting. Please report the number of independent events, the per-event Direction for each configuration, and confidence intervals with the event as the unit of analysis. At present there is no statistical inference anywhere in the paper.
Weakness 2 Another major concern is that the title and Finding 1 speak about LLM-based social simulation in general, but all the evidence comes from eight short-horizon financial events, which are close to a worst case for any simulation method. Please scope the claims to short-horizon financial-market prediction, or add a non-financial setting.
Weakness 3 Missing recent reference, including "Empowering Economic Simulation Through Situation-Aware LLM-Driven Generative System"
Questions for authors
What is the realized direction of each of the eight events? What is the within-event cross-asset sign agreement in Panel B? Per configuration, does SGBA beat the structural model that supplies its moments?
3 comments
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Thank you for the detailed response. However, my cocerns still remain after the rebuttal, and I therefoe maintain my rating.
We thank the reviewer for detailed comments on statistical inference and generalizability. We provide event-level statistics, a non-financial evaluation, and the missing reference.
W1: Insufficient Statistical Evidence
Each setting has four independent events, with 3 simulators × 3 backbones × 3 seeds (27 runs/event; 108/setting). CIs use events as independent units.
Setting-level Direction (%):
Configuration Direction (%): E1/E2/E3/E4, mean, event-level 95% CI. DS = DeepSeek-V4-Flash; GLM = GLM-5.3-Flash; GPT = GPT-4o; M = MarketSim; S = StockAgent; T = TwinMarket.
Index Constituent Adjustment
Fed Rate Change
W2: Potentially Overgeneralized Claims
We aim to improve LLM behavioral fidelity, not rank all models universally. SGBA reweights policies using population moments while freezing the LLM, complementing work on social-outcome prediction [1], response fine-tuning [3], and industrial post-training [4].
We add a non-financial evaluation using MobLab human-choice data [5], covering the Dictator and Trust games. Across 13 experimental conditions, we assess whether SGBA can reproduce human choice distributions. All LLM methods use GPT-4o; BFS is the structural social-preference model. SGBA matches BFS moments without target-condition choices.
The same policy pool improves from 20.93 to 14.98;
W3: Missing Recent References
Thank you for bringing this recent reference to our attention. We will add the suggested reference [2] to Section 2.1 ("LLM-Based Social Simulation"):
"Chen and Chen [2] introduce SAMAS, which integrates macroeconomic knowledge and recent experiences for situation-aware agent decisions. SGBA complements this approach by aligning population-level behavioral policies using structural moments."
Q1: Realized Directions, Cross-setting Transfer, and Structural Comparison
Realized signs: Index = first post-announcement trading day to effective-date close; Fed = 20-minute announcement return.
Cross-setting transfer (Table 2). We test whether SGBA relies on setting-specific structural models by applying moments from China's stock market to the U.S. Fed setting. These moments capture population-level behavioral distributions and preferences, such as demand sensitivity and substitution, which may partially map across countries. The improvement, though smaller than with Fed-specific moments, supports partial cross-country transferability. SGBA may be applied future behavioral cloning or RL under moment constraints.
Structural comparison. SGBA does not beat the structural benchmark in every configuration. For Fed, MarketSim-SGBA exceeds P-R (91.7% vs. 83.3%); StockAgent and TwinMarket do not. Its contribution is policy alignment with a frozen backbone and unchanged interaction mechanism.
References: [1] Ashokkumar et al. (2026), Nature, “Large language models can predict the results of social science experiments.” [2] Chen & Chen (2026), ICASSP, “Empowering Economic Simulation Through Situation-Aware LLM-Driven Generative System.” [3] Kolluri et al. (2025), EMNLP, “Finetuning LLMs for human behavior prediction.” [4] Latent Space (n.d.), “Simulation: The new scaling law” (Simile AI podcast). [5] Xie et al. (2025), PNAS, “Using large language models to categorize strategic situations and decipher motivations behind human behaviors.”