acceptodds
Under review as a conference paper at ICLR 2027

How Well Do LLM-Based Simulations Predict? Lessons from Structural Models

Abstract

Recent advances in large language models (LLMs) have driven rapid progress in social simulation, enabling heterogeneous agents to sustain complex interactions and reproduce a range of emergent phenomena. Yet it remains unclear whether such simulations can reliably predict how real populations respond to interventions. We examine the real-world predictive validity of LLM-based social simulations by comparing with structural models, which derive behavioral mechanisms from economic theory and estimate key parameters from observed data. Across the evaluated settings, LLM simulators consume substantial inference resources yet consistently underperform structural models in directional prediction. Scaling the simulated population does not close this gap. Analyzing agent response distributions, we find that larger deviations from structural response distributions are consistently associated with lower directional accuracy. Motivated by this finding, we propose Structure-Guided Behavioral Alignment (SGBA), which uses structural moments estimated from observable market data as behavioral supervision to calibrate distributions over executable agent policies while keeping LLM parameters fixed. SGBA improves directional accuracy by an average of 26.6 percentage points across the two evaluation settings and moves simulated response distributions closer to their structural references. Ablations show successive gains from explicit policy representation and structural distribution optimization. These findings highlight behavioral distribution alignment as promising principles for improving the empirical reliability of LLM-based social simulation. Code is available at https://anonymous.4open.science/r/SGBA-2027.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based social simulation, Structural models, Predictive validity, Behavioral alignment, Distribution alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PALwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLM-driven social simulations can predict how real populations respond to interventions. The question studied here is very important, but the empirical basis is only eight events, and the below-chance accuracies are never checked against a constant predictor.

Critical strengths

The question raised is timely, the paper is easy to read, and the protocol is carefully specified: a shared information cutoff, prespecified assets and horizon, three seeds per event, and token consumption reported as a first-class metric.

Critical weaknesses

Weakness 1 One of my major concerns is that the effective sample size is four events per setting. Please report the number of independent events, the per-event Direction for each configuration, and confidence intervals with the event as the unit of analysis. At present there is no statistical inference anywhere in the paper.

Weakness 2 Another major concern is that the title and Finding 1 speak about LLM-based social simulation in general, but all the evidence comes from eight short-horizon financial events, which are close to a worst case for any simulation method. Please scope the claims to short-horizon financial-market prediction, or add a non-financial setting.

Weakness 3 Missing recent reference, including "Empowering Economic Simulation Through Situation-Aware LLM-Driven Generative System"

Questions for authors

What is the realized direction of each of the eight events? What is the within-event cross-asset sign agreement in Panel B? Per configuration, does SGBA beat the structural model that supplies its moments?

3 comments

We thank the reviewer for detailed comments on statistical inference and generalizability. We provide event-level statistics, a non-financial evaluation, and the missing reference.

W1: Insufficient Statistical Evidence

Each setting has four independent events, with 3 simulators × 3 backbones × 3 seeds (27 runs/event; 108/setting). CIs use events as independent units.

Setting-level Direction (%):

SettingE1 / E2 / E3 / E4Mean95% CI
Index35.56 / 48.61 / 36.11 / 48.8942.29[30.42, 54.16]
Fed65.24 / 31.00 / 2.16 / 2.2225.15[-22.54, 72.85]

Configuration Direction (%): E1/E2/E3/E4, mean, event-level 95% CI. DS = DeepSeek-V4-Flash; GLM = GLM-5.3-Flash; GPT = GPT-4o; M = MarketSim; S = StockAgent; T = TwinMarket.

Index Constituent Adjustment

SimLLME1/E2/E3/E4Mean95% CI
MDS20/50/37.5/5039.4[16.78,61.97]
MGLM30/50/37.5/7046.9[19.05,74.70]
MGPT40/75/37.5/5050.6[23.38,77.87]
SDS40/37.5/25/4035.6[24.20,47.05]
SGLM40/50/12.5/6040.6[8.08,73.17]
SGPT20/25/25/017.5[-1.44,36.44]
TDS40/62.5/37.5/7052.5[26.72,78.28]
TGLM50/37.5/75/5053.1[28.10,78.15]
TGPT40/50/37.5/5044.4[33.91,54.84]

Fed Rate Change

SimLLME1/E2/E3/E4Mean95% CI
MDS83.33/25/8.33/029.2[-30.63,88.96]
MGLM91.67/41.67/0/033.3[-35.99,102.66]
MGPT90.91/16.67/0/026.9[-42.16,95.94]
SDS50/33.33/11.11/2028.6[1.67,55.56]
SGLM25/16.67/0/010.4[-9.47,30.31]
SGPT75/44.44/0/029.9[-28.49,88.21]
TDS66.67/42.86/0/027.4[-25.25,80.01]
TGLM54.55/25/0/019.9[-21.39,61.16]
TGPT50/33.33/0/020.8[-18.95,60.61]

W2: Potentially Overgeneralized Claims

We aim to improve LLM behavioral fidelity, not rank all models universally. SGBA reweights policies using population moments while freezing the LLM, complementing work on social-outcome prediction [1], response fine-tuning [3], and industrial post-training [4].

We add a non-financial evaluation using MobLab human-choice data [5], covering the Dictator and Trust games. Across 13 experimental conditions, we assess whether SGBA can reproduce human choice distributions. All LLM methods use GPT-4o; BFS is the structural social-preference model. SGBA matches BFS moments without target-condition choices.

MethodNormalized Wasserstein distance ↓
Default GPT-4o24.96
Equal-weight LLM policy pool20.93
Adapted DA17.05
BFS15.75
SGBA14.98

The same policy pool improves from 20.93 to 14.98;

W3: Missing Recent References

Thank you for bringing this recent reference to our attention. We will add the suggested reference [2] to Section 2.1 ("LLM-Based Social Simulation"):

"Chen and Chen [2] introduce SAMAS, which integrates macroeconomic knowledge and recent experiences for situation-aware agent decisions. SGBA complements this approach by aligning population-level behavioral policies using structural moments."

Q1: Realized Directions, Cross-setting Transfer, and Structural Comparison

Realized signs: Index = first post-announcement trading day to effective-date close; Fed = 20-minute announcement return.

SettingEvent (date)PositiveNegative
IndexE1 (2024-11-29)4/106/10
IndexE2 (2025-05-30)5/83/8
IndexE3 (2025-11-28)3/85/8
IndexE4 (2026-05-29)7/103/10
FedE1 (2024-09-18)11/121/12
FedE2 (2024-11-07)5/127/12
FedE3 (2024-12-18)0/1212/12
FedE4 (2025-01-29)0/1212/12

Cross-setting transfer (Table 2). We test whether SGBA relies on setting-specific structural models by applying moments from China's stock market to the U.S. Fed setting. These moments capture population-level behavioral distributions and preferences, such as demand sensitivity and substitution, which may partially map across countries. The improvement, though smaller than with Fed-specific moments, supports partial cross-country transferability. SGBA may be applied future behavioral cloning or RL under moment constraints.

Structural comparison. SGBA does not beat the structural benchmark in every configuration. For Fed, MarketSim-SGBA exceeds P-R (91.7% vs. 83.3%); StockAgent and TwinMarket do not. Its contribution is policy alignment with a frozen backbone and unchanged interaction mechanism.

References: [1] Ashokkumar et al. (2026), Nature, “Large language models can predict the results of social science experiments.” [2] Chen & Chen (2026), ICASSP, “Empowering Economic Simulation Through Situation-Aware LLM-Driven Generative System.” [3] Kolluri et al. (2025), EMNLP, “Finetuning LLMs for human behavior prediction.” [4] Latent Space (n.d.), “Simulation: The new scaling law” (Simile AI podcast). [5] Xie et al. (2025), PNAS, “Using large language models to categorize strategic situations and decipher motivations behind human behaviors.”

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Thank you for the detailed response. However, my cocerns still remain after the rebuttal, and I therefoe maintain my rating.

Reviewer_1tqROpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first examines if LLM simulation matches with structural models on real-world predictive validity. The paper then proposes Structure-Guided Behavioral Alignment (SGBA), using observable market data as behavioral supervision to calibrate distributions over executable agent policies while keeping LLM parameters fixed. SGBA improves directional accuracy by an average of 26.6 percentage points across the two evaluation settings.

Critical strengths
  • Grounding LLM simulation in established economics. The paper compares three recent LLM market simulators (MarketSim, StockAgent, TwinMarket) against two well-known structural models (Koijen-Yogo demand system and Pflueger-Rinaldi monetary model) and shows the superiority of structural models in terms of predictive power and token usage. It sends a positive message to the research community.
  • Formal analysis of why population scaling cannot eliminate systematic response error. One popular explanation for LLM social simulation discrepancy is the difference in scaling. However, the paper presents a formal analysis against this explanation. In Section 4.2, the paper presents a bias-variance decomposition, separating sampling variance from systematic response bias.
Critical weaknesses
  • The comparison with structural models is not exactly fair. The structural economics model were specifically developed to predict these types of market outcome, e.g. index-inclusion demand shocks and monetary-policy surprises, from dedicated holdings and macro data. It is unfair to compare them with the three LLM-based market simulations mentioned here, which are designed for easy manipulation and observation rather forecasting.
  • SGBA's gains are more or less expected. The paper shows that, applying SGBA, which is to seed the initial policy distribution and state with structural model guided data, improves its predictive alignment. The ablation (Fig. 8, Table A4) compares only the original simulator and unaligned policies, so it cannot show that LLM-generated policies add value. This limits the claim that SGBA improves LLM-based simulation specifically, and a non-LLM baseline seeded with the same structural information would resolve it.
Questions for authors
  • Have you evaluated a non-LLM numerical agent model, such as rule-based agents or the parametric policies, initialized and aligned with the same structural moments? If it achieves similar directional accuracy, my concern in Weakness 2 would remain. If it performs clearly worse, that would largely address it.
3 comments

We thank the reviewer for constructive feedback on comparison fairness and LLM added value. We respond with economics-informed prompting and a structurally aligned non-LLM baseline.

W1: Potential Unfair Comparison

Structural models are developed for specific financial predictions using dedicated economic data, whereas LLM simulators offer flexible agents and scenarios. We acknowledge this asymmetry. Our aim is not to establish that general-purpose LLM simulators universally outperform structural models, but to assess their predictive fidelity and how it can be improved. Predicting human behavior is an active research and industrial objective [1,2,4,6].

Without task-specific calibration, LLM agents may not represent target-population behavior accurately. SGBA addresses this by generating executable candidate policies and reweighting them to match structural response moments estimated from observable market data. It requires no private individual-level decisions, leaves the LLM backbone frozen, and preserves the simulator's interaction and price-formation mechanisms. This complements human-data post-training [5].

We add Economics-Informed Prompting (EIP). EIP supplies agents with economic mechanisms, few-shot examples, and concise summaries of structural moments and parameters; unlike SGBA, it does not optimize policy weights. This contrasts prompt-based economic guidance with population-policy alignment. On four Fed Rate Change events, we obtain the following results (Direction / Directional Value [DV], %, higher is better):

SimulatorOriginal LLMEIPSGBAStructural (P-R)
MarketSim26.9 / -48.047.9 / 2.991.7 / 94.383.3 / 66.3
StockAgent29.9 / -37.726.1 / -36.841.7 / -0.683.3 / 66.3
TwinMarket20.8 / -50.247.2 / -1.966.7 / 26.983.3 / 66.3

EIP event-level outcomes (Direction / DV, %):

SimulatorE01E02E03E04
MarketSim83.3 / 84.133.3 / -31.10.0 / -100.075.0 / 58.8
StockAgent60.0 / 51.022.2 / -47.222.2 / -50.80.0 / -100.0
TwinMarket77.8 / 75.950.0 / 18.944.4 / -23.316.7 / -79.0

EIP improves MarketSim and TwinMarket, but SGBA has higher Direction and DV than both Original and EIP in all three simulators. Thus, providing economic information in prompts does not reproduce the observed benefits of policy-distribution alignment.

W2: Unproven LLM Added Value

We add a non-LLM multi-asset rule-based ABM [3]. Agents follow mean-reversion, trend-following, or mixed portfolio rules; a market maker updates prices using aggregate excess demand. On four Index Constituent Adjustment events, the unaligned and aligned ABMs use the same candidate rules, initial holdings/cash, and price formation. Only policy weights change: equal weights versus SGBA's minimum-KL projection onto two K-Y demand-response moments for net additions and deletions. Both ABM variants preserve the rule set and price formation.

MethodDirection (%)DV (%)MAE (pp)RMSE (pp)
Rule-based ABM52.08-0.924.81675.8984
Rule-based ABM + SGBA52.08-0.984.81765.8983

Direction is unchanged by event:

Announcement dateStocksBefore → After (%)
2024-11-291050.00 → 50.00
2025-05-30850.00 → 50.00
2025-11-28875.00 → 75.00
2026-05-291033.33 → 33.33

The alignment matches the structural moments but does not improve the rule-based ABM's Direction. One possible reason is the limited expressiveness of preset trading rules. LLM-generated policies can combine event interpretation, state-dependent belief updates, and trading plans; reweighting them may alter several linked decisions. This may explain the larger gains observed with LLM-generated policies.

Q1: Non-LLM numerical-agent baseline

The aligned rule-based baseline and event-level results are reported in W2. Under this controlled comparison, matching structural moments does not improve the rule-based ABM's average Direction.

References: [1] Aher et al. (2023), ICML, “Using large language models to simulate multiple humans and replicate human subject studies.” [2] Ashokkumar et al. (2026), Nature, “Large language models can predict the results of social science experiments.” [3] Chiarella, Dieci & He (2007), Journal of Economic Behavior & Organization, “Heterogeneous expectations and speculative behavior in a dynamic multi-asset framework.” [4] Kolluri et al. (2025), EMNLP, “Finetuning LLMs for human behavior prediction in social science experiments.” [5] Latent Space (n.d.), “Simulation: The new scaling law—Joon Sung Park, Simile AI” (podcast). [6] Simile (n.d.), “Simulating human behavior” (website).

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Thank you for the detailed response and the additional experiments, in particular the Economics-Informed Prompting (EIP) results and the rule-based ABM baseline. I appreciate the effort. I will keep my score. As I noted in my review, my confidence is low and I do not have sufficient expertise to assess this submission reliably, so please weigh my comments accordingly.

W1 (fairness of the comparison). Thank you for acknowledging this point. Since the structural models are purpose-built for these outcomes, I would encourage softening the framing of Finding 1 (“effective and efficient alternatives”) and the token-cost comparison, presenting the structural models as a domain-specialist reference.

W2 / Q1 (added value of LLM-generated policies). Thank you for running the non-LLM baseline. The unaligned rule-based ABM already reaches 52.1% Direction on the Index events, while the SGBA-aligned LLM simulators average roughly 54.7% on the same setting. Therefore, a non-LLM model is comparably accurate without LLM cost, so the experiment does not yet show that LLM-generated policies add predictive value.

Thank you again for the thoughtful engagement. I defer to the other reviewers and the AC on the technical merits.

Reviewer_ifJKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the accuracy of LLM-based simulations. By evaluating the predictive validity of LLM-based social simulation in two financial market scenarios, it finds that LLM simulators consistently underperform structural models in directional prediction. To remedy this issue, it proposes SGBA, which uses structural moments estimated from observable market data to calibrate distributions over executable agent policies for LLM simulators. Experiments demonstrate the effectiveness of SGBA in improving directional accuracy.

Overall, the findings and the proposed solution is interesting. On the other hand, there are several major flaws in the paper, which are stated in the Critical Weaknesses section below.

Critical strengths

S1. The question is timely and well posed. It is an important finding that LLM-based simulation is worse than structural models in directional accuracy.

S2. SGBA only calibrates executable agent policies, with LLM parameters frozen.

S3. The experiments are extensive, with multiple LLMs evaluated.

Critical weaknesses

W1. The evaluation of SGBA is circular. This is because SGBA constrains the simulated population to reproduce the structural model's moments, and it is scored against a benchmark that the structural model already wins. The improvement is expected by construction. For real-world applications (e.g., when human subjects' data are unavailable or too expensive), we are unsure whether structural models would perform well.

W2. Baseline approaches are missing in evaluating SGBA. The paper cites Mixture of Personas (Bui et al., 2025) and Distributional Alignment for Social Simulation (Xie et al., 2026), but they are not empirically compared with. In addition, an immediate way of calibrating an LLM agent with a structural model is to put the structural model's information in the prompt, e.g., telling agents the structural model's predicted direction/magnitude, describing its economic mechanism, or showing few-shot examples.

W3. The scope of the paper needs to be constrained, because the evaluation only considers financial markets. It is unknown whether the observation also applies to other scenarios, e.g., simulating social behavior such as in an SNS. The authors may want to specify financial market simulations as the scope in the title, abstract, and conclusions.

Questions for authors

Q1. The two scenarios favor structural models, i.e., structural models are about 73.8% and 83.3% directionally correct, respectively. What if LLM simulators are evaluated for scenarios where structural models are not so accurate?

7 comments

We sincerely thank the reviewer for the thoughtful assessment and constructive suggestions. We clarify SGBA's use of structural moments, add baseline comparisons, and assess cross-setting transfer and non-financial generalizability.

W1: Potential Circularity in Evaluation

SGBA aligns agent-policy distributions with behavioral moments (e.g., price sensitivity, substitution responses), not structural predictions. The moments are estimated from historical data available before each target event and fixed before simulation; neither realized returns nor target-event directional labels are used to estimate them or calibrate policy weights. Thus, evaluation outcomes do not enter the alignment process. Final predictions emerge from simulator interactions rather than being imposed by the structural model. In the reported Fed Event 2 example, the structural model attains 41.7% Direction, while SGBA (MarketSim) reaches 91.7%, illustrating that the two accuracies need not coincide.

Structural moments capture population preferences and may transfer. In Table 2, using moments from the Chinese stock market to align policies in the U.S. Fed setting also improves results. Because the behavioal moments between china and us could mapped in some extent. SGBA don't need private account data, and it only use market data.

W2: Missing Baselines

Existing alignment methods. MoP relies on behavioral exemplars, while Distributional Alignment (DA) fits observed response distributions. Direct application to our financial setting would require investor-level behavioral observations unavailable in our data. MoP's repository provides neither runnable code nor data; DA's released resources do not provide a directly runnable implementation for our setting. We therefore adapted code and MobLab human-choice data released in a separate PNAS study by the DA authors.

We evaluate the Dictator game ($10 allocation) and the Banker role in the trust game (returning part of a tripled transfer). We predict choice distributions using GPT-4o and the BFS social-preference model. SGBA uses BFS moments without target-condition choice labels.

MethodWasserstein distance ↓
Default GPT-4o24.96
BFS (no LLM policies)15.75
Adapted DA (seven source roles; no target refitting)17.05
Equal-weight SGBA policy pool20.93
SGBA (BFS moments; no target-condition labels)14.98

Economics-Informed Prompting (EIP). EIP puts economic mechanisms, few-shot examples, and summaries of structural moments/parameters directly into prompts, without optimizing population policy weights. We test it on four Fed events. Entries are Direction / Directional Value (DV), in % (higher is better):

SimulatorOriginalEIPSGBA
MarketSim26.9 / -48.047.9 / 2.991.7 / 94.3
StockAgent29.9 / -37.726.1 / -36.841.7 / -0.6
TwinMarket20.8 / -50.247.2 / -1.966.7 / 26.9

EIP helps MarketSim and TwinMarket, but SGBA performs better on both metrics for all three simulators. Prompting with structural information alone does not reproduce SGBA's gains in these tests.

W3: Limited Scenario Generalizability

The MobLab evaluation above extends beyond financial markets: with the same LLM policy pool, SGBA reduces Wasserstein distance from 20.93 to 14.98 without target-condition human-choice labels. This offers initial evidence that moment-based policy alignment can improve non-financial social-behavior simulation.

Q1: What if the structural models are less accurate?

Three points address this concern. First, SGBA uses behavioral moments rather than structural forecasts, so its accuracy need not track the structural model's accuracy (see the Fed example in W1). Second, Table 2 shows that moments derived from the Chinese stock market can also improve alignment in the U.S. Fed setting. Third, SGBA targets a different aspect of social-simulation fidelity: aligning agent behavior while retaining the existing simulator and frozen LLM backbone. Related work predicts social-science outcomes [1], fine-tunes LLMs on individual responses [2], or post-trains behavioral foundation models with human data [3]. SGBA instead reweights LLM-generated policies using population-level moments. In future work, rollouts from its KL-constrained policy mixture could also supply demonstrations for behavioral cloning or initialize RL under moment constraints.

References: [1] Ashokkumar et al. (2026), Nature, “Large language models can predict the results of social science experiments.” [2] Kolluri et al. (2025), EMNLP, “Finetuning LLMs for human behavior prediction in social science experiments.” [3] Latent Space (n.d.), “Simulation: The new scaling law—Joon Sung Park, Simile AI” (podcast).

Dear Reviewer,\

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.\

Thank you for your time and effort in reviewing our paper.

Reviewer_ifJKOpenReview ↗

Re: Author Rebuttal

Thanks for the rebuttal. It partially addresses W2, but leaves W1, W3, and Q1 largely open.

W1

The authors argue that SGBA aligns to behavioral moments rather than structural predictions. My point is that the moments encode the structural model's mechanism, so SGBA inherits its directional implications. Whether the targets are called "moments" or "predictions" doesn't change that.

I suppose that 91.7% is MarketSim's four-event average in Table 2, not an Fed Event 2 result. Do you have a per-event table of structural model accuracy vs. SGBA accuracy?

W2

The rebuttal explains why MoP and DA don't fit the financial setting, and the authors run DA on MobLab games instead.

SGBA (14.98) barely beats the structural model (BFS with no LLM, 15.75). Most of SGBA's gain over the equal-weight pool (20.93) brings it to roughly where BFS already is. That reinforces W1 rather than answering it.

W3

The MobLab dictator and trust games are still economic games. So the scope of this paper needs to be narrowed in the title, abstract, and conclusions.

Q1

First point: This repeats the response to W1. Second point: The China -> US transfer is about cross-setting moments, not about structural models being inaccurate. Third point: This positions SGBA within related work, which does not answer the question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.