- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PoliticalSim, an LLM-based multi-agent framework for simulating political processes, including electoral voting, parliamentary deliberation, and interactions with bureaucracies and interest groups. The framework incorporates mechanisms from political science and allows controlled experiments by enabling or disabling individual factors. The authors evaluate four LLMs using hypothetical elections, real-world election settings across seven countries, and parliamentary simulations. The results demonstrate some consistency with established political science theories and explore how different political mechanisms affect simulated outcomes.
Critical strengths
-
Comprehensive political simulation framework. The paper integrates electoral voting, parliamentary deliberation, and institutional interactions into a unified LLM-agent framework, extending beyond prior studies that focus on individual political processes.
-
Mechanism-level analysis. The framework supports controlled ablation experiments to examine the influence of political factors such as party discipline, constituency interests, and interest groups, providing a useful approach for studying simulated political dynamics.
Critical weaknesses
-
Political behaviors may be directly induced by the prompts rather than emerging naturally. Many expected behaviors are explicitly specified in the agent instructions. For example, voters are encouraged to avoid wasted votes, while legislators are instructed to prioritize constituency interests or follow party discipline. Consequently, the observed agreement with political science theories may primarily reflect instruction following rather than genuine political behavior simulation. The existing prompt ablations do not fully resolve this concern.
-
Insufficient real-world validation and lack of meaningful baselines. The real-world election results are relatively weak, with the best-performing model correctly identifying the largest party in only two of seven countries. More importantly, the paper lacks comparisons with simple baselines based on historical election results or traditional statistical models, despite providing agents with previous election outcomes and habitual party preferences. It is therefore unclear whether the proposed framework offers meaningful predictive value beyond these inputs.
Questions for authors
-
How can the authors distinguish political behaviors that genuinely emerge from LLM agents from those directly induced by the prompts? Would the main findings still hold without explicit instructions encouraging strategic voting, constituency prioritization, and party discipline?
-
Given that agents already receive historical election results and habitual party preferences, how much predictive value does PoliticalSim provide beyond simple historical or statistical baselines? Can the authors provide such comparisons to substantiate the framework's effectiveness?
5 comments
W2:
Insufficient real-world validation and lack of meaningful baselines. The real-world election results are relatively weak, with the best-performing model correctly identifying the largest party in only two of seven countries. More importantly, the paper lacks comparisons with simple baselines based on historical election results or traditional statistical models, despite providing agents with previous election outcomes and habitual party preferences. It is therefore unclear whether the proposed framework offers meaningful predictive value beyond these inputs.
Lack of directly comparable baselines:
Previous political simulation frameworks have used approaches such as directly supplying election related social media posts to LLMs [5] or providing detailed candidate profiles constructed from legislators’ Wikipedia articles[6]. These approaches have problems, as the concern raised in Reviewer sfns’s Weakness 1: “Consequently, the observed agreement with political science theories may primarily reflect instruction following rather than genuine political behavior simulation.”
To address this concern, we propose a novel framework grounded in voters’ economic circumstances generated through economic simulations and constituency profiles constructed from national statistics, including religious composition, industrial structure, and urban/rural classifications, that are free from detailed SNS and Wikipedia texts.
To the best of our knowledge, no existing framework provides a directly comparable baseline under the same information constraints. Furthermore, our framework simulates seven countries with substantially different electoral systems, and we are unaware of an existing statistical modeling framework that covers all seven within a comparable experimental setting.
Predictive performance:
Our evaluation considers 4 political science metrics, rather than only whether the model correctly identifies the largest party. Performance should therefore be assessed across all 4 dimensions.
For the best performing model, the rank correlation of party vote totals exceeds 0.66 in more than half of the countries. The simulations also achieve absolute errors within 1.0 on political science measures such as the Gallagher index and the effective number of parties in several national settings.
These results are achieved with limited real-world grounding information and without detailed textual inputs from social media [5] or Wikipedia[6]. We therefore emphasize the framework’s predictive performance under these information constraints.
[5] Maryam Davoodi, Dan Goldwasser, Analysis of State-Level Legislative Process in Enhanced Linguistic and Nationwide Network Contexts, NAACL2024. [6] Hao Li, Ruoyuan Gong, Hao Jiang, Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models, AAAI2025.
Q1:
How can the authors distinguish political behaviors that genuinely emerge from LLM agents from those directly induced by the prompts? Would the main findings still hold without explicit instructions encouraging strategic voting, constituency prioritization, and party discipline?
First, we emphasize that we do not claim that political behavior “emerges” from LLM agents.
Second, all experiments include ablations in which prompt conditions are enabled or disabled, allowing us to assess their influence on the simulated outcomes.
Q2:
Given that agents already receive historical election results and habitual party preferences, how much predictive value does PoliticalSim provide beyond simple historical or statistical baselines? Can the authors provide such comparisons to substantiate the framework's effectiveness?
PoliticalSim offers three main contributions: (1) real-world grounding without additional model training or detailed agent profiles; (2) a benchmark for evaluating LLMs’ ability to understand political settings and emulate political behavior; and (3) a platform for investigating political theories while accounting for interactions among agents, particularly where controlled real-world experiments are difficult or infeasible.
(1) Real-world grounding with limited information:
PoliticalSim simulates human political behavior using voters’ economic circumstances generated through economic simulations and limited statistical information, such as religious composition and party support distributions. It does not require additional model training or detailed agent profiles constructed from political related texts written by human such as SNS or Wikipedia. The concern raised in Reviewer sfns’s W1 applies to prior LLM agent simulation approaches for political science that grounding real-world using social media posts [5] or detailed candidate profiles derived from Wikipedia [6]. Our framework reduces reliance on such detailed textual inputs.
(2) Evaluating political understanding and behavioral simulation:
Despite substantial advances in LLM capabilities, their reasoning ability of complex political, legal, and economic settings remains insufficient. PoliticalSim requires models to act not only as voters and legislators but also as bureaucrats and interest groups, enabling evaluation across diverse political roles. For example, Figures 8 and 10 in the Appendix indicate that, when assigned bureaucratic or interest-group roles, Llama-3.1-8B-Instruct makes decisions that are less consistent with maximizing the assigned actor’s interests than those of the other models. Such results illustrate the framework’s potential as a LLM benchmark for assessing political reasoning and role-specific decision-making.
(3) Controlled investigation of political theories:
Comparing competing theories under otherwise identical conditions is often difficult in social science, and some interventions cannot feasibly be conducted in real societies. PoliticalSim provides a platform for examining the implications of theory derived mechanisms under controlled simulated conditions while accounting for interactions among actors. These experiments can support hypothesis development and identify predictions for subsequent empirical testing, although simulation results alone do not establish a theory’s validity in the real world.
[5] Maryam Davoodi, Dan Goldwasser, Analysis of State-Level Legislative Process in Enhanced Linguistic and Nationwide Network Contexts, NAACL2024. [6] Hao Li, Ruoyuan Gong, Hao Jiang, Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models, AAAI2025.
We thank the reviewer for taking time to consider our paper.
First, we clarify that this paper does not claim that political behavior spontaneously emerges from individual LLM agents’ decision making.
The paper addresses two research questions. First, does the ability of LLM agents to emulate human behavior and interactions extend to political simulations? (Abstract, lines 18–22; Introduction, lines 55–58). Second, when simulations are grounded in the real world by directly supplying social media posts or Wikipedia articles, can we distinguish the model’s simulation capabilities from its tendency to reproduce information supplied in the prompt? (Introduction, lines 50–58; Appendix, Figure 11, where “#Ground” denotes methods for real-world grounding).
Regarding the first question, previous research on LLM agent simulations has demonstrated substantial capabilities in emulating human behavior and interactions in domains such as economics [1,2]. However, it is not self-evident that these capabilities extend to the behavior of voters and politicians. One contribution of our study is to investigate this question in the political domain and benchmark LLM models’ ability to simulate political science.
Regarding the second question, simulations of voter opinions that use social media posts as input may produce outputs strongly affected by those input prompts. Such experiments alone do not establish the principle that LLMs can reliably simulate unseen political scenarios for which comparable grounding texts are unavailable. We therefore propose a novel framework that conducts political simulations without detailed textual inputs from social media or Wikipedia. Instead, it grounds the simulations in statistical information as a reference about each country, including religious composition, constituency-level industrial structure, and urban/rural classifications, together with voters’ economic circumstances (Introduction, line 70).
[1] Gati Aher, Rosa I. Arriaga, Adam Tauman Kalai, Using large language models to simulate multiple humans and replicate human subject studies, ICML2023. [2] John J. Horton, Apostolos Filippas, Benjamin S. Manning, Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?, EC '24: Proceedings of the 25th ACM Conference on Economics and Computation.
W1:
First, we emphasize that this paper does not claim that political behavior “emerges spontaneously” in LLM agent simulations. Therefore, the criticism concerning emergence does not apply to the claim we make.
Second, the concern that “the observed agreement with political science theories may primarily reflect instruction following rather than genuine political behavior simulation” was already considered in our experimental design and addressed through ablation studies. All conditions specified in the prompts can be individually enabled or disabled, and results with the relevant conditions disabled are reported in Round 1 of Table 1 and in Table 4.
Third, specifying agents’ behavioral principles through prompts and studying the collective outcomes arising from their interactions is common practice in prior work on LLM agent based economic simulations [3,4] and does not compromise the performance or reliability of the simulation framework.
[3] Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao, EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities, ACL2024. [4] Yuzhe Yang, Yifei Zhang, Minghao Wu, Kaidi Zhang, Yunmiao Zhang, Honghai Yu, Yan Hu, Benyou Wang, TwinMarket: A Scalable Behavioral and Social Simulation for Financial Markets, NeurIPS2025.