acceptodds
Under review as a conference paper at ICLR 2027

ElectionBench: Evaluating LLM Agents through Competitive Elections

Abstract

Evaluating LLM agents requires analyzing their behavior in dynamic environments: how they accumulate information, take actions, and pursue objectives over long-horizon interactions. Yet existing benchmarks rely on static environments or single-answer tasks that lose discriminative power as model performance saturates. To address these limitations, we introduce ElectionBench, a novel competitive benchmark where LLMs campaign head-to-head to persuade a shared synthetic electorate in procedurally generated elections. Operating under incomplete information and costly action trade-offs, agents must infer various hidden information such as voter preferences, and publish persuasive articles. Across 11 environment profiles, we benchmark 15 LLM families using mirrored matches and Bradley-Terry scaling anchored to a scripted baseline. Experimental results indicate that winning models (led by Claude-Sonnet-5 and Gemma-4-31B-it) succeed through strategic restraint, taking fewer actions and preserving capital. Furthermore, closing debates overturn 24% of election outcomes, highlighting the decisive interplay between dynamic persuasion and long-horizon resource management. Together, ElectionBench provides a rigorous testbed for evaluating how autonomous agents integrate active information gathering, economic decision-making, and natural-language communication under direct competition.

Primary area:
datasets and benchmarks
Keywords:
LLM Simulation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_i7hoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The problem itself—evaluating agents in competitive settings, and in particular in choice-based settings—is an important one. However, I struggle to identify any meaningful contribution in the submitted paper, certainly not one that would justify publication at ICLR.

The experimental setup is weak, the intellectual depth of the paper is low, the writing appears to have been produced by AI, and it is evident that little serious human thought was invested in the work.

I opened the paper’s code, which, like a large fraction of academic code written in 2026, appears to have been generated to a substantial extent using generative AI. That, in itself, is perfectly acceptable. However, the authors explicitly state that “we have no using generative AI tools... for develope evaluation.” This is not a minor mistake or an imprecise statement—it is a blatant falsehood. There is no place for such misrepresentation in academic work.

For this reason, I recommend rejecting the paper outright.

Critical strengths

None.

Critical weaknesses

See Summary.

Questions for authors

None.

1 comment

We believe the basis of rejection should contain weaknesses and lapses in the method or novelty of the paper, not assumptions that the authors didn't put work into writing and designing the benchmark. This is the standard reviewer procedure in ICLR.

As for AI policy, it is an industry and academia norm since 2025 to use coding agents for implementation. Our statement "We have not used generative AI tools to ... develop evaluation code..." is for analyzing the results (both scripts for ranking and manually going through chat logs to observe behaviors), NOT for implementation.

As for the contribution of our work, we cite several papers that evaluate models under competitive setting and clearly highlight how our benchmark is different from existing works (Table 1).

We leave the decision to the Area Chair to judge whether the paper writing, results, and the review are up to ICLR standards or not.

Reviewer_haYFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ElectionBench, a competitive benchmark in which two LLM candidates campaign head-to-head over a shared synthetic electorate in procedurally generated elections. Candidates operate under incomplete information and costly actions: they survey hidden voter preferences, publish articles, raise funds, and conclude with a final debate. Outcomes are judged by a pool of persona-conditioned LLM voters. The authors evaluate 15 LLMs across 11 environment profiles and aggregate results with a Bradley-Terry rating anchored to a scripted baseline. They report that winners tend to take fewer actions and that the final debate frequently changes the outcome.

Critical strengths
  1. The benchmark mechanics are fully specified: complete prompts, favorability formulas, and environment configurations are given in the appendix, and the code is released, which supports reproducibility. The authors also recognize the importance of evaluator reliability and attempt to examine it.
  2. The chosen domain and motivation are novel to some extent. Evaluating LLM capabilities through long-horizon election campaigns is a topic worth studying.
Critical weaknesses
  1. The evaluator validation is insufficient and partly undermines the benchmark’s premise.

    • Key details of Section 5.1 are missing. These include how voters are assigned to LLM backbones, the composition of the persona pool, and the response preferences of each voter model.
    • The cross-persona experiment shows that most voter models return identical reactions regardless of persona. This suggests that persona conditioning has little effect on voter judgments. If so, persuasion reduces to choosing the right topic plus generic text quality, and the benchmark’s central claim of persuading a heterogeneous electorate is substantially weakened.
    • The voter-removal experiment lacks a null baseline, such as randomly removing the same fraction of voters. It also cannot detect biases shared across all voter models, so it does not support the claim that the voter pool is robust.
    • The within-family bias effect is reported without a definition of how it is estimated.
  2. The human annotation study is under-specified and does not validate the key property. The paper does not explain how the persona-article pairs were sampled, which voter model the annotations are compared against, or how agreement is computed. More importantly, overall agreement on article reactions cannot validate persona-conditioned judgment. A meaningful test would check whether human judgments of the same article vary across personas, and whether the voter models follow that variation.

  3. Figure 2 does not support the paper’s behavioral claim that “winning candidates take fewer campaign actions”. Also, the x-axis averages action counts over environments with very different horizons and budgets, which inflate apparent “restraint”. The linear fit is reported without any statistical support.

  4. The results are poorly organized, and the conclusions are unclear. The 11 environments are a core part of the design, yet per-model, per-environment results are barely analyzed. They are not connected back to the capabilities each environment is claimed to test in Table 2. Section 5.3 mixes validation of the benchmark mechanics (seat disadvantage, debate effect) with analysis of model behavior. Overall, the results read as a list of observations rather than a clear account of which capability differences the benchmark reveals.

  5. Emergent behaviors and benchmark flaws are not distinguished. The collusion behaviors in the case studies may be largely induced by the design as fund_rival and meet_rival as tools already suggests their use. These behaviors therefore point to misalignment between the benchmark and real elections as much as to emergent strategy. The case study in Section 5.5 is framed positively, even though the winning model was willing to sell states. Similarly, the fabrication behavior described in Appendix A.1 may be directly rewarded by the ranking, because candidate claims are never fact-checked.

  6. Table 4 lacks uncertainty estimates. The paper describes bootstrap resampling, but no confidence intervals are reported. The top-ranked models are separated by very small margins, so the claimed ranking may not be statistically meaningful. As a minor point, grouping entries by model family and placing FC/ReAct configurations side by side would make the table easier to read.

Questions for authors
  1. What are the details of the matchups? Which opponents did each model face, how many games were played per entry, and what were the raw win rates? How are matchups with a perfect sweep handled in the Bradley-Terry fit, given that the maximum-likelihood estimate diverges in that case? Do the top-ranked models play each other directly, and are their ratings distinguishable?
  2. Why are all voter models so small? Is there a risk that strong candidate models systematically exploit weak LLM voters, for example through length or rhetorical style? Beyond the article annotations, has the debate, which determines a substantial share of outcomes, been validated against human judgments? What are the per-model response distributions and preferences of the voters?
4 comments

We thank the reviewer for finding our work valuable and for their constructive feedback. Below, we address each point and specify the changes we made to the manuscript.

Weaknesses

W1: We address each concern below:

  • Each voter is assigned one of the 12 backbones uniformly at random when the world is generated and keeps it for both mirrored games, so each backbone plays about 1/12 of the electorate. Personas are sampled uniformly from a pool of 20k curated from Nemotron-Personas-USA. Each voter also has demographic attributes, an engagement value, a fixed seat preference, and three ranked issue priorities that combine personal beliefs with state-level issue weights.
  • We largely agree. The cross-persona test used one event and does not show persona sensitivity. The electorate’s heterogeneity is carried by ranked issue priorities, seat lean and engagement, which differ across voters and states and which the simulator applies to each label (weights 1, 0.6, 0.3 or 0 by priority rank). Voters also differ by backbone, which varies the strength of their reactions, although candidates cannot observe or target this.
  • As a null, we re-scored the logged games after dropping a random 10% of voters in each state, slightly more than the 8.3% removed with one backbone. The winner changes in 1.9% of games, against 1.6-4.2% when a specific backbone is removed. The ranking remains unchanged.
  • We measured same-family preference using voter reactions to campaign articles across all games. For each voter model, we regressed its reaction scores on the mean reaction of other models to the same article. This adjusts for differences in baseline generosity and sensitivity to article content. We then compared that model’s residual scores for same-family candidates versus their opponents, using games with exactly one same-family candidate. A positive difference indicates same-family preference. These differences are the values reported in §5.1: Qwen +0.02, Granite +0.02, Llama −0.01, and −0.48 for Gemma-3-4B on Gemma-4 candidates.

W2: We sampled 100 persona-article pairs uniformly at random from the logs. Four were dropped due to illegible characters, leaving 96. Each pair is compared against the label of the voter model that judged it in the game, so backbones appear roughly in proportion to their share of voters. Each pair yields two labels, one per candidate, and agreement is exact match on the five-point label over these 192 labels. Annotators saw the same persona card, article and answer options as the voter model and worked independently. Exact agreement with the three annotators is 75.0%, 70.8% and 86.5% (mean 77.4%). All three agree on 68.2% of labels.


W3: Figure 2. It plots ratings against mean actions across configurations. It is not evidence for the within-game statement. The within-game statement comes from a paired comparison of the winner and loser of each game. The winner buys an extra action on 32% of days and the loser on 40%. We have removed “strategic restraint” and “preserving capital” from the abstract and conclusion: unspent budget has no terminal value, we have not controlled for game state, and trailing candidates spending to catch up is a plausible explanation for the gap.

Pooling across Environments. Every configuration plays the same number of games in each profile, so the pooled mean weights profiles identically for all configurations and does not favour any of them. We agree the axis is still dominated by long-horizon profiles: marathon supplies 30 of the 107 campaign days in a set.

Linear fit. We fitted an ordinary least-squares regression of Elo on average executed campaign actions per game across the 23 model-interface entries, giving each entry equal weight and excluding the scripted baseline. The association is positive and moderate: r = 0.592 (95% CI 0.24 to 0.81, p ≈ 0.003), R² = 0.35, so action count alone explains ~35% of the variation in Elo. The 23 entries are not fully independent, since eight models appear under both interfaces. This is a descriptive association and does not establish that taking more actions causes better performance.


W4: During the rebuttal period, we have updated the manuscript to better organize results. We have also added per-environment results in Appendix G and refer to it in §5.

W5: The rival channel and the transfer action are created by design, included to observe whether agents take actions that do not serve the stated objective. In a zero-sum contest a transfer to the rival can only hurt the sender or serve a side deal. We never instruct the model to use it, or what to say. Their presence alone does not produce the behavior. 20 of 23 configurations never transfer funds and 14 never open the rival channel, while use is concentrated in lower-rated models.

Case study in §5.5 states the interaction between Kimi-K2.7 and Gemma-4-31B. We preface this case study by mentioning that it is unethical and is similar to bribing and colluding in the real world. We do not frame it positively.

We agree this is a flaw in what the rating rewards. Candidates have no record in the fictional world, and neither voters nor donors can check a claim, so invented evidence carries no cost and may help. App. A.1 is retitled to describe the fabrication, and fact-checking is listed as the main missing mechanic in the limitations.


W6: We added 95% confidence intervals in Table 4 for all 23 entries. The top entries are Claude-Sonnet-5 1946 (1889-2073), Gemma-4-31B-it (FC) 1938 (1867-2041) and GPT-6-Luna 1932 (1848-2035). These intervals overlap almost entirely, so the ordering within this group is not statistically meaningful. We no longer claim a uniquely best model in the Abstract and §5. Ratings are estimated jointly from 82 complete matchups (3,608 games).

We have also added a new table grouping by model family, with each model’s FC and ReAct configurations side by side.


Questions

Q1: Matchups. The evaluation comprises 82 complete matchups (3,608 games, 44 per matchup). New Appendix F lists, for every entry, each opponent it faced. Because the ladder pairs entries of similar strength, raw overall win rates are not comparable across entries, which is why we report them per pairing.

Perfect sweeps. 6 of the 82 matchups ended 44-0. A sweep in one matchup does not by itself make the maximum-likelihood estimate diverge. The estimate is finite as long as no group of entries is unbeaten against the rest, which holds in our data.

Top-ranked entries. The top four met each other directly in six pairings. Their ratings are not distinguishable. The 95% intervals are Claude-Sonnet-5 1946 (1889-2073), Gemma-4-31B-it (FC) 1938 (1867-2041) and GPT-6-Luna 1932 (1848-2035), and we do not claim an order among them.


Q2: Why small voters. Our choice to use smaller LLMs comes from resource constraints. We simulate up to 250 voters each in up to 45 states. Voters with preference plus some spillover are called on every event done by the candidate. Simulating such a huge environment on every matchup is computationally infeasible. We have stated this limitation clearly in the updated manuscript for future readers (§3.3).

Exploitation through length or style. We agree this is a risk, and checked what the logs allow. Articles are capped at 200 words, which bounds length effects. Candidate labels are swapped at random on every call, which controls for label position. The 100 randomly sampled articles in the human review didn’t showcase any sign of exploitation, however we have not tested rhetorical style beyond this.

Human validation of the debate. During the rebuttal period, we have had the three annotators label 20 debate transcripts under the same protocol. Exact agreement with the voter models is 45%, and agreement on which candidate gained more is 65%.

Per-model response distributions. New Table 7 reports, for each of the 12 voter models over all campaign calls, the share of each label given to the publishing candidate, the mean reaction to the publishing candidate and to the rival, and the effect of the display label (A or B). Three patterns stand out. First, generosity differs. For eleven models the mean reaction to the publishing candidate ranges from +0.92 (Granite-4.1-3B, GLM-4-9B) to +1.79 (Yi-1.5-9B). Most concentrate on “better” (89% for Granite-4.1-3B, 85% for Qwen3-8B), while Yi-1.5-9B, EXAONE-3.5 and InternLM3 mostly answer “much better” (85%, 70%, 64%). The 80% ceiling in App. C.2 applies to the qualification batch, which mixes positive and negative controls and both candidates’ labels, not to campaign calls. Second, models differ in how far an article lowers the rival, from about zero (Qwen3-4B −0.02, Granite-3.3-8B −0.05) to −0.91 (Qwen3-1.7B). Third, Gemma-3-4B is an outlier: it rates the publishing candidate “worse” or “much worse” in 49.5% of calls (mean −0.26), and its rating shifts by 1.10 levels with the display label. Granite-3.3-8B (0.51) and the two smaller Qwen models (0.28 to 0.29) also show label effects; the other eight are within 0.21. Because labels are swapped at random on every call, these effects do not favour either candidate, but they add noise. Removing Gemma-3-4B voters changes the winner in 4.2% of games, against 1.9% for a random 10% of voters, and leaves the top-four point-estimate order unchanged.

Reviewer_haYFOpenReview ↗

Response to author's rebuttal

Thanks authors for the detailed response. I went through the response and some of my concerns are resolved. However, explanations on human annotation and rival channel design still remain concerning. The authors also admit the flaws in persona design, which is an important part of validating the robustness of voter models. Overall, I would keep my score.

Reviewer_j6JWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ElectionBench, where 2 LLM candidates run a multi-day campaign in a generated fictional country and try to win a majority of states. Each day they spend budget on surveys, articles, a one-time manifesto, donor meetings and private messages or money to the rival, and after the last day they debate for 12 turns each. Voters are personas played by 12 small LLMs that only give a 5-level label (much worse to much better) after reading an article or the debate, and a fixed simulator turns these labels into favorability and winner-take-all state results. 15 LLMs (23 FC/ReAct entries) play 44-game matchups over 11 environment profiles, opponents are picked with a ladder, and ratings come from Bradley-Terry on the Elo scale with a scripted baseline at 1000. Claude-Sonnet-5 (1946), Gemma-4-31B (1938), GPT-6-Luna (1932) and Gemini-3.8-Flash (1928) are at the top... paper reports that winners take fewer actions and keep more budget, the debate changes the winner in 24% of games, and case studies show bribes and collusion b/w candidates. They also report voter checks (77.4% agreement with 3 annotators) and a small human study.

Critical strengths

1- Good careful competitive setup... every world is played twice with seats swapped so each model plays the hard seat once in underdog and outspent, candidate labels are shuffled in each voter call, and odd number of states and voters avoid ties. The 11 profiles each test (scale, scarcity, unequal money, short vs long campaign, polarized voters) and all prompts and scoring rules are written out in App. B and C. Since difficulty depends on the opponent, this kind of setup can keep separating models after static tasks saturate.

2- The case studies are good. It shows agent behaviors that static benchmarks cant show... a 6-day cash-for-states negotiation with threats b/w Kimi-K2.7-Code and Gemma-4-31B, GPT-OSS-120B and Granite using collusion language in 18 of 44 debates, and Claude-Sonnet-5 inventing past projects, workers for a donor while reasoning "the donor is a bot". These are useful for agent safety research.

Critical weaknesses

1- Its not shown that the rating measures strategy (getting information, using money, adapting) and not just what 12 small LLMs like to read + the simulator constants. Every result comes from 5-level labels by 1.7B-9B models judging text written by frontier models. From the App. C formulas, one "much better" article on a voter's first priority adds +35 favorability, while the best possible manifesto adds only about +11 and the whole underdog tilt is 10.. so the voter label decides most of the outcome. Sec 5.1 says 9/12 voter models give the same label for all 50 personas, so the persona text does nothing, which is known flattening problem of LLM simulated voters. Money also matters very little... a 4x budget gap only reduce the poor seat from 49.4% to 37.8%, and in the large profile LLMs beat a fixed script only 56.3% of the time. The debate reaches every voter and comes last, while a day-1 article keeps about a tenth of its weight by vote time (0.75^8), so "debate changes 24% of outcomes" (and less in marathon, which also uses slower decay) mostly follows from the update rule. Khan et al. (ICML 2024) found that weaker judges give a smaller range of debater Elo, so the near tie at the top may come from the voter pool and not the agents. There is no ablation (no debate, manifesto + debate only, free surveys, stronger voters) to show what is actually measured.

2- The ranking and the behavior claims have no statistical support. Sec 3.3 says bootstrap is used but Table 4 has no intervals. The top 4 are within 18 Elo (about 52.6% expected win rate) and with 44 games per matchup the standard error of a win rate is at least 7.5%, so "led by Claude-Sonnet-5 and Gemma-4-31B-it" in the abstract is not supported, same for the FC vs ReAct gaps of 4 to 48%. The total number of matchups and games and who played whom in the ladder are never reported. "82.7% of the games produce a dominant winner" is not a reliability check... a fair coin already gives 3-1 or better in 62.5% of 4-game groups. The abstract also says winners succeed through "strategic restraint" and "preserving capital", but this is a correlation with no control, and leftover budget is worth nothing at the end so it cant cause a win.. more likely the side that is ahead stops spending and the one behind spends to catch up. Sec 5.3 says higher rated models use more actions overall. So this claim is misleading.. Please clarify.

Questions for authors

1- Can you give bootstrap 95% CIs for all 23 entries, the total no. of matchups and games, and who played whom in the ladder? Are the top 4 separable at all?

2- Can you re-score the saved campaigns with a voter pool that has no overlap with candidate families (Gemma-3-4B rates Gemma-4 candidates 0.48 levels lower) and with larger voter models (30B-70B)? What is the Kendall's tau with Table 4, and does the gap at the top increase?

3- Where do simple policies land on the Elo scale... manifesto + debate only, the same games without debate, the scripted strategy with LLM-written articles, and free full survey results? This would show how much of the rating is strategy and how much is text.

4- Sec 3.2 says the debate has "no word limit" but App. B.6 shows a word-limit message, which one is correct? How do 24-turn transcripts fit the 4K context of Yi-1.5-9B-Chat, and what is the win rate of the opening speaker (the same model opens in both mirrored games)? Have you analyzed the raw model inputs/outputs?

5- What are the temperature, thinking settings, max tokens and timeout rates per model, and which LLM plays the donors? The 400 Elo gap for OLMo3 FC vs ReAct comes from token budget, and Moonlight-16B-A3B has only 8K context, so settings can change ratings a lot. These are all really important details in an LLM/agent paper like these. So, these should be there.

4 comments

We thank the reviewer for the careful read and constructive suggestions. Below, we address each point and specify the changes we made to the manuscript.

Weaknesses

W1: We address each concern below:

Strategy Measurement. It is difficult to score a specific strategy in isolation. For instance, assume there are 100 voters in a state, should a candidate survey all 100 voters to gather accurate information, or should it survey a few (say 60) to get a somewhat reliable insight and save budget. Arguments can be made for both cases and there is no “correct” choice. However, we can measure which topics the candidate chose to publish articles on and match them with the voter’s pre-defined preference. The top candidate, Claude-Sonnet-5, is able to publish an article with the top preference of voters 73% of the time, while the scripted baseline only matches 44%.

Impact on Evaluation and Environment. +35 is the ceiling of one impulse before decay. At the ballot a final-day article is worth at most 26 and a day-1 article 3.5. Realized label effects do not swamp the fixed terms: a 10-point lean still moves the disadvantaged seat from 49.4% to 30.5%.

Budget and the scripted baseline. The disadvantaged seat’s win rate decreases from 49.4% in core to 37.8% in outspent, an 11.6pp difference. Moreover, the 4x gap concerns starting budgets, fundraising remains available. This is evidence of an outcome difference under starting resource asymmetry, it does not isolate resource-management. Similarly, the scripted baseline is a survey-driven policy that selects an issue from survey results, aligns its manifesto and articles with that issue, and reserves funds for donor meetings. The 56.3% win rate in large therefore indicates limited advantage over this policy in that environment, rather than comparison against a policy without information gathering or resource allocation.

Persona Sensitivity. The 9/12 result is on one shared event across 50 personas, rather than invariance across all campaign material. We agree that this does not validate sensitivity to personas. Voter heterogeneity does not rest on the label, however: the simulator applies the same label at weight 1, 0.6, 0.3 or 0 according to the voter’s ranked priorities, so candidates must still find which topics matter where. Identical ordinal reactions therefore need not imply identical voting behavior.

Final Debate. We agree that 24% describes the chosen dynamics and is not evidence of long-horizon strategy. However, our reason for choosing such a setup is to prevent models from spamming articles early in the campaign and lock in a win. Moreover, the debate provides a direct interaction conditioned on the campaign recap and the candidate’s notes, allowing candidates to draw on campaign history. Both candidates can take advantage of the debate to persuade voters. The starting order is switched across runs to prevent any positional bias as well.

Evaluator Strength. We agree that prior work [1] provides relevant evidence that stronger judges can produce a wider range of debater ratings. Our choice to use smaller LLMs comes from resource constraints. We simulate up to 250 voters each in up to 45 states. Voters with that preference plus some spillover are called on every event done by the candidate. Simulating such a huge environment on every matchup is computationally infeasible. We have stated this limitation clearly in the updated manuscript for future readers (§3.3).

[1] Khan, Akbir, et al. "Debating with more persuasive llms leads to more truthful answers." arXiv preprint arXiv:2402.06782 (2024).

W2: Intervals and the top of the ranking. Ordering in Table 4 represents estimated ratings, not evidence that every adjacent entry is statistically distinguishable. One clarification is that 44 games are played per evaluated pair. Ratings are jointly estimated from all completed matchups in the comparison graph, rather than from a single 44-game comparison. We have updated Table 4 to include 95% bootstrap intervals as well. The top three are Claude-Sonnet-5 1946 (1889-2073), Gemma-4-31B-it 1938 (1867-2041), and GPT-6-Luna 1932 (1848-2035). We now distinguish the numerical ordering from a statistically established ordering and no longer claim a uniquely best model in the Abstract and §5.

Evaluation coverage and interface comparisons. There are a total of 82 complete matchups (3608 games). The FC-ReAct differences in §5.2 are measured in Elo points, not percentages, and the section describes them as point estimates. We agree that small differences should not be interpreted as established interface advantages, and §5.2 now says so.

Outcome concentration versus reliability. We agree that the 82.7% statistic is not a reliability check. It is the share of four-game matchup-environment groups, not games, in which one candidate wins at least three. Under four independent fair-coin outcomes, the probability of a four-game sweep is 12.5%, compared with our observed 48.6%. This describes how concentrated outcomes are. We have revised §5.1 to explicitly reflect this.

Strategic restraint. An unspent budget has no terminal value, so a higher final balance cannot cause a win. The paper’s two observations sit at different levels: within a game the winner buys an extra action on 32% of days and the loser on 40%, while across configurations higher-rated models take more actions (Fig. 2). We have not controlled for game state, and trailing candidates spending to catch up is a plausible explanation for the gap. We have removed “strategic restraint” from the abstract.


Questions

Q1: Table 4 now reports 95% bootstrap CIs for all 23 entries. The evaluation comprises 82 complete matchups (3,608 games), and Appendix F lists every pairing. The top four are not separable and have almost fully overlapping intervals. We report them as a top group and no longer order them (see W2).


Q2: Family overlap. §5.1 reports near-zero average within-family effects for Qwen, Granite, and Llama, but a −0.48 ordinal-level effect for Gemma-3-4B evaluating Gemma-4 candidates. Holding the recorded campaigns fixed, removing all voters using the Gemma-3-4B backbone changes the winner in 4.2% of games. After refitting the ratings, the top-four point-estimate ordering remains unchanged. This establishes limited sensitivity to that particular voter backbone.

Larger Voters. We have not completed re-scoring the saved campaigns with 30B-70B voter models within the available rebuttal compute budget. We therefore cannot report Kendall’s tau or determine whether the top-rating gaps widen under the requested larger-voter evaluation.


Q3: We agree these placements would separate strategy from text most directly. One can be computed from the saved games. The other three change what candidates observe during the campaign, so they cannot be re-scored and need new games.

Same games without the debate. The debate follows the last campaign day, so removing it leaves every candidate observation unchanged. It changes 24% of the existing outcomes, including the top-four. Claude-Sonnet-5 still leads the table, followed by GPT-6-Luna, and both Gemma-4-31B-it FC and ReAct take #3 and #4.

Manifesto + debate only, scripted strategy with LLM-written articles, free full surveys. Articles move voter opinion and later surveys reflect it, so candidate’s subsequent choices depend on them. Each of these is therefore a new ladder entry rather than a re-score, which we could not complete in the response period. We will add them to the final version.


Q4: Word limit. The word-limit is set to null by default. We provide an environment variable to change it in the simulator, hence the error message in Appendix B.6. We have added this note in §3.2.

Voter context. Debate transcripts have a median of 977 tokens (max 1268). Voter models were served with their default context limits. None of the debate calls to Yi-1.5-9B-Chat exceeded its limit.

Opening speaker. The opening assignment travels with the competitor, so the same competitor opens both games within a mirrored pair, from opposite seats. Across the two worlds of a profile, the opening competitor alternates. The opening speaker wins 47.4% of games.

Raw inputs and outputs. Our reported case studies examine candidate traces. Since there are thousands of games, we only analyzed some games from the top rankers and bottom rankers.

Q5: Open-weight models were served with vLLM using the generation settings published on each model card, with thinking on by default where the model supports it. GPT-6-Luna, Gemini-3.8-Flash and Claude-Sonnet-5 were called through their APIs with reasoning effort set to high. Temperature is not configurable in these APIs. Moonlight-16B-A3B runs within its 8K context through the conversation budget and memory compression described in §3.2.

OLMo3-32B-Think is the only model whose ReAct configuration forfeited turns (8.3%) while its FC configuration did not, which accounts for its 400-point gap. For the other seven models, either both interfaces produced valid actions (e.g. Gemma, 0.3% forfeits) or both forfeited at similar rates (e.g. Inkling-Small, 7.6% and 7.1%), and their gaps are much smaller (4-116 points).

Donors are played by Qwen2.5-32B-Instruct (AWQ) and OLMo3-32B-Instruct. We have updated §4 to state this.

Reviewer_j6JWOpenReview ↗

Reply to Author's Rebuttal

Thanks authors for the response. These address my W2 and Q1 4 5. I am keeping my score 2 because W1 is still open.. re-scoring with larger voters and policy ablations were not run and the response agrees that debate/persona/ budget results do not show strategy. The topic match result (73% vs 44%) is one model against the script so what the Elo is measuring is still unclear to me..

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.