acceptodds
Under review as a conference paper at ICLR 2027

Delegating AI Evaluation to AI-Generated Automata in Games

Abstract

AI benchmarks predominantly evaluate models based on their direct, interactive capabilities. However, a principled evaluation of AI usefulness should recognize that true intelligence externalizes cognition, creating durable artifacts that make future work reliable without continuous intervention. The central measure of an AI is shifting from what it can do through regular interaction in a live setting to the utility of the autonomous tools it can build to accomplish goals. We introduce Strautomata, a framework that evaluates models not as interactive players, but as builders of autonomous artifacts (executable policies or automata). We measure this artifact-generating utility within the space of well-known strategic board games. Across 20 distinct games, we sample 5,200 Python-based automata generated by 52 Large Language Models and execute over 2,500,000 offline, head-to-head matches to establish comprehensive Elo ratings. By separating code generation from execution, we ensure highly cost-efficient evaluation while testing vital LLM capabilities, placing our benchmark on the cost–alignment Pareto frontier.

Primary area:
datasets and benchmarks
Keywords:
open-ended framework, evaluation, LLM Benchmark, cost-efficient

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1LxhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Strautomata, a framework that evaluates models as builders of autonomous artifacts within well-known strategic board games. Across 20 distinct games, the authors sample 5,200 Python-based automata generated by 52 LLMs and execute over 2,500,000 offline, head-to-head matches to establish Elo ratings. The authors report that separating code generation from execution enables highly cost-efficient evaluation and places the benchmark on the cost-alignment Pareto frontier.

Critical strengths
  • Extensive experiments are conducted on various frontier models across 20 games, with comparisons against human-written engines and live LLM play.
  • The authors examine ranking stability across repeated runs and different numbers of sampled policies.
  • The reasoning-level experiments show that additional reasoning during code generation can improve the strength of policies that later run without further model intervention.
Critical weaknesses
  • The intra-model qualifier over five sampled programs is not sufficiently motivated, as it only captures minor variations caused by random seeds. The candidates are generated independently, without any negotiation. A more reasonable approach would allow five subagents to collaborate or challenge each other's solutions, revise their programs, and then select the most suitable program.
  • The paper does not establish what the full cross-model tournament adds over evaluation against a small set of opponents. If a simpler evaluation produces similar model rankings, the additional value of the proposed tournament would be limited.
  • The paper provides limited analysis of why stronger LLMs produce better policies. The code-length correlations do not establish how much of the performance gap comes from fewer errors and timeouts, and how much comes from better strategies.
Questions for authors
  • In Go and Checkers, what proportion of the random baseline's wins comes from opponents making illegal moves or timing out, and what proportion comes from normally completed games?
  • Were all models evaluated using the same agent harness? Were they allowed to execute their generated programs and revise them based on execution feedback?
  • What model does each point in Figure 7 represent?
5 comments
AuthorsOpenReview ↗

Discussion with 1Lxh (1/3)

We thank the reviewer for their thoughtful constructive comments. Please find below answers to your questions and concerns. We hope that these results will strengthen the reviewer's belief towards our work.

W1: A more reasonable approach would allow five subagents to collaborate or challenge each other's solutions, revise their programs, and then select the most suitable program.

Q2: Were all models evaluated using the same agent harness? Were they allowed to execute their generated programs and revise them based on execution feedback?

Models were all evaluated under the same conditions with zero-shot generation and no revision. We provide below results on tournaments which include Claude and Codex agents which have zero-shot automata counterparts. We note that for all agents, we provided a fresh working directory which contained the game files, a brief explanation of the task, and useful harness/tools that let them test their policies against random opponents. We conducted this agentic automata creation for seven models in five games (Chess, Go, Checkers, Othello, Connect4).

Within each model and game, the median automaton is the third of the five zero-shot automata ordered by tournament Elo while the top automaton has the highest tournament Elo. The table takes the median across the five games separately for the agent, median automaton, and top automaton.

Model (agent effort: high)AgentMedian automatonTop automaton
Fable 5.12,052.81,997.72,152.5
Opus 4.51,318.11,086.41,169.4
Opus 4.61,668.11,412.91,523.0
Opus 4.71,884.41,568.21,833.5
Sonnet 4.61,416.9984.81,514.2
GPT-5.6 Sol1,804.71,866.21,943.8
GPT-5.6 Sol (5 subagents)1,915.91,866.21,943.8
GPT-6 Astra1,961.41,952.32,201.3
GPT-6 Astra (5 subagents)2,009.81,952.32,201.3
Overall median (35 model/game comparisons)1,746.61,568.21,731.0

Figure 1 shows the median Elo of agent policies, median automata, and top automata across the five games.

The two Codex models were additionally run with 5 subagents to incorporate the reviewer's suggestion since the Codex agents under the regular harness provided automata at or weaker than the median zero-shot automata. Each subagent policy enters the same tournament for all five games, and its row reuses that model's automata.

The above comparisons show that in the games we test, the difference between zero-shot automata creation and agentic automata creation is mostly significant in sonnet/opus and not Fable/Astra most likely due to the games being well known and the frontier LLMs not needing to iterate on the automata to understand the game.

We emphasize however that we think it is important for the automata to come directly from the LLM to have a fair LLM benchmark that all LLMs can compete in as opposed to pairing certain LLMs with agentic harnesses they were not optimized for. If the goal of Strautomata was to measure if a generated automaton can exceed a fixed threshold in capabilities (more akin to SWE-bench tasks) then we agree with the reviewer that it is critical to test agentic automata creation as opposed to zero-shot. However, the goal of Strautomata is to evaluate the LLMs in a pairwise fashion as opposed to finding the strongest possible automata the LLM can make under any and all circumstances, and since the evaluation is done pairwise between the LLMs and the reported scores are relative, having the unified 'zero-shot' standard amongst the LLMs is a fair evaluation that does not disrupt the final Elo scores or rankings (assuming rankings are preserved under a harness). Additionally, we spend some effort ensuring that Strautomata is cost-efficient (which ensures we are at the cost-alignment Pareto frontier) and agentic use of LLMs is heavily cost inefficient.

We hope the above experiment provides a comparison that gives stronger justification to the zero-shot generation process.

AuthorsOpenReview ↗

Discussion with 1Lxh (2/3)

W3: The paper provides limited analysis of why stronger LLMs produce better policies. The code-length correlations do not establish how much of the performance gap comes from fewer errors and timeouts, and how much comes from better strategies.

The rankings in the paper show that stronger LLMs produce stronger automata but here we tackle the reviewer's two main questions: how much of the gap between models comes from execution failures vs better strategy and which design choices in the generated code are associated with stronger play. Throughout, we split the models into thirds based on median Elo (top, middle, bottom) and analyze the 2,080 automata that entered the tournament.

Failures versus play

Each tournament match ends either normally (win, loss, draw) or with a failure from an automaton. Failure rates differ by a large margin depending on which third of the leaderboard we look at. In the top third automata only lose 2.4% of games due to failures, in the middle it's 11.6% and in the bottom third it is 23.6%. To separate failures from play in the final score, we re-fit Elo using only completed games where no failure happened. This no-failure Elo measures how well automata play when failure isn't considered and this leaderboard is nearly identical to the paper's leaderboard (Spearman ρ = 0.983, we illustrate this change in Figure 2). Additionally, to attribute the final Elo, we regress each model's Elo in each game on its no-failure Elo and its failure rate (R2R^2 = 0.88). A Shapley decomposition of this R2R^2 attributes 72% to no-failure Elo and 28% to failures. Thus, an automaton in the bottom of the leaderboard can raise its Elo by avoiding illegal moves but the top of the leaderboard is affected very little when failed matches are removed and is mostly decided through play strategy.

TierIllegal movesRuntime errorsTimeouts
Top0.38%0.00%1.98%
Middle5.71%1.17%4.68%
Bottom14.25%5.82%3.57%

Design choices

We attempt to extract design features from automata with a mix of AST and regular expressions. We find that almost all search architectures used by the automata fall in one of four types:

  1. 'no recursive search' (hand-written rules or scoring)
  2. 'fixed-depth search' (recursive minimax/negamax/expectimax or Monte Carlo with no wall-clock control)
  3. 'time-based search' (similar to 2 but bounded by a deadline, typically iterative deepening)
  4. 'time-based search with engine enhancements' (similar to 3 but includes multiple optimization tricks such as transposition tables, move ordering, quiescence search, heuristics, etc.).

In Figure 3, we show how many automata use the different search architectures split by which third of the leaderboard that automaton is placed in. In Figure 4, we show the mean within-game standardized Elo per search architecture. And in Figure 5, we show the fraction of games lost to errors per search architecture broken down into the types of errors.

We notice several important factors in these figures that dictate strength based on the policy. First, from Figure 3 we see that almost all time-based search with enhancements is written by top-tier automata and that the majority of top automata (72%) rely on some form of time-based search unlike weaker automata that rely less on time-based search (17% for middle and 5% for bottom tiers) and rely mostly on weaker search strategies that fail to find a good move and often time out which makes them not competitive.

Analyzing the AST/regex for a specific game like Chess, we notice that many automata rely on embedded heuristics of piece-square tables (assigning values on the specific squares each piece type can land on) thus implementing a heuristic evaluation function which vastly improves the search strategy but requires a higher amount of game knowledge. We find that 20 out of 36 top automata embed such heuristic values while only 3 of 34 bottom automata attempt implementing them. Automata which use such a strategy in Chess average an Elo of 1,788 compared to automata that don't which average 1,336 Elo.

AuthorsOpenReview ↗

Discussion with 1Lxh (3/3)

Q1: In Go and Checkers, what proportion of the random baseline's wins comes from opponents making illegal moves or timing out, and what proportion comes from normally completed games?

In Go, Random wins 49.3% of its games: 27.3% through opponent failures and 22.0% through normal play. In Checkers it wins 50.0%: 37.1% through opponent failures and 12.9% through normal play. In Chess it wins 36.0%: 35.5% through opponent failures and only 0.4% through normal play. Random's surprisingly high rank in Chess and Checkers is mostly a statement about how often weak automata fail at legality. Go is special in that all games end due to either the 722 max game length rule (twice the number of board points) or by both players passing (thus agreeing to end the game) as opposed to other games that have a regular win condition. This means that the random policy playing white (and getting the 7.5-point komi) only loses against a competent opponent that successfully captures the dead stones that are placed by the random policy before they start accumulating and overwhelm the player with a localized pattern. We believe a visual replay of such games provides an easy understanding of what's happening, thus we'll provide a website that provides a comprehensive visualization of the games and automata battling alongside the leaderboards.

W2: The paper does not establish what the full cross-model tournament adds over evaluation against a small set of opponents. If a simpler evaluation produces similar model rankings, the additional value of the proposed tournament would be limited.

We agree with the reviewer's intuition that a simpler sampling strategy exists. Our main focus was lowering inference cost from the LLMs instead of lowering CPU cost. Thus, we did not attempt to propose any strategy other than the naive O(N2)\mathcal{O} (N^2) round-robin matchup. However, in the interest of making evaluation scalable, we incorporate the reviewer's idea and introduce a simple O(N)\mathcal{O} (N) evaluation. We re-ran the tournament while making each model only match up against KK other models each for rr games. We report agreement with the full leaderboard and the alignment from Figure 2:

ScheduleOrderSpeedupSpearman ρ vs fullKendall τ vs fullFigure 2 Alignment
Full round robin, r=20O(N2)\mathcal{O} (N^2)1×110.748
K=10, r=20O(N)\mathcal{O} (N)5.3×0.9850.9050.736
K=20, r=10O(N)\mathcal{O} (N)5.3×0.9870.9180.736
K=40, r=5O(N)\mathcal{O} (N)5.3×0.9940.9500.748
K=5, r=10O(N)\mathcal{O} (N)21.1×0.9690.8590.721
K=10, r=5O(N)\mathcal{O} (N)21.0×0.9790.8920.734
K=20, r=2O(N)\mathcal{O} (N)26.3×0.9820.8990.738

The reviewer is right that a much smaller evaluation recovers a similar ranking. With 20 random opponents and only 2 games per pairing, which is a 26× speedup and only O(N)\mathcal{O} (N), the leaderboard has ρ = 0.982 with the full one and nearly the same alignment with other benchmarks. We view this as a strength of the framework rather than a limitation of the tournament. Because automata are frozen artifacts, an offline schedule runs without further LLM calls. We use the full round robin for this paper because it is simple, precise, and affordable at the current scale. It also serves as the reference that makes the validation above possible. We will add the above to the appendix and describe the sparse O(N)\mathcal{O} (N) schedule as a path for scaling up Strautomata.

Q3: What model does each point in Figure 7 represent?

We acknowledge the plot should have the points labeled. We thank the reviewer for pointing out the missing labels and have updated the figure to add some more data and labels for both Chess 6 and Connect4 7.

We will incorporate the additional results and experiments into the paper and upload the revised polished version.

We hope the above comments and additional experiments address the reviewer's concerns.

Reviewer_5vYyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces STRAUTOMATA, a benchmark that evaluates LLMs as strategy designers by testing the game-playing programs they generate. For each model and game, five policies are sampled, two are selected through an internal tournament, and the selected policies compete against those from other models. The study covers 52 model/configuration entries and 20 games, with 5,200 generated policies. The experiments examine ranking stability, agreement with other benchmarks, inference costs, and comparisons with live LLM play and conventional game engines.

Critical strengths

A well-motivated evaluation setting. The shift from evaluating an LLM as a player to evaluating it as a strategy designer captures a practical form of model utility: producing reusable programs that can act autonomously. Separating generation from execution also allows extensive evaluation without repeated model calls.

A substantial evaluation with a sensible sampling design. Sampling five policies and selecting two reduces dependence on a single generated program while keeping the generation budget fixed. The evaluation spans a broad set of games and models, and the stability checks provide useful support for the aggregate rankings.

Critical weaknesses
  1. The manuscript appears unfinished in places. Section R, “Additional Details,” contains only a heading on page 46. The paragraph about benchmark saturation at the bottom of page 1 ends abruptly at “included” and is disconnected from the surrounding discussion. These issues should be corrected, with clarification of whether any intended material is missing from the supplied version.

  2. The role of prior exposure to strategy code is unclear. Since the benchmark uses established games, strong performance may partly reflect reuse or adaptation of strategy implementations encountered during pretraining. This is still valuable for producing useful programs, but the paper does not distinguish it from the ability to design strategies for unfamiliar settings. The interpretation of the scores as evidence of strategic reasoning therefore needs more qualification.

  3. A practical gap in the code-generation setup. In practice, LLM-based coding workflows commonly use an agent harness to generate code, whereas this benchmark evaluates code generated directly through a single model call. This creates a gap between the benchmark setting and how LLMs are used for code generation in practice.

  4. The analysis could provide deeper insight into why strategies succeed or fail. The main text emphasizes rankings, correlations, and cost comparisons. Appendices H, L, and P contain useful observations about memory, code length, and execution failures, but these are not developed into a clear account of which design choices improve performance. A few comparative case studies linking generated code to behavior during matches would strengthen the paper and help identify concrete directions for improving the generated strategies.

Questions for authors

Overall, I am positive about the evaluation setting and design, but my main concerns are the manuscript’s presentation and the depth of its analysis; I hope the authors can polish the paper and complement the aggregate statistics with more manual observations and deep analysis from examining the generated strategies and their behavior.

6 comments
AuthorsOpenReview ↗

Discussion with 5vYy (1/3)

We thank the reviewer for their thoughtful constructive comments. Please find below answers to your questions and concerns. We hope that these results will strengthen the reviewer's belief towards our work.

W1: The manuscript appears unfinished in places. [...] These issues should be corrected, with clarification of whether any intended material is missing from the supplied version.

We apologize for the formatting errors. We have cleaned up the text, polished the paper, and will upload a version including all of the additional experiments in the rebuttal that we hope will strengthen the appendix. We can confirm that the intended material is in the supplied version.

W2: The role of prior exposure to strategy code is unclear. Since the benchmark uses established games, strong performance may partly reflect reuse or adaptation of strategy implementations encountered during pretraining. This is still valuable for producing useful programs, but the paper does not distinguish it from the ability to design strategies for unfamiliar settings. The interpretation of the scores as evidence of strategic reasoning therefore needs more qualification.

We agree with the reviewer that LLMs are very aware of the supplied games as well as prior strategies for them in the pretraining data. Our main motivation for not straying away from such familiar games is that empirical results do in fact show a clear separation between model capabilities with newer models exceeding their predecessors. We believe that the time budget enforces a strict constraint which heavily incentivizes code optimization and smarter strategic heuristics that are specifically adapted to the specification. However, Strautomata can incorporate out-of-distribution games with little human effort such as adding variants of Chess where a subset of pieces are removed and/or replaced with different pieces. Similar modifications can be made to other games to increase the number of dimensions (3D Go, 4D Chess, etc.) which is a simple environment to create and complex to solve. We haven't seen the need to increase the complexity as models have yet to saturate on the easier in-distribution games (other than simple tic-tac-toe/nim). This separation exists even for solved games such as Connect4 which have many online analytical solvers yet still have clear ranking differences between frontier models and early 2026 models.

For additional Strautomata environments we do agree that testing harder and modified out-of-distribution games would most likely differentiate the future frontier in a more robust manner.

AuthorsOpenReview ↗

Discussion with 5vYy (2/3)

W3: A practical gap in the code-generation setup. In practice, LLM-based coding workflows commonly use an agent harness to generate code, whereas this benchmark evaluates code generated directly through a single model call. This creates a gap between the benchmark setting and how LLMs are used for code generation in practice.

We provide below results on tournaments which include Claude and Codex agents which have zero-shot automata counterparts. We note that for all agents, we provided a fresh working directory which contained the game files, a brief explanation of the task, and useful harness/tools that let them test their policies against random opponents. We conducted this agentic automata creation for seven models in five games (Chess, Go, Checkers, Othello, Connect4).

Within each model and game, the median automaton is the third of the five zero-shot automata ordered by tournament Elo while the top automaton has the highest tournament Elo. The table takes the median across the five games separately for the agent, median automaton, and top automaton.

Model (agent effort: high)AgentMedian automatonTop automaton
Fable 5.12,052.81,997.72,152.5
Opus 4.51,318.11,086.41,169.4
Opus 4.61,668.11,412.91,523.0
Opus 4.71,884.41,568.21,833.5
Sonnet 4.61,416.9984.81,514.2
GPT-5.6 Sol1,804.71,866.21,943.8
GPT-6 Astra1,961.41,952.32,201.3
Overall median (35 model/game comparisons)1,746.61,568.21,731.0

Figure 1 shows the median Elo of agent policies, median automata, and top automata across the five games.

The above comparisons show that in the games we test, the difference between zero-shot automata creation and agentic automata creation is mostly significant in sonnet/opus and not Fable/Astra most likely due to the games being well known and the frontier LLMs not needing to iterate on the automata to understand the game.

We emphasize however that we think it is important for the automata to come directly from the LLM to have a fair LLM benchmark that all LLMs can compete in as opposed to pairing certain LLMs with agentic harnesses they were not optimized for. If the goal of Strautomata was to measure if a generated automaton can exceed a fixed threshold in capabilities (more akin to SWE-bench tasks) then we agree with the reviewer that it is critical to test agentic automata creation as opposed to zero-shot. However, the goal of Strautomata is to evaluate the LLMs in a pairwise fashion as opposed to finding the strongest possible automata the LLM can make under any and all circumstances, and since the evaluation is done pairwise between the LLMs and the reported scores are relative, having the unified 'zero-shot' standard amongst the LLMs is a fair evaluation that does not disrupt the final Elo scores or rankings (assuming rankings are preserved under a harness). Additionally, we spend some effort ensuring that Strautomata is cost-efficient (which ensures we are at the cost-alignment Pareto frontier) and agentic use of LLMs is heavily cost inefficient.

AuthorsOpenReview ↗

Discussion with 5vYy (3/3)

W4: The analysis could provide deeper insight into why strategies succeed or fail. [...]

Q: I hope the authors [...] deep analysis from examining the generated strategies and their behavior.

We appreciate the interest in deeper analysis on the specific generated automata and hope to provide more insights.

The rankings in the paper show that stronger LLMs produce stronger automata but here we'll try to tackle two questions: how much of the gap between models comes from execution failures vs better strategy and which design choices in the generated code are associated with stronger play. Throughout, we split the models into thirds based on median Elo (top, middle, bottom) and analyze the 2,080 automata that entered the tournament.

Failures versus play

Each tournament match ends either normally (win, loss, draw) or with a failure from an automaton. Failure rates differ by a large margin depending on which third of the leaderboard we look at. In the top third automata only lose 2.4% of games due to failures, in the middle it's 11.6% and in the bottom third it is 23.6%. To separate failures from play in the final score, we re-fit Elo using only completed games where no failure happened. This no-failure Elo measures how well automata play when failure isn't considered and this leaderboard is nearly identical to the paper's leaderboard (Spearman ρ = 0.983, we illustrate this change in Figure 2). Additionally, to attribute the final Elo, we regress each model's Elo in each game on its no-failure Elo and its failure rate (R2R^2 = 0.88). A Shapley decomposition of this R2R^2 attributes 72% to no-failure Elo and 28% to failures. Thus, an automaton in the bottom of the leaderboard can raise its Elo by avoiding illegal moves but the top of the leaderboard is affected very little when failed matches are removed and is mostly decided through play strategy.

TierIllegal movesRuntime errorsTimeouts
Top0.38%0.00%1.98%
Middle5.71%1.17%4.68%
Bottom14.25%5.82%3.57%

Design choices

We attempt to extract design features from automata with a mix of AST and regular expressions. We find that almost all search architectures used by the automata fall in one of four types:

  1. 'no recursive search' (hand-written rules or scoring)
  2. 'fixed-depth search' (recursive minimax/negamax/expectimax or Monte Carlo with no wall-clock control)
  3. 'time-based search' (similar to 2 but bounded by a deadline, typically iterative deepening)
  4. 'time-based search with engine enhancements' (similar to 3 but includes multiple optimization tricks such as transposition tables, move ordering, quiescence search, heuristics, etc.).

In Figure 3, we show how many automata use the different search architectures split by which third of the leaderboard that automaton is placed in. In Figure 4, we show the mean within-game standardized Elo per search architecture. And in Figure 5, we show the fraction of games lost to errors per search architecture broken down into the types of errors.

We notice several important factors in these figures that dictate strength based on the policy. First, from Figure 3 we see that almost all time-based search with enhancements is written by top-tier models and that the majority of top automata (72%) rely on some form of time-based search unlike weaker automata that rely less on time-based search (17% for middle and 5% for bottom tiers) and rely mostly on weaker search strategies that fail to find a good move and often time out which makes them not competitive.

Analyzing the AST/regex for a specific game like Chess, we notice that many automata rely on embedded heuristics of piece-square tables (assigning values to the specific squares each piece type can land on) thus implementing a heuristic evaluation function which vastly improves the search strategy but requires a higher amount of game knowledge. We find that 20 out of 36 top automata embed such heuristic values while only 3 of 34 bottom automata attempt implementing them. Automata which use such a strategy in Chess average an Elo of 1,788 compared to automata that don't which average 1,336 Elo.

We will incorporate the additional results and experiments into the paper and upload the revised polished version.

We hope the above comments and additional experiments address the reviewer's concerns.

Reviewer_AFnaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an evaluation framework named STRAUTOMATA that shifts LLM assessment from direct interaction to the generation of executable, reusable policy programs. Models produce Python automata for board games in a single generation step, after which their downstream utility is evaluated through offline tournaments and Elo ratings. The authors generate 5,200 automata from 52 models across 20 games and conduct over 2.5 million matches, with additional comparisons against human-written engines, live play, and other benchmarks. The results support the framework’s model discrimination and cost efficiency, although the evaluation only measures one-shot program generation without execution feedback.

Critical strengths
  • The paper offers a clear “LLM-as-builder” evaluation perspective. Decoupling artifact generation from execution allows generated policies to be reused in repeated tournaments, evaluated against newly added opponents, and assessed without incurring additional LLM inference costs.
  • The empirical evaluation is extensive and multidimensional, covering diverse games and models, a large number of offline matches, and comparisons with human-written engines, live play, and benchmark cost–alignment analyses. These results provide substantial evidence for the framework’s practical value.
Critical weaknesses
  • This paper describes STRAUTOMATA as a paradigm for “evaluating strategic reasoning” (Sec. 7). However, the score reflects not only the quality of the strategy, but also whether the generated program contains code or formatting errors. The paper should clarify whether the benchmark evaluates strategic reasoning specifically or the broader ability to build executable game-playing programs.
  • The qualifier chooses two of the five programs using only matches among those five. However, a program that performs well against programs from the same model may not perform well against programs from other models. Conversely, the qualifier may discard a program that would perform well in the main tournament. The paper should therefore examine whether the final model rankings change when all five programs enter the cross-model tournament.
  • Appendix E shows useful stability results, but the reported confidence intervals assume that the generated programs are fixed. They do not capture how rankings may change if the five programs are generated again and two are reselected. In addition, each game has its own Elo scale, so an Elo of 1800 in one game may not be directly comparable to an Elo of 1800 in another. The paper should justify taking the median of these scores and show whether using within-game ranks produces a similar overall ranking.
  • The cost comparison includes the API cost of generating the automata, but not the computational cost of running the qualifiers and tournaments. It therefore supports the claim of lower LLM inference cost, but not yet lower total evaluation cost.
Questions for authors
  • If the full evaluation is repeated with newly generated sets of five automata, how much does the model leaderboard change? Please report the rank correlation and top-model overlap across independent runs, preferably for several games.

  • The qualifier selects two automata using only matches among the five generated by the same model. Are these two also the strongest when competing against programs from other models? How does the leaderboard change when using one randomly selected automaton, all five automata, or the two selected by the qualifier?

  • Each game has its own Elo scale, so the same Elo score may not have the same meaning across games. Why is it appropriate to take the median of these scores? Does using each model’s rank within each game produce a similar overall leaderboard?

  • The cost comparison focuses on LLM inference. Could the authors clarify this scope and briefly report the hardware and runtime required for the offline qualifiers and tournaments?

6 comments
AuthorsOpenReview ↗

Discussion with AFna (1/3)

Thank you for your thoughtful review and constructive comments. Please find below answers to your questions. We will update the paper to include these results and hope that these results will strengthen the reviewer's belief towards our work.

Q1: If the full evaluation is repeated with newly generated sets of five automata, how much does the model leaderboard change? Please report the rank correlation and top-model overlap across independent runs, preferably for several games.

To test whether the qualifier produces a leaderboard that depends on which automata happen to be sampled, we ran the full pipeline (automata generation, intra-model qualifier, and tournament) on a subset of LLMs twice with completely disjoint sets of automata over all 20 games. Due to time and cost constraints we focus on two disjoint sets of 3 automata each. In run A we consider automata #1, #2, #3 while in run B we consider automata #4, #5, #6. In each run the qualifier selects 2 of the 3 per model, and the qualified automata enter the tournament. Each model is represented in a game by its highest-rated qualified automaton, and the overall leaderboard is the median Elo over the 20 games, as in the paper. The correlations between the two runs are:

Spearman ρKendall τTop-5 overlapTop-10 overlapSame Top-3
All 35 entries0.9380.8124/58/10Yes

Most differences are swaps between adjacent models with similar median Elo. Using mean within-game rank instead of median Elo also gives a similar result (ρ = 0.920).

The above experiment shows that even with only 3 automata per model, the leaderboard produced by the qualifier is highly consistent. We will add this replication experiment to the appendix.

W3: [...] an Elo of 1800 in one game may not be directly comparable to an Elo of 1800 in another [...]

Q3: Each game has its own Elo scale, [...] Why is it appropriate to take the median of these scores? Does using each model’s rank within each game produce a similar overall leaderboard?

We thank the reviewer for the good observation. First, we ensure that each game's Elo is centered so that the mean Elo of the tournament entries is exactly 1500. We use the standard 400-point scale (Eq. 6 where E0=1500E_0=1500, S=400S=400). This makes an Elo of 1800 mean the same across games which is an expected score of ∼85%\sim 85 \% against the pool mean. However, we agree with the reviewer that the spread of the scores is different based on the game's skill gap where harder games with larger skill gap spread Elo further from 1500. This is the reason we aggregate with the median so a few games with very wide spreads cannot dominate the score. We provide alternative aggregation methods below to validate our choice of median Elo.

We provide a rank-based view that the reviewer asks for in Table 3 which reports each model's CI-separated rank by game, and Figure 14 plots each model's game and mean ranks. The equations for the Elo confidence intervals and rank are given in Appendix A.

We compare the median-Elo leaderboard against several scale-free aggregations:

AggregationSpearman ρKendall τ_bTop-5 overlapTop-10 overlap
Median CI-separated rank0.9920.9355/59/10
Mean CI-separated rank0.9910.9254/59/10
Median point-estimate rank0.9900.9244/59/10
Mean point-estimate rank (Borda)0.9910.9363/59/10
Mean of per-game z-normalized Elo0.9910.9315/59/10

All aggregations agree with the median-Elo leaderboard at ρ≈0.99\rho \approx 0.99 and τ≈0.93\tau \approx 0.93. Most differences are a few swaps between adjacent models with nearly identical scores. We will add the above table to the appendix with the justification for using the median Elo.

AuthorsOpenReview ↗

Discussion with AFna (2/3)

W2: The paper should therefore examine whether the final model rankings change when all five programs enter the cross-model tournament.

Q2: How does the leaderboard change when using one randomly selected automaton, all five automata, or the two selected by the qualifier?

We agree with the reviewer that the robustness of the qualifier is important to verify. To test the robustness of the qualifier and how it changes the leaderboard, we run multiple variants as proposed by the reviewer:

variant: randomly selected automaton

We run a variant with no qualifier where every automaton competes against automata from other models. At each step we select two models whose Elo confidence interval is still wide, pick one random automaton from each of the two models and play a single match between them. A model's rating pools all of its automata, and the CIs are recomputed as matches finish. The tournament stops once every model's 95% CI is narrow. We run this on five games (Chess, Checkers, Go, Othello, Connect4). Logically, this is different from our current tournament as a model which generates two extremely good automata and three horrible ones would get a very good Elo in the main tournament yet possibly score mediocre in this version of the pipeline as the three bad automata would pull down its Elo.

Additionally, we compare this random automata leaderboard with the main leaderboard along with the replication runs A and B described in our answer to Q1. For each tournament setup, we aggregate the five games into one leaderboard, using either the median Elo or the mean rank, and report the Pearson correlation with the main leaderboard:

MethodMedian EloMean rank
Run A (automata #1, #2, #3)0.900.94
Run B (automata #4, #5, #6)0.870.92
Random automata (no qualifier)0.900.93

The leaderboard where all automata enter and are randomly sampled agrees with the main leaderboard as closely as a rerun of the regular pipeline with 3 automata. We note that the above experiments share many underlying automata and are thus correlated. The goal of the experiment is to measure the robustness of the qualifier, not the variance of the automata generation process. We will add this experiment to the appendix.

variant: all five automata

We run a full round robin tournament with no qualifiers in which all five automata play. This is computationally expensive which limits our analysis to Chess. This gives each automaton a final Elo (a final tournament Elo unlike the model-specific qualifier Elo). For each model we check whether the automaton with the highest Elo in the O(N2)\mathcal{O} (N^2) tournament was actually selected by the qualifier round in our paper:

ModelsRateRandom selection
Best O(N2)\mathcal{O} (N^2) automaton is qualifier's #138/5273%20%
Best O(N2)\mathcal{O} (N^2) automaton is qualifier's top 248/5292%40%

Out of the 52 LLMs in the tournament, in the 4 models whose best automaton was not picked by our qualifier, the misses are due mostly to disagreements between automata whose O(N2)\mathcal{O} (N^2) Elos are statistically indistinguishable (using the 95% CI).

Finally, the same round robin lets us compare Chess leaderboards where all five automata enter the cross-model tournament against the main leaderboard. Over the models shared by all runs, the correlation of each method's Chess leaderboard with the main leaderboard is:

MethodElo (Pearson)Rank (Kendall τ)
Run A (automata #1, #2, #3)0.870.69
Run B (automata #4, #5, #6)0.920.75
Random automata (no qualifier)0.890.76
All automata (O(N2)\mathcal{O} (N^2), no qualifier)0.900.79

Note that the correlations here are lower due to only comparing on Chess as opposed to the usual median of 20 games. Letting all five automata enter the tournament produces a Chess leaderboard that mostly agrees with the main leaderboard as well as the multiple reruns of the pipeline. We will add both tables to the appendix.

We agree with the reviewer that theoretically it is possible for an automaton to lose against its sibling automata yet be better than them in the overall tournament and vice-versa especially given the non-transitive property of strategies. However, we aim to find the automaton that is the most representative of the capabilities of the LLM not necessarily the most capable Chess-playing automaton, thus we are not interested in a surprisingly good outlier automaton if the model's own variants were able to beat it and knock it out of the qualifiers. Beyond the robustness measurements above, we were motivated to model the tournament more akin to the Olympics or various international competitions where each country needs to decide within itself a small subset of entrants to represent the country using a national selection process, and our qualifiers were a simple variant of this process.

AuthorsOpenReview ↗

Discussion with AFna (3/3)

W1: However, the score reflects not only the quality of the strategy, but also whether the generated program contains code or formatting errors. The paper should clarify whether the benchmark evaluates strategic reasoning specifically or the broader ability to build executable game-playing programs.

We agree with the reviewer that if there was indeed a large fraction of code/formatting errors then much of the score would reflect that as opposed to broader ability and strategy. However, a breakdown of the automata shows that errors rarely happen outside the bottom third of the leaderboard. Looking at failure rates, we see that they differ by a large margin depending on which third of the leaderboard we look at. In the top third automata only lose 2.4% of games due to failures, in the middle it's 11.6% and in the bottom third it is 23.6%. To separate failures from play in the final score, we re-fit Elo using only completed games where no failure happened. This no-failure Elo measures how well automata play when failure isn't considered and this leaderboard is nearly identical to the paper's leaderboard (Spearman ρ = 0.983, we illustrate this change in Figure 1). Additionally, to attribute the final Elo, we regress each model's Elo in each game on its no-failure Elo and its failure rate (R2R^2 = 0.88). A Shapley decomposition of this R2R^2 attributes 72% to no-failure Elo and 28% to failures. Thus, an automaton in the bottom of the leaderboard can raise its Elo by avoiding illegal moves but the top of the leaderboard is affected very little when failed matches are removed and is mostly decided through play strategy.

TierIllegal movesRuntime errorsTimeouts
Top0.38%0.00%1.98%
Middle5.71%1.17%4.68%
Bottom14.25%5.82%3.57%

W4: The cost comparison includes the API cost of generating the automata, but not the computational cost of running the qualifiers and tournaments. It therefore supports the claim of lower LLM inference cost, but not yet lower total evaluation cost.

Q4: The cost comparison focuses on LLM inference. Could the authors clarify this scope and briefly report the hardware and runtime required for the offline qualifiers and tournaments?

The reviewer is correct that the cost in Figure 2 (cost-alignment Pareto frontier plot) is the LLM inference cost only. This choice keeps the comparison consistent with the other benchmarks in the pool which almost never report computational costs aside from LLM inference. However, below we report the offline compute cost which when added to our cost still keeps Strautomata on the Pareto frontier. All qualifiers and tournaments run on CPU and each match is a single process. We sum the measured elapsed time of every match:

PhaseMatchesMovesMean time per matchTotal compute
Qualifiers714,60014.2M7.8 s1,556 core-hours
Tournament2,282,70044.8M7.5 s4,784 core-hours
Total2,997,30059.0M7.6 s6,340 core-hours

At 128 workers this is roughly 2 days on a server cluster. To estimate the cost of running this on the cloud, we consider a single Google Cloud c3-highcpu-44 instance (44 vCPUs, about \0.61/hour). The ~6,400 vCPU-hours take about 6 days and cost about \89 in total. This is the cost for the entire leaderboard of 52 models across 20 games. Thus, for 2 of the 52 models (Fable and Astra as in Figure 2) that is approximately $4 which when added to the cost-alignment calculation has minimal impact.

We would like to note that to scale the evaluation to many more models without substantial computational costs, we can introduce adaptive Elo techniques which cut down the number of matchups needed as opposed to our trivial full round-robin matchup. We will add the runtime cost tables to the appendix.

We hope the above comments and additional experiments address the reviewer's concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank all three reviewers for their thoughtful feedback and constructive suggestions. We are grateful that all three reviewers gave ratings leaning toward acceptance, recognizing the importance of Strautomata as a clear and extensive evaluation platform.

  • Reviewer AFna highlights that the evaluation is a clear “LLM-as-builder” setup and recognizes the benefits of decoupling artifact generation from execution, as well as the extensiveness and multidimensional nature of the evaluation.
  • Reviewer 5vYy highlights the well-motivated evaluation, which captures a practical measure of model utility. Overall, the reviewer is positive about the evaluation setting and design.
  • Reviewer 1Lxh also highlights the extensive experiments conducted on frontier models along with various stability and robustness checks.

During the discussion phase, we have addressed the reviewers' remaining concerns through additional experiments, analyses, and manuscript revisions:

  • For Reviewer AFna, we tested the robustness of the leaderboard in three ways: replicating with disjoint sets (Appx. E.3), evaluating multiple variants with no qualifiers (Appx. B.1), and comparing with scale-free aggregation (Appx. E.4), which all show very high agreement. We additionally analyze separating failures from play (Appx. L.1) and report details about offline CPU cost, which keeps Strautomata on the Pareto frontier (Appx. P.1).
  • For Reviewer 5vYy, we polished the manuscript and fixed formatting issues. We added a discussion of prior exposure to games (Appx. K.1) and compared our pipeline with agentic automata generation (Appx. M). We also added a strategy analysis (Appx. L.1 and L.2).
  • For Reviewer 1Lxh, we evaluated their suggestion to use subagents for automata generation (Appx. M). We added further analysis of sparse tournament schedules recovering the full leaderboard at 26x lower compute (Appx. P.2). We provided additional details on the wins by the Random policy (Appx. D.1), labeled the live play figures (Figures 7.a and 7.b), and provided additional strategy analysis (Appx. L.1 and L.2).

We have incorporated these additional experiments and analyses into the revised manuscript and provided detailed explanations in our individual responses to reviewers.

We hope these revisions address the reviewers' remaining concerns and would be grateful if the reviewers would take them into account in their final assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.