- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This paper introduces GENSTRAT, which uses a parameterized generator to combine game phases, actions, information visibility, and settlement rules into two-player, zero-sum betting games with imperfect information. Of 12,351 generation seeds attempted, 2,000 passed the acceptance checks. The authors selected 50 of these games to evaluate 12 models, retaining 40,412 matches after data cleaning. The paper estimates five game characteristics through simulation and examines their relationships with model performance. The results show that newer frontier models generally perform better, while models with similar overall strength can excel at different types of games.
Critical strengths
-
By generating new game flows, opportunities to acquire information, and payoff rules, GENSTRAT helps reduce the risk that models have memorized specific game instances and tests how they handle new combinations of rules.
-
The authors swap seats to reduce positional effects on match outcomes. Experiments that remove individual games or use randomly sampled subsets of games also show that the overall ranking of model strength is relatively stable.
-
Beyond model rankings, the authors conduct additional experiments to analyze model behavior. For example, they disclose or hide a programmed opponent's strategy to test whether models can use information about the opponent to improve their returns, extending the evaluation beyond comparisons of which models win against one another.
Critical weaknesses
-
The complete games and rulebooks assembled from components require more thorough validation. Unlike established games whose rules have been tested through extensive use, newly generated games introduce an additional validation burden. Appendix A explains that the system programmatically combines preimplemented mechanics and stitches together their natural-language descriptions. Even if each component functions correctly on its own, this does not ensure that the combined mechanics work coherently or that the rulebook fully explains the relationships among action order, costs, and information visibility. Appendix A reports that 2,000 of 12,351 candidates were accepted, a rejection rate of approximately 83.8%. The screening criteria include whether phases can be reached, game length, and seat balance under random play. Nevertheless, games that passed these checks and were ultimately used for evaluation still had problems in their rule descriptions. In Appendix J, the authors and LLM auditors checked the served rulebooks against the engine and found omissions or errors in 34 of the 50 games. In 12 games, the rulebooks stated that costs for hidden actions were settled at the end of the game, whereas the engine deducted them immediately and the opponent could observe the change in chips. A model that trusts the stated timing may assume that its action does not immediately change its publicly visible chip count, thereby underestimating the opponent's opportunity to infer its action from that change. Stable rankings after excluding games in each defect class only show that this leaderboard is relatively robust to those exclusions. This is a post hoc check of existing results; it does not answer the underlying question of how to validate new games reliably and identify omissions or contradictions in their rules. The persistence of these rule problems despite screening that rejects a large proportion of candidates indicates a need for more thorough validation of the entire process, from generation and screening to rulebook checks.
-
Novel instances do not necessarily pose novel strategic problems. Section 3 defines the generated games as generalized betting games (GBGs), a family that includes Kuhn poker and Leduc poker and extends them with non-betting actions, information structures, and conditional branches. These games are built around decks, private hands, and chips. The generator varies the organization and parameters of betting, simultaneous-move, auction, and observation phases. Appendix A further explains that games combine preimplemented operations such as dealing, peeking, swapping cards, and wagering, with conditional branches and bounded loops creating different game flows. This can reduce the likelihood that models have seen or memorized a complete game instance. However, different rules and flows do not necessarily require different strategic experience. Even without having seen a particular game, a model may perform well by transferring general strategic knowledge—akin to "meta-knowledge"—from similar games, such as inferring private information from an opponent's actions, weighing betting risks, and deciding when to bluff. Thus, while the paper makes a reasonably clear case for instance-level novelty, the experimental evidence and discussion remain insufficient to establish how much generating new instances reduces reliance on prior strategic experience.
-
The formatting and presentation of the main text could be improved. In the "Models" paragraph of Section 5, Tournament Design, excessive spacing between model names makes the text difficult to read. A clearer, more consistent presentation would help. In addition, ICLR allows up to 9 pages of main text for the initial submission, rather than 8. The authors could use the remaining space to include more material that helps readers assess the paper's contribution.
Questions for authors
-
Q1: In light of the rule omissions and errors identified in Appendix J, could the authors further examine the limitations of the current screening and rulebook checks? Which key steps in generating new games still require manual inspection?
-
Q2: Could the authors further discuss whether, although the game instances are newly generated, the methods for solving them may still come from models' prior experience with similar games? If models' pretrained knowledge contains substantial general strategic knowledge applicable to the different generation components, would this weaken the claimed advantages of the benchmark?
-
Q3: Section 6 reports that, on the eight difficult games selected using the composite score, the gap between the highest and lowest fitted model strength scores increases from 3.3 on the full game set to 5.6 on the difficult subset, approximately 1.7 times the original gap. If the difficult games themselves allow more chips to be won or lost, the larger gap may partly reflect differences in payoff scale. Please clarify whether the chip settings are comparable across games.
6 comments
Thank you again for your detailed review. Since the discussion closes tomorrow, we wanted to check whether our reply and the revised paper address your concerns, and please let us know if any further analysis would help. A brief recap:
Validation. Each rulebook sentence comes from a module or a shared preamble: each is checked individually and a fix reaches every game that uses it. Every composed game is then checked automatically by code that compares the rulebook with the engine, and we also had an LLM rebuild each game from its rulebook alone. Most of these checks, including the audits by LLM reviewers, were built after the tournament. We applied them to the games we added during the discussion period. On benchmark games, they found one engine bug, now fixed. They also found wording issues in the rule text, mostly details omitted from the text that created some small ambiguities. To confirm that these did not change the results, we reran in full the four games where they were most likely to matter. In the overall leaderboard, no model moves more than one rank, and the maximum change in a model's strength is 0.13 chips per game. Each issue we found was fixed at its source, in a module or the shared rule text, so all future games will inherit the fix. Once the module library is verified in this way, the code checks run on every new game, and neither manual auditing nor an LLM rebuild is needed, although we recommend LLM checking new games as a cheap safeguard.
We also reran 542 matches across all 34 games with revised rulebooks, on the same deals but with the corrected text and engine. No model changed tier (Appendix O). Finally, gpt-6-astra picks an equilibrium-quality move in 99% of its decisions on the solved games. This suggests that rulebooks convey rules consistent with engine logic. All reruns and audit results will be released with the final version.
Prior strategic knowledge. We note that games composed from the same base pieces, but with different objectives and scoring rules, can pose new strategic problems/trade-offs. Model rankings hold on games outside the betting family, and controlling for the complexity axes, a game's resemblance to Kuhn or Leduc poker does not predict how well the ranking holds (Section 9, Appendix M).
Payoff scale. With a game's payoffs scaled by their standard deviation under random play, hard games still spread the models 1.35 times as far as the full set (Appendix E).
Following your suggestion, the main text now uses the full page allowance for the new Section 9. We would be glad to clarify anything before the deadline.
Thank you very much for your responses. I recognize GENSTRAT's value as a new-game-generation framework. However, its positioning as a benchmark imposes stronger requirements on the reliability and validity of the generated games and their ability to produce meaningful strategic measurements.
1. Task reliability as a benchmark prerequisite. Unlike QA tasks, where the outputs can be directly checked against an external answer (or rubric), game benchmarks evaluate agent capability from outcomes generated through the game. The evaluation signal is therefore only as meaningful as the game itself: for example, an executable but poorly designed game can distort model scores rather than measure true strategic ability. The current checks for aspects like runnability and reachability are basic filters that remove obvious failures. Passing these checks does not establish that the retained games are reliable and well-designed evaluation environments. In contrast, using mature games can mitigate this concern because their rules and strategic properties have already undergone substantial test.
2. Reliability of the new games and rulebooks. Validating individual modules is not equivalent to validating their composition. Locally correct modules can produce an engine that executes successfully but implements unintended global interactions. The same composition issue applies to rulebooks: individually accurate text fragments need not fully or accurately describe these interactions when stitched together.
3. Incremental value over established games. The benefit of generating new game instances, rather than directly using existing mature and well-validated games, remains insufficiently established to me. In particular, if the authors can show that memorization of established games actually produces misleading evaluations, while GENSTRAT provides a more faithful assessment of strategic capability or even can reveal some model capability gaps that old games fail to expose, I believe it would substantially strengthen the paper's contribution and constitute an important as well as interesting finding to the community.
After carefully reconsidering the authors' response and revisions, I do not believe these concerns have been resolved. The authors could reposition GENSTRAT primarily as an automated pipeline for synthesizing new game environments. Alternatively, retaining the benchmark positioning would require addressing the above concerns. As for the current version of the manuscript, it does not meet the bar for acceptance at ICLR. I respect the other reviewers' assessments and my review only reflects my own judgment regarding the evidence required for a reliable and informative benchmark. I am comfortable leaving the decision to the AC.
Thank you for your thoughtful review. You raise two fundamental points: whether generated games can be trusted, and whether they sufficiently pose novel strategic problems.
Q1. Validation and inspection. GENSTRAT consists of modules, each with engine logic and a natural-language description, and a generator that provides minimal structure but otherwise stitches modules together at random into a novel multi-agent game. Two things must work for the result to be trusted: the generator must assemble a working game (and append any preambles correctly), and each module's description must fully and correctly explain its code. Since every rulebook sentence comes from a module or a shared preamble, each is checked once, and a fix reaches every game that uses it. Verification therefore scales with the number of modules rather than the complexity of the games, and once modules are verified, with every composed game then checked automatically (below), we can generate games of new structures and kinds without verifying each one by hand (Appendix O). Two steps remain manual: checking the text of each new module once, and adjudicating the discrepancies the automated checks raise. The acceptance gates screen whether a game is playable and balanced.
Beyond pre-tournament screening, over Oct/Nov, we used LLMs to run many audits: (1) of individual modules, checking each module's rule text against what its code does, with a runnable example for every mismatch; (2) of generated rulebooks, for consistent and unambiguous rules; (3) seeded replays of the engine against recorded play, and a checker for 19 kinds of rulebook–engine consistency. A frontier model also re-implemented each generated game from its rulebook alone, to confirm that rulebook and engine match. We will release these audit results in the camera-ready repository.
On the 50 benchmark games these checks found (1) one engine bug, in about 1% of matches, now fixed, and (2) wording issues, mostly omitted details. Your hidden-cost example is one: a single shared sentence, now fixed in all 12 games, that moved no chips, since the rulebook also says both stacks are public. To test whether wording changed decisions, we re-asked models the same prompts with only that sentence clarified, and no model's decisions changed in any of the 12 test games, except for one auction sentence we already knew about. We reran that game and the engine-bug games in full, and no model moved more than one rank. We reran 542 matches across all 34 games with revised rulebooks, on the same deals but with the corrected text and engine. Models made the same decision 84-94% of the time (a similar rate to repeating a prompt), and no model changed tier.
Q2. Prior strategic knowledge. We agree that a genuine question exists about the degree to which a new instance is a new strategic problem, rather than a repackaging of existing strategic thinking. Novel games rule out memorizing a game's solution, but not transferring general strategic knowledge, a kind of "compositional generalization" from principles learned elsewhere. We note that transfer is in many ways what we intend GENSTRAT to measure.
Stated this way, prior knowledge does not weaken the benchmark's advantages: games cannot be memorized, complexity can be raised on demand. The ranking also holds on generated games with no betting and on six games built with the same engine outside the betting-game generator, including a board game, a market and a negotiation. Controlling for the complexity axes, a game's resemblance to Kuhn or Leduc poker does not predict how well it reproduces the leaderboard.
Composing existing pieces in unusual ways can also produce qualitatively different games: Go and Chess are both built from a board, pieces and rules for moving and capturing, yet call for very different skills. Even at the simplest level, making a player's card public in Kuhn poker removes any reason for that player to bluff. Our results show this too: playing at equilibrium quality requires solving each new game, and only one model manages that on almost every game we could solve (Appendix L). Finally, by expanding to non-betting-game environments, and showing a high correlation in rankings, we show that performance seems to generalize even if we move beyond the card based features you mentioned. Future work would incorporate further abstraction and unusualness into the modules from which our procedural generators are built.
Q3. Payoff scale. You are right that stakes differ across games, so chips are comparable within a game but not between games. With each game's margins divided by the standard deviation of its payoff under random play, hard games still spread the models 1.35 times as far as the full set (Appendix E).
Presentation. The model list is reformatted, and the ninth page now holds Section 9.
We would be grateful to hear whether these results address your main concerns.