acceptodds
Under review as a conference paper at ICLR 2027

GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games

11 views

Abstract

Post-training language models on two-player zero-sum games provides a verifiable reward signal, and the resulting gains transfer to general reasoning. However, existing pipelines rely on a handful of hand-written games, with advantage estimators that are tied to those games. In this paper, we present GAMESMITH, a framework that scales this recipe on both fronts. To scale the supply of games, a language model synthesises games inside templates whose reward, observation, and hidden-information code is fixed. The generated game logic is verified by seed-paired replay and by an independent re-implementation of the same specification, and each verified game is then expanded into encoding, rule, and size variants. To scale post-training, GAMESMITH first distills a teacher’s self-play on the synthesised games into the model and then runs self-play reinforcement learning with CRAB, an advantage estimator that credits each decision against a shrinkage estimate of the expected return of its position and is defined on every variant of the corpus. Building on the three templates derived from the SPIRAL games, GAMESMITH produces 26 new games in 29 families, with 336 verified game variants. On Qwen3-4B-Base, distillation from the synthesised games raises the 13-benchmark reasoning average from 45.5 to 62.5, exceeding the same budget of the three SPIRAL games, and self-play with CRAB further improves it to 64.9. This ordering holds consistently across models, and transfer grows with the number of game families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
zero-sum game, self-play, post-train

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_dFreOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the use of self-play on two-player zero sum games as a way to transfer learning to other tasks. This is done by generating new variants of games from an earlier work, SPIRAL, and designing improved baselines. Experiment shows improved performance over SPIRAL.

Critical strengths
  • The work is competently done, with well designed templates, and improved baselines which can serve as references for further work.
  • The experiment results are positive, validating the earlier work, and improving upon it.
Critical weaknesses

Novelty is not high, mostly showing that results can be further improved with more data and better baseline estimation. The baseline estimation is improved from SPIRAL but the method is also not highly novel.

Questions for authors
  • What are the new insights from this work, beyond what was shown in SPIRAL?
  • (From AI review) Table 2 reports 64.9 for Qwen3-4B-Base with CRAB. However, Figure 5a, explicitly described as showing the corresponding runs, appears to peak near 64.2, around step 140. Table 22 identifies step 120 as the selected CRAB checkpoint, whose plotted score appears closer to 63.9. Can the authors explain the discrepancy?
4 comments
AuthorsOpenReview ↗

Response to Reviewer dFre

We appreciate the reviewer for the thorough review and constructive feedback. All comments are addressed below. The manuscript has been updated accordingly. We sincerely hope our responses can be taken into account in re-evaluating the merit of our paper. Much appreciated.

Q1

"This is done by generating new variants of games from an earlier work, SPIRAL, and designing improved baselines. Novelty is not high, mostly showing that results can be further improved with more data and better baseline estimation. The baseline estimation is improved from SPIRAL but the method is also not highly novel. What are the new insights from this work, beyond what was shown in SPIRAL?"

A1: We respectfully disagree. Scaling is central in the recent success of AI, which is technically not easy. Our paper provides an efficient scaling approach in a new and important direction.

SPIRAL manually designs three games as supply to self-play. The paper acknowledged two key problems in scaling the paradigm: (1) designing games needs significant human labour, and (2) post-training is becoming unstable when game number grows.

Our paper solves the two problems.

We automate the supply of games via a carefully designed synthesis method for generating two-player zero-sum games without human labour. Each game is verified with replay tests and an independent re-implementation (Appendix A.3), so its rewards can be trusted. In our experiments, we created 26 new games in 29 families. Appendix A.4.3 now shows nine of them as game cards.

In addition, our CRAB makes self-play stable at scale. It shrinks the leave-one-out baseline towards a prior shared across the corpus. Lemma 1 shows that gradient variance is controlled by CRAB, whose performance never falls more than 0.4 points below its start.

Our work also provides new insights that could generalise to new research:

  1. Diversity of games matters. At a fixed number of trajectories, going from 4 to 23 families raises the mean from 59.1 to 63.5 (Figure 3). Adding rule, size and encoding variants of SPIRAL's three games at the same budget lowers it (57.7 vs. 60.6, see our reply to WGGx, A1.2).

  2. Games without annotation can beat annotated reasoning. GameSmith beats distillation from eight annotated, answer-filtered datasets on three of four models (Table 1).

Q2

"Table 2 reports 64.9 for Qwen3-4B-Base with CRAB. However, Figure 5a, explicitly described as showing the corresponding runs, appears to peak near 64.2, around step 140. Table 22 identifies step 120 as the selected CRAB checkpoint, whose plotted score appears closer to 63.9. Can the authors explain the discrepancy?"

A2: Thanks and addressed. Figure 5a plots a centred three-point moving average. Step 120 scores 64.9 (Table 2) and is plotted as the mean of steps 100, 120 and 140. The plotted peak at step 140 (64.2) is likewise the mean of steps 120, 140 and 160. We have now clarified this in the caption of Figure 5 and plotted the overlaid raw scores. The results are still unchanged.

Dear Reviewer dFre,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response has addressed your concerns. We would be glad to answer any further questions.

Thanks for the rebuttal. I still have novelty concerns and retain my rating.

Thank you for your reply, and we take it that the question on Figure 5 has been resolved and the remaining concern is novelty, on which we respectfully hold a different view. We thank you for your review and it has helped us improve the clarity of the paper.

Reviewer_iKEhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GameSmith introduces a framework for scaling LLM post-training with automatically generated two-player zero-sum games. It synthesizes and verifies diverse game variants, then trains models through teacher self-play distillation followed by self-play RL with CRAB, a baseline that combines batch-level returns with prior performance across related games. The resulting dataset contains multiple game families and verified variants, and consistently improves reasoning performance over training on the original SPIRAL games.

Critical strengths
  1. The paper tackles an important problem in scaling game-based self-play. GameSmith automates game generation and verification, reducing the need for manually designed environments while preserving verifiable rewards.
  2. The experiments are comprehensive and convincing. Results across multiple models and benchmarks, together with scaling and ablation studies, show clear benefits from increased game diversity and self-play training.
Critical weaknesses
  1. The novelty is somewhat limited. The idea of generating synthetic tasks for self-improvement has been explored in prior works [1 - 3]. GameSmith extends this idea to game-based self-play with verifiable rewards, but the core direction is not fundamentally new.
  2. The scope of the framework is still fairly narrow. GameSmith only considers two-player zero-sum games generated from three predefined templates, so it is unclear how well the approach would extend to multiplayer, less structured, or more open-ended environments. The paper also does not provide enough analysis of how the choice of templates or different types of game variations affect the final gains. Broader game settings and more targeted ablations would strengthen the paper.
Questions for authors
  1. How sensitive are the results to the three predefined game templates? Do some templates contribute substantially more than others?
  2. How well would GameSmith generalize beyond two-player zero-sum games to multiplayer games, less structured environments, or games with fundamentally different interaction mechanisms?
  3. While Figure 3 studies the number of game families and training trajectories, could the authors further disentangle the contributions of newly synthesized game families and rule, size, and encoding variants? In particular, which types of variation are most important for downstream reasoning performance?

[1] Huang, Chengsong, et al. "R-zero: Self-evolving reasoning llm from zero data." International Conference on Learning Representations. Vol. 2026. 2026. [2] He, Yicheng, et al. "Visplay: Self-evolving vision-language models from images." arXiv preprint arXiv:2511.15661 (2025). [3] Li, Zongxia, et al. "Self-rewarding vision-language model via reasoning decomposition." arXiv preprint arXiv:2508.19652 (2025).

4 comments
AuthorsOpenReview ↗

Response to Reviewer iKEh

We appreciate the reviewer for the thorough review and constructive feedback. All comments are addressed below. The manuscript has been updated accordingly. We sincerely hope our responses can be taken into account in re-evaluating the merit of our paper. Much appreciated.

Q1:

"The idea of generating synthetic tasks for self-improvement has been explored in prior works [1 - 3]. GameSmith extends this idea to game-based self-play with verifiable rewards, but the core direction is not fundamentally new."

A1: We respectfully disagree. Scaling AI through data and tasks synthesis is central to the recent success of AI, which has profound complexity and difficulty. Our paper provides an efficient scaling approach in a new and important direction.

To the best of our knowledge, our paper is the first in the literature that scales LLM post-training by self-play on synthesised two-player zero-sum games, and every game is verified by replay tests and an independent re-implementation before training.

Our approach enables post-training without annotation or a reward model, while constructing verified rewards through a game engine that verifies which player wins in games. We developed CRAB that is provably an empirical-Bayes model realising variance reduction, which enables generalisation across environments and stabilise training (Lemma 1, Appendix B.1). Extensive experiments validate them. In contrast, the listed papers do not involve games at all. R-Zero and VisPlay generate single-turn questions and label them by majority vote over the model's own answers, so their reward is only as reliable as the model itself. Vision-SR1 generates no new tasks, and instead adds a self-reward to training on annotated questions. None of them builds multi-turn environments whose reward is exact and verified.

We have carefully discussed and cited these papers in the related work section.

Q2.1:

"The scope of the framework is still fairly narrow. [...] it is unclear how well the approach would extend to multiplayer, less structured, or more open-ended environments."

A2.1: Thanks. We focus on two-player zero-sum games on purpose. In this setting the game engine alone decides the winner, so the reward is exact and needs no annotation and no judge model. It is also the setting of SPIRAL, which keeps the comparison fair. Within this setting, GameSmith already broadens the scope from three hand-written games to 29 game families. Extending the synthesis to multiplayer and more open-ended environments is an interesting direction, and we leave it to future work.

Q2.2:

"The paper also does not provide enough analysis of how the choice of templates or different types of game variations affect the final gains.How sensitive are the results to the three predefined game templates? Do some templates contribute substantially more than others?"

A2.2: The three templates follow SPIRAL to ensure fair comparison.

To further justify how the results depend on the templates, we added a template sensitivity analysis. We distilled Qwen3-4B-Base on each template alone, with the same budget for all four runs (about 18M tokens). The results are shown in the following table.

TemplatesMean
Grid only63.8
Card only63.0
Negotiation only59.1
All three63.2

Grid game and card game provide the most transferability and negotiation provides the least. SPIRAL observed the similar results (negotiation 40.0 vs. 41.3 and 41.4, Table 9 in their paper) but they decided to keep negotiation game, because it trains strategic optimisation, which shows up in strategic games rather than in maths benchmarks. We added this table to Appendix D.1.

Q2.3

"[...] could the authors further disentangle the contributions of newly synthesized game families and rule, size, and encoding variants? In particular, which types of variation are most important for downstream reasoning performance?"

A2.3: Thanks for this interesting suggestion. Accordingly, we conducted additional experiments in the rebuttal session.

To separate new families from variants, we built an augmented SPIRAL baseline. It uses SPIRAL's original three games and extends them to 23 game variants, with the same teacher and token budget as SPIRAL. We then distil Qwen3-4B-Base on it with the same recipe. The results are shown in the following table.

Mean
SPIRAL60.6
Augmented SPIRAL57.7
GameSmith62.5

Increasing the number of families is more beneficial than adding variants. More variants improve maths and knowledge (+4.2 and +1.6) but affect logic and spatial reasoning performance. New families tackle this imbalance issue. At a fixed number of trajectories, going from 4 to 23 families boosts the performance from 59.1 to 63.5 (Figure 3).

These results have been included in the revised draft (§5.2, RQ3, and Appendix D.1).

Dear Reviewer iKEh,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response has addressed your concerns. We would be glad to answer any further questions.

Reviewer_iKEhOpenReview ↗

GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games

Thx for the rebuttal.I still feel that the contributions are not novel enough and the scope of the framework (two-player zero-sum game) is narrow. So I will maintain my rating.

Thank you for your reply, and we take it that the experimental questions have been addressed and the remaining concerns are novelty and scope. Two-player zero-sum games are the training environment rather than the target, and our contribution is to scale them up through verified synthesis, so that self-play keeps improving general reasoning as more games are added. We respect your view, and thank you for the feedback.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer WGGx (1/2)

We appreciate the reviewer for the thorough review and constructive feedback. All comments are duly addressed below. The manuscript has been updated accordingly. We sincerely hope our responses can be taken into account in re-evaluating the merit of our paper. Much appreciated.

Q1.1:

"Limited evidence that the proposed game synthesis achieves substantial diversity beyond structured variations of existing games. [...] the betting and negotiation templates primarily vary the scoring or ranking functions while preserving most of the underlying interaction dynamics. Furthermore, 173 variants modify only the encoding without changing the game mechanics."

A1.1: Thanks. The evidence was well presented in Appendix D.1. Following your suggestion, we have moved it to the main text at §5.2, to make it clearer. Specifically,

  1. Figure 4 shows that the synthesised game families spread over all three seed games. In contrast, the SOTA method, SPIRAL only used three hand-written games. According to the Vendi score, the corpus holds 9.1 effectively distinct games by mechanic, against 2.5 for the SPIRAL games.

  2. RQ4 in Section 5 and Appendix D.1 show that changing the scoring mechanisms and rules in the synthesis significantly changes the winning strategies of the generated games. We leveraged this to synthesise diverse rule variants within new game families that lead to different outcomes from the same moves in a median of 63% of episodes.

Further, our newly conducted experiments (please see more details in A1.2) clearly demonstrate that just varying game instances, while fixing game families, cannot achieve a satisfactory result. We augmented SPIRAL, maintaining the game families / settings, while varying specific rules, sizes (of boards, number of cards, etc.), and encoding variants, which achieves 57.7 (see A1.2 below), below both SPIRAL (60.6) and GameSmith (62.5).

Q1.2:

"Can the authors compare GameSmith against an augmented SPIRAL baseline that generates a comparable number of rule, size, and encoding variations from the original three games, while keeping the teacher and training-token budget fixed?"

A1.2: Thanks for this helpful suggestion. Accordingly, we conducted additional experiments during the rebuttal session. We augmented SPIRAL to involve 23 game variants, following the same setting, and compared it with our method. The results are shown below.

MeanMathsKnowl.LogicSpatial
Augmented SPIRAL57.743.849.974.769.1
GameSmith62.545.749.683.478.4

The results clearly suggest our method significantly outperforms the augmented SPIRAL.

These results have been included in the revised draft (§5.2, RQ3 and Appendix D.1).

Q2.1:

"CRAB combines a leave-one-out return estimate with a descriptor-conditioned prior using a standard shrinkage formulation. Although well motivated for sparse game coverage, the estimator is conceptually close to established empirical-Bayes and control-variate approaches."

A2.1: Thanks for this interesting comment! We fully agree that CRAB is an empirical-Bayes estimator and enables variance reduction. This is exactly our motivation in designing it, and we have provided relevant theory in Lemma 1, Appendix B.1.

This design has impressive advantages at scale: (1) as an empirical-Bayes estimator it helps generalise the baseline performance across games and training steps; particularly, it extends the baseline estimation to games with few or no episodes in the current batch, and (2) variance reduction helps control the impact of rare cases, outliers, and noises, so that effectively stabilise training.

The advantages have been validated empirically: (1) our experiments clearly show this Bayesian estimator significantly outperforms RAE (the previous SOTA of self-play RL on zero-sum games) on every model, while removing it will cause considerable performance drop on the downstream tasks (Appendix D.4), and (2) CRAB is stable across different random seeds while RAE does not (Appendix D.5).

AuthorsOpenReview ↗

Response to Reviewer WGGx (2/2)

Q2.2:

"What is the individual contribution of the corpus-level prior and the template-derived state-value term in Eq. (5)? In particular, how do RLOO, prior-only, and CRAB perform with and without the state-value term under otherwise identical RL settings?"

A2.2: The corpus-level prior enables characterisation of how well a seat (first / second player) does in a game even when that game has few or no episodes in the current training batch, which is particularly helpful in training across games. The template-derived state-value term V(sk)V(s_k) provides much finer characterisation of advantage for each decision, while existing methods rely on a coarser estimation in episode level.

For further justification, we conducted an ablation study based on a distilled Qwen3-4B checkpoint, as shown in the following table. To remove the state-value term, we set λ=0\lambda=0. To keep only the prior, we play each variant once (G=1G=1).

BaselineState valueMean
CRAB (LOO + prior)yes63.6
CRABno (λ=0\lambda=0)61.6
Prior only (CRAB at G=1G=1)yes63.4
LOO only (RLOO)yes61.4
RAEyes61.1

The results clearly show that both parts make significant contributions. The prior gives most of the gain, while without the state-value term, the performance drops compared with CRAB. Without the prior, RLOO is 2.2 points below CRAB. We added this table to Appendix D.4.

The analysis in Appendix B.2 also supports the justification. At step 120, the baseline error is 0.240 for LOO, 0.129 for the prior and 0.096 for CRAB.

Q3

"How robust are the main performance improvements across independent training seeds? Could the authors report uncertainty estimates for the [...] CRAB-versus-RAE comparisons, especially where the improvements are below two percentage points? Could the authors report uncertainty estimates for the matched-budget distillation [...] comparisons, especially where the improvements are below two percentage points?"

A3: Thanks, it’s a good question. Following your suggestion, we conducted experiments with independent seeds. We trained with two more seeds for each Qwen3-4B self-play pipeline, as shown in the table below (three runs each, mean ± std).

CRABRAESPIRAL
Checkpoint selected as in Table 263.7 ± 1.262.5 ± 1.161.9 ± 0.6
Mean over all checkpoints62.9 ± 0.860.8 ± 1.961.7 ± 0.9

The experiments fully support our method. CRAB is consistently outperforming. More importantly, across all three runs, CRAB’s lowest-scoring checkpoint is no more than 0.9 points below its distilled starting point, while RAE’s is 3 to 12 points below. We added these results to Appendix D.5.

In addition, our experiment (Figure 5) shows a clear, significant declining trend of pass@1 accuracy during training with RAE across all settings, which is strong side evidence, as it shows the same pattern consistently exists over different cases.

Q4.1:

"The Mistral-Nemo-Instruct results show that non-game reasoning distillation substantially outperforms GameSmith, despite GameSmith outperforming SPIRAL. What factors explain this discrepancy [...]?"

A4.1: Thanks.

It is not entirely fair to directly compare GameSmith with other methods in the non-game setting. The non-game setting here uses eight annotated datasets, keeping only traces that match the gold answer: it involves in-domain, annotated data which our method doesn’t need / have.

However, even under this unfair comparison, we win in most cases (on Qwen3-4B, Qwen3-8B, and GLM-4-9B).

Mistral-Nemo-Instruct-2407 is the exceptional case that we fail. Even there, GameSmith improves 4.8 points (43.1 to 47.9) over the baseline Mistral-Nemo-Instruct-2407 model, while the SOTA, SPIRAL, has declining performance (43.1 to 42.7).

Q4.2:

"[...] what evidence supports the generality of the proposed training approach across different model families and initialization regimes?"

A4.2: Thanks.

Our paper (Table 1) has covered four models (Qwen3-4B, Qwen3-8B, Mistral-Nemo-Instruct, GLM-4-9B) from three families (Qwen, Mistral, GLM), as well as two initialisation regimes (base and instruct). GameSmith beats the SOTA, SPIRAL, in all settings.

Following your suggestion, we conducted additional experiments in rebuttal, covering one more model from the fourth family, MiMo-7B-Base (Xiaomi). The new experiment follows the same settings on data, token budgets and training recipe with Table 1. The results are shown in the table below.

MiMo-7B-BaseMeanMathsKnowl.LogicSpatial
Base30.015.632.748.633.7
SPIRAL69.649.252.097.288.0
Non-game67.246.150.593.886.9
GameSmith69.850.051.796.588.4

The results fully support our method. GameSmith still gives the best performance on MiMo-7B-Base.

Dear Reviewer WGGx,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response has addressed your concerns. We would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.