- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the use of self-play on two-player zero sum games as a way to transfer learning to other tasks. This is done by generating new variants of games from an earlier work, SPIRAL, and designing improved baselines. Experiment shows improved performance over SPIRAL.
Critical strengths
- The work is competently done, with well designed templates, and improved baselines which can serve as references for further work.
- The experiment results are positive, validating the earlier work, and improving upon it.
Critical weaknesses
Novelty is not high, mostly showing that results can be further improved with more data and better baseline estimation. The baseline estimation is improved from SPIRAL but the method is also not highly novel.
Questions for authors
- What are the new insights from this work, beyond what was shown in SPIRAL?
- (From AI review) Table 2 reports 64.9 for Qwen3-4B-Base with CRAB. However, Figure 5a, explicitly described as showing the corresponding runs, appears to peak near 64.2, around step 140. Table 22 identifies step 120 as the selected CRAB checkpoint, whose plotted score appears closer to 63.9. Can the authors explain the discrepancy?
4 comments
Dear Reviewer dFre,
Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response has addressed your concerns. We would be glad to answer any further questions.
Thanks for the rebuttal. I still have novelty concerns and retain my rating.
Thank you for your reply, and we take it that the question on Figure 5 has been resolved and the remaining concern is novelty, on which we respectfully hold a different view. We thank you for your review and it has helped us improve the clarity of the paper.
Response to Reviewer dFre
We appreciate the reviewer for the thorough review and constructive feedback. All comments are addressed below. The manuscript has been updated accordingly. We sincerely hope our responses can be taken into account in re-evaluating the merit of our paper. Much appreciated.
Q1
A1: We respectfully disagree. Scaling is central in the recent success of AI, which is technically not easy. Our paper provides an efficient scaling approach in a new and important direction.
SPIRAL manually designs three games as supply to self-play. The paper acknowledged two key problems in scaling the paradigm: (1) designing games needs significant human labour, and (2) post-training is becoming unstable when game number grows.
Our paper solves the two problems.
We automate the supply of games via a carefully designed synthesis method for generating two-player zero-sum games without human labour. Each game is verified with replay tests and an independent re-implementation (Appendix A.3), so its rewards can be trusted. In our experiments, we created 26 new games in 29 families. Appendix A.4.3 now shows nine of them as game cards.
In addition, our CRAB makes self-play stable at scale. It shrinks the leave-one-out baseline towards a prior shared across the corpus. Lemma 1 shows that gradient variance is controlled by CRAB, whose performance never falls more than 0.4 points below its start.
Our work also provides new insights that could generalise to new research:
Diversity of games matters. At a fixed number of trajectories, going from 4 to 23 families raises the mean from 59.1 to 63.5 (Figure 3). Adding rule, size and encoding variants of SPIRAL's three games at the same budget lowers it (57.7 vs. 60.6, see our reply to WGGx, A1.2).
Games without annotation can beat annotated reasoning. GameSmith beats distillation from eight annotated, answer-filtered datasets on three of four models (Table 1).
Q2
A2: Thanks and addressed. Figure 5a plots a centred three-point moving average. Step 120 scores 64.9 (Table 2) and is plotted as the mean of steps 100, 120 and 140. The plotted peak at step 140 (64.2) is likewise the mean of steps 120, 140 and 160. We have now clarified this in the caption of Figure 5 and plotted the overlaid raw scores. The results are still unchanged.