- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the transfer of LLM chess capabilities to variants that modify the initial arrangement, piece movements, or winning conditions. It evaluates complete games against Fairy Stockfish and measures move quality throughout play. Additional experiments examine input representations, reasoning effort, opponent strength and style, and conversation persistence. The results reveal substantial performance differences between standard chess and several variants. The authors interpret these findings as evidence that memorization of the training distribution accounts for a substantial portion of LLM chess capabilities.
Critical strengths
- The paper brings together an interesting and diverse set of chess variants, including Chess960, rook-bishop, bishop-extinction, and leap-pawn. These variants modify different aspects of the game, such as the initial arrangement, piece movements, and winning conditions, providing a concrete and varied testbed for examining whether models can adapt familiar chess knowledge to altered rules.
- Evaluating complete games and tracking move quality throughout play offers a richer assessment than checking move legality alone. Running 1,000 games per setting also provides substantial empirical coverage within the selected configurations.
- The additional experiments on input representation, reasoning effort, opponent characteristics, and conversation persistence help contextualize the main findings. In particular, the limited benefits of increased reasoning effort and retained conversation history are useful observations about the evaluated models.
Critical weaknesses
- The experiments do not clearly show that the performance differences are due to memorization. The opening-to-middlegame decline may also reflect changes in positional difficulty, while weaker performance on variants may result from limited transfer of chess-specific knowledge. Without stronger controls or direct evidence about training exposure, the paper’s conclusion about the role of memorization remains insufficiently supported.
- The analysis provides limited insight into why models fail. Although the paper includes several useful comparisons, it does not sufficiently distinguish rule misunderstanding, board-state tracking errors, and strategic planning failures. The authors collected reasoning traces but leave their analysis to future work. A targeted investigation of these failure modes would make the findings more informative and provide clearer guidance for improving generalization.
Questions for authors
- Positional difficulty: How do you distinguish the effects of memorization from changes in positional difficulty, particularly when interpreting the opening-to-middlegame decline? Comparisons on difficulty-matched positions would help clarify this.
- Evidence of memorization: Could controlled continued pretraining, with separate evaluation on seen positions, unseen standard-chess positions, and held-out variants, help distinguish memorization from learning chess-specific knowledge with limited transfer?
- Input representation: Since algebraic notation provides move history while FEN provides the current board state, how do you separate notation familiarity from differences in historical information and state-reconstruction requirements?
- Failure analysis: Could you analyze the collected reasoning traces and game states to clarify whether failures arise from misunderstanding the rules, incorrectly tracking the board state, or poor strategic planning?
- Scope of conclusions: Most additional experiments focus on a single model. Can you provide further evidence that these findings extend to other models, or clarify which conclusions should be restricted to the evaluated settings?
2 comments
W2 + Q4: Investigation of Reasoning Traces
We agree that an analysis of LLM failure modes would be a useful addition to the paper, and commit to adding that in the camera-ready version.
Q2: Controlled Continued Pretraining
We agree that this would be a useful way to distinguish memorization from generalization, and commit to running a chess pretraining experiment on a small model for comparison in the camera-ready.
Q3: Notation Familiarity and History/Board-state Availability
We’ve added additional experiments in Section 4.2 to distinguish notation familiarity from historical information and state-reconstruction difficulty. These experiments are as follows.
In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 295):
We create the FEN List notation, which consists of a list of FEN strings over the entire history of the game, and compare it to a single FEN string in the current position. Average move quality over the course of play in both settings is shown in Figure 7. This results in no significant improvement over a single FEN string, suggesting that providing the move history does not drive the differences in performance observed on algebraic and FEN notation.
In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 302):
We create the FEN + Algebraic notation, which prepends the current board state in FEN to the algebraic move history. Average move quality over the course of play in both settings is shown in Figure 7. This performs roughly equivalent to the performance on algebraic notation, suggesting that having access to the board state provides no advantage.
In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 308):
Finally, we compare performance on a single FEN string to performance on an ASCII representation of the board, which is not commonly used for chess notation. Average move quality over the course of play in both settings is shown in Figure 7. This performs significantly worse throughout most of the game, further suggesting the effects of narrow generalization. All additional notations are tested on Gemini-3.8-Flash playing standard chess.
Q5: Additional Models
We have added GPT-5.6-Luna to all experiments in Sections 4.7 (Similarity of Play to Humans Does Not Affect LLM Performance) and 4.8 (Conversation Persistence Does Not Affect LLM Performance). Furthermore, we added GPT-5.6-Luna to the FEN vs Algebraic experiment in Section 4.2 (LLMs Play Slightly Better with Familiar Notation) and to the no-reasoning experiments in Section 4.6 (Reasoning Effort Does Not Significantly Affect LLM Chess Skill). We found that most findings on Gemini-3.8-Flash generalize to GPT-5.6-Luna. Finally, we have added Gemini-4-Argon to Sections 4.1 (LLMs Require Several Retries) and 4.3 (LLMs Play Worse on Chess Variants), and find its performance to be quite similar to Gemini-3.8-Flash.
We hope these revisions address your concerns, and we are happy to discuss further.
Thank you for the detailed review. We’ve uploaded a revised document with all changes marked in blue. We respectfully ask reviewers to read our responses and to let us know if there are any further questions or concerns, so that we have the opportunity to further improve the paper. We address each concern below:
W1: Memorization and Positional Difficulty
We agree that our claim regarding memorization in the original framing was too strong, and have modified our claims from asserting that models are memorizing the training data to asserting that models are narrowly generalizing from the training data, which covers a broader range of potential mechanisms. Our main edits in this regard are as follows. In the Abstract on Lines 13 and 17:
In the Introduction (Section 1, Line 40):
In the Conclusion (Section 6, Line 505):
Additionally, in Section 5, we have provided further evidence of training exposure to different variants by scanning the Dolma-3-Mix-150B-1025 dataset for chess variants. This dataset is a subset of the full Dolma-3-Mix-6T dataset with the same upsampling method, thereby making it representative of a standard pretraining dataset. The first stage of our scan was a prefilter, which identified text following any of the following three patterns: any number of numbered moves in a variety of notations, runs of at least six moves of unnumbered SAN or UCI notation, and FEN strings. The second stage of our scan classified each record according to whether its moves or positions were consistent with standard chess or some variant, as well as if a variant name was mentioned in the surrounding text. The vast majority of records were not detected as variants. The full results of the scan are as follows, as adapted from Table 1 (Line 433) of the paper.
Table: Labels of records in the
dolma3_mix-150B-1025datasetRegarding your question about positional difficulty, we agree this would be a useful addition to the paper, and commit to add an assessment of positional difficulty based on the strength of the best move(s) compared to other moves in the camera-ready. In the meantime, we have added a sentence to our limitations paragraph in Section 6 addressing this.