acceptodds
Under review as a conference paper at ICLR 2027

Comparing LLM Performance on Chess Variants

Abstract

Recently, frontier LLMs have achieved rapid capability gains on reasoning-intensive benchmarks including math and coding. However, recent work has indicated that many of those capabilities may come from memorizing expanded training data rather than general gains in reasoning ability. To assess this possibility, we use chess as a testbed and test the performance of LLMs on a variety of non-standard chess variants, which are less represented in the training data than standard chess, against the chess variant engine Fairy Stockfish. We find that standard chess performance is strong in the opening moves and degrades in the midgame, while performance on chess variants remains weak throughout the game, providing support for the claim that memorization/overfitting to the training distribution accounts for a sizable portion of the high performance on reasoning benchmarks, which is further supported by our observation that LLMs perform better when given input chess moves in algebraic notation as compared to FEN. We also test a variety of other settings and find that enabling long chain-of-thought offers only marginal performance improvement in the midgame, and play against a “human-like” opponent or a non-human-like opponent of the same skill does not significantly affect performance. Overall, LLM performance is strongest in the settings most commonly represented in training data, and weaker on less represented settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Evaluation, Out-of-Distribution Generalization, Large Language Models, Reasoning, Game Playing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bGzXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the transfer of LLM chess capabilities to variants that modify the initial arrangement, piece movements, or winning conditions. It evaluates complete games against Fairy Stockfish and measures move quality throughout play. Additional experiments examine input representations, reasoning effort, opponent strength and style, and conversation persistence. The results reveal substantial performance differences between standard chess and several variants. The authors interpret these findings as evidence that memorization of the training distribution accounts for a substantial portion of LLM chess capabilities.

Critical strengths
  1. The paper brings together an interesting and diverse set of chess variants, including Chess960, rook-bishop, bishop-extinction, and leap-pawn. These variants modify different aspects of the game, such as the initial arrangement, piece movements, and winning conditions, providing a concrete and varied testbed for examining whether models can adapt familiar chess knowledge to altered rules.
  2. Evaluating complete games and tracking move quality throughout play offers a richer assessment than checking move legality alone. Running 1,000 games per setting also provides substantial empirical coverage within the selected configurations.
  3. The additional experiments on input representation, reasoning effort, opponent characteristics, and conversation persistence help contextualize the main findings. In particular, the limited benefits of increased reasoning effort and retained conversation history are useful observations about the evaluated models.
Critical weaknesses
  1. The experiments do not clearly show that the performance differences are due to memorization. The opening-to-middlegame decline may also reflect changes in positional difficulty, while weaker performance on variants may result from limited transfer of chess-specific knowledge. Without stronger controls or direct evidence about training exposure, the paper’s conclusion about the role of memorization remains insufficiently supported.
  2. The analysis provides limited insight into why models fail. Although the paper includes several useful comparisons, it does not sufficiently distinguish rule misunderstanding, board-state tracking errors, and strategic planning failures. The authors collected reasoning traces but leave their analysis to future work. A targeted investigation of these failure modes would make the findings more informative and provide clearer guidance for improving generalization.
Questions for authors
  1. Positional difficulty: How do you distinguish the effects of memorization from changes in positional difficulty, particularly when interpreting the opening-to-middlegame decline? Comparisons on difficulty-matched positions would help clarify this.
  2. Evidence of memorization: Could controlled continued pretraining, with separate evaluation on seen positions, unseen standard-chess positions, and held-out variants, help distinguish memorization from learning chess-specific knowledge with limited transfer?
  3. Input representation: Since algebraic notation provides move history while FEN provides the current board state, how do you separate notation familiarity from differences in historical information and state-reconstruction requirements?
  4. Failure analysis: Could you analyze the collected reasoning traces and game states to clarify whether failures arise from misunderstanding the rules, incorrectly tracking the board state, or poor strategic planning?
  5. Scope of conclusions: Most additional experiments focus on a single model. Can you provide further evidence that these findings extend to other models, or clarify which conclusions should be restricted to the evaluated settings?
2 comments

Thank you for the detailed review. We’ve uploaded a revised document with all changes marked in blue. We respectfully ask reviewers to read our responses and to let us know if there are any further questions or concerns, so that we have the opportunity to further improve the paper. We address each concern below:

W1: Memorization and Positional Difficulty

We agree that our claim regarding memorization in the original framing was too strong, and have modified our claims from asserting that models are memorizing the training data to asserting that models are narrowly generalizing from the training data, which covers a broader range of potential mechanisms. Our main edits in this regard are as follows. In the Abstract on Lines 13 and 17:

However, recent work has indicated that many of those capabilities may come from narrow generalization of skills from training data rather than broad generalization of reasoning ability. [...] We find that standard chess performance is strong in the opening moves and degrades in the midgame, while performance on chess variants remains weak throughout the game, providing support for the claim that narrow generalization on the training distribution accounts for a sizable portion of the high performance on reasoning benchmarks.

In the Introduction (Section 1, Line 40):

However, recent academic work has indicated that many of those capabilities may result from narrow generalization of skills from training data rather than broad generalization of reasoning ability.

In the Conclusion (Section 6, Line 505):

This indicates that, at least in the domain of chess, a large share of the measured gains in LLM reasoning capability are due to narrow generalization in specific domains rather than broad generalization across reasoning.

Additionally, in Section 5, we have provided further evidence of training exposure to different variants by scanning the Dolma-3-Mix-150B-1025 dataset for chess variants. This dataset is a subset of the full Dolma-3-Mix-6T dataset with the same upsampling method, thereby making it representative of a standard pretraining dataset. The first stage of our scan was a prefilter, which identified text following any of the following three patterns: any number of numbered moves in a variety of notations, runs of at least six moves of unnumbered SAN or UCI notation, and FEN strings. The second stage of our scan classified each record according to whether its moves or positions were consistent with standard chess or some variant, as well as if a variant name was mentioned in the surrounding text. The vast majority of records were not detected as variants. The full results of the scan are as follows, as adapted from Table 1 (Line 433) of the paper.

Table: Labels of records in the dolma3_mix-150B-1025 dataset

Text label
Moves labelNoneStandardArmageddonChess960OtherOut of scopeTotal
Move records
Standard29,127831036576029,454
Chess96050013900144
Rook-bishop0000000
Bishop-extinction0000000
Leap-pawn0000000
Other variant30008011
Uncategorized74,001025839184774,489
Total103,136833612432687104,098
FEN positions
Standard7,83930341807,894
Chess9604003007
Rook-bishop0000000
Bishop-extinction0000000
Leap-pawn0000000
Other variant0000000
Uncategorized46300060469
Total8,30630372408,370

Regarding your question about positional difficulty, we agree this would be a useful addition to the paper, and commit to add an assessment of positional difficulty based on the strength of the best move(s) compared to other moves in the camera-ready. In the meantime, we have added a sentence to our limitations paragraph in Section 6 addressing this.

W2 + Q4: Investigation of Reasoning Traces

We agree that an analysis of LLM failure modes would be a useful addition to the paper, and commit to adding that in the camera-ready version.

Q2: Controlled Continued Pretraining

We agree that this would be a useful way to distinguish memorization from generalization, and commit to running a chess pretraining experiment on a small model for comparison in the camera-ready.

Q3: Notation Familiarity and History/Board-state Availability

We’ve added additional experiments in Section 4.2 to distinguish notation familiarity from historical information and state-reconstruction difficulty. These experiments are as follows.

In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 295):

We create the FEN List notation, which consists of a list of FEN strings over the entire history of the game, and compare it to a single FEN string in the current position. Average move quality over the course of play in both settings is shown in Figure 7. This results in no significant improvement over a single FEN string, suggesting that providing the move history does not drive the differences in performance observed on algebraic and FEN notation.

In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 302):

We create the FEN + Algebraic notation, which prepends the current board state in FEN to the algebraic move history. Average move quality over the course of play in both settings is shown in Figure 7. This performs roughly equivalent to the performance on algebraic notation, suggesting that having access to the board state provides no advantage.

In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 308):

Finally, we compare performance on a single FEN string to performance on an ASCII representation of the board, which is not commonly used for chess notation. Average move quality over the course of play in both settings is shown in Figure 7. This performs significantly worse throughout most of the game, further suggesting the effects of narrow generalization. All additional notations are tested on Gemini-3.8-Flash playing standard chess.

Q5: Additional Models

We have added GPT-5.6-Luna to all experiments in Sections 4.7 (Similarity of Play to Humans Does Not Affect LLM Performance) and 4.8 (Conversation Persistence Does Not Affect LLM Performance). Furthermore, we added GPT-5.6-Luna to the FEN vs Algebraic experiment in Section 4.2 (LLMs Play Slightly Better with Familiar Notation) and to the no-reasoning experiments in Section 4.6 (Reasoning Effort Does Not Significantly Affect LLM Chess Skill). We found that most findings on Gemini-3.8-Flash generalize to GPT-5.6-Luna. Finally, we have added Gemini-4-Argon to Sections 4.1 (LLMs Require Several Retries) and 4.3 (LLMs Play Worse on Chess Variants), and find its performance to be quite similar to Gemini-3.8-Flash.

We hope these revisions address your concerns, and we are happy to discuss further.

Reviewer_TezaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper reports on LLMs playing chess variants, both real and invented. The results show that LLMs play best in standard positions and that various factors matter a bit to not at all. The claim is that this reflects memorization.

Critical strengths

The variety of variants, including invented ones, is logical and could in principle distinguish between competing plausible hypotheses behind what's going on.

Critical weaknesses

First, the main measure seems inappropriate for the study and could be confounding all of the results. Lichess's centipawns to win% function was calibrated on human games in standard chess, and it's completely unclear that they would transfer to the settings studied in this paper. Second, the games are all against near-perfect engines and are all losses, so the signal is very weak. The metric being incomparable across variants invalidates the comparisons across variants that they answers try to make, and thus the conclusions and interpretations don't have a sound basis.

Questions for authors

What do the variant curves look like for a reference agent (i.e. non-LLM) of matched playing strength judged by the same metric? What do your results look like if you fix the measurement problem?

2 comments

Thank you for the detailed review. We’ve uploaded a revised document with all changes marked in blue. We respectfully ask reviewers to read our responses and to let us know if there are any further questions or concerns, so that we have the opportunity to further improve the paper. We address each concern below:

W1 + Q2: Metric Comparability

Adapting the methodology used for the calibration of the Stockfish WDL model, we’ve recalibrated our scoring metric on Fairy Stockfish self-play separately for each variant, and rescored each experiment using these new metrics (Section 3.5, Appendix F). A description of our model and coefficients are stated below, with the model described in Appendix F on Line 917 and the coefficient table also found as Table 3 (Line 919) in the paper. Finally, we tested these fits on an additional 200 held-out Fairy Stockfish self-play games, and found they were well-calibrated, with calibration curves provided in Figure 6 (Line 950).

[W]e adapt Stockfish’s Win/Draw/Loss (WDL) model pipeline to derive win probabilities (The Stockfish developers, 2026). For each variant, we ran 1800 self-play games using Fairy Stockfish at depth 16. The opening moves of each game were chosen randomly to avoid determinism, and the resulting positions were filtered to include those of equivalent balance to Unbalanced Human Openings book (The Stockfish developers, 2023), which is standard in Stockfish WDL calibration. We then fit the following model to each variant, where mm represents the material on the board according to the formula 9Q+5R+3B+3N+P9Q+5R+3B+3N+P, a(m)a(m) and b(m)b(m) are cubics in m/58m/58, and WW, DD, and LL are the probabilities of win, draw, and loss respectively. D=1−W−LD = 1 - W - L W=11+exp⁡(−c−a(m)b(m))W = \frac{1}{1 + \exp(-\frac{c-a(m)}{b(m)})} L=11+exp⁡(−−c−a(m)b(m))L = \frac{1}{1 + \exp(-\frac{-c-a(m)}{b(m)})}

Coefficients of the fitted per-condition WDL models.

Conditiona3a_3a2a_2a1a_1a0a_0b3b_3b2b_2b1b_1b0b_0
Standard51.39−118.0759.94143.95125.62−297.05225.36−4.53
Chess960−70.68180.03−182.33217.6631.24−41.275.5460.78
Rook-Bishop−14.3849.04−84.85234.14103.61−206.15143.5744.30
Bishop-Extinction−763.002431.43−2677.521242.80−156.17508.28−536.15282.25
Leap-Pawn−303.96849.77−748.89279.07−50.91159.72−104.8879.55

After rescoring, we find that our original observations hold, that LLMs do indeed perform worse on chess variants than on standard chess. However, many of our findings with regard to LLM underperformance at specific sections of the game have changed. Specifically, our updated findings are as follows. In LLMs Play Worse on Chess Variants (Section 4.3, Line 312):

In Chess960, all models [...] remain worse than standard chess in the midgame. In LLMs Play Worse on Chess Variants (Section 4.3, Line 322): GPT-5.6-Luna, however, retains the same quality of moves [in bishop extinction] as in standard chess while still having shorter games. In LLMs Play Worse on Chess Variants (Section 4.3, Line 328): In rook-bishop, the underperformance returns: all models begin with a similar move quality compared to standard chess, but quickly perform much worse by the late opening.

W2: Games Against Non-Perfect Engines

In LLMs Fail to Broadly Generalize Against Imperfect Engines (Section 4.4, Line 360):

In order to verify that move quality remains worse on variants against an imperfect opponent as well as a strong opponent, we use the Mixed Elo 1600 engine, calibrated with the probability specified in Appendix E as an opponent for Gemini-3.8-Flash. While the performance of the LLM against a weaker opponent improves overall, the relative performance on variants remains similar. The Elo 1600 Mixed engine plays randomly with probability 0.166 and as Fairy Stockfish otherwise.

Q1: Reference Agent

In Classical Engines Do Generalize Across Variants (Section 4.5, Line 367):

In order to verify that it is in principle possible to broadly generalize across variants, we test the performance of the Mixed Elo 1600 engine with the probability calibrated in Appendix E against Fairy Stockfish. The Mixed agent is able to successfully maintain high move quality across all variants, while the LLM loses significant move quality on the variants.

We hope these revisions address your concerns, and we are happy to discuss further.

Thanks for your response and updates. I appreciate the metric update, although how much the results changed gives me a bit of pause for how robust the rest of the results are.

The other changes aren't very convincing to me, since the "Mixed Elo 1600 engine" is so artificial (part random part strong engine).

i don't think the updates are enough to raise my score, unfortunately.

Reviewer_RHJ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether LLM performance in chess comes from memorizing familiar patterns or from general reasoning ability. The authors compare standard chess with Chess960 and three less common variants, using Fairy Stockfish to evaluate move quality. The main finding is that LLMs play relatively well in standard chess openings, but their performance drops in the midgame and on unfamiliar variants. The paper also tests different input formats, reasoning levels, and opponent types.

Critical strengths
  1. I think using chess variants to study memorization versus generalization is an interesting idea. The paper compares standard chess with Chess960 and 3 less common variants, which makes the comparison meaningful.
  2. The results in Figure 1 are interesting. The models perform well in standard chess openings but become weaker in the midgame, while performance on most variants is worse.
  3. I also like that the authors tested other factors, such as algebraic notation vs. FEN (Section 4.2), reasoning effort (Section 4.4), and different opponents (Section 4.5).
Critical weaknesses
  1. My main concern is the conclusion that "a large share of gains in LLM reasoning are due to memorization" (Section 5). The experiments show that models perform worse on unfamiliar variants, but I don't think this is enough to conclude that memorization is the main reason. The models might simply struggle with unfamiliar rules or planning.
  2. In Section 4.2, the authors find that models perform slightly better with algebraic notation than FEN. However, algebraic notation contains the move history, while FEN only provides the current board state. I'm not sure this is a fair comparison.
  3. The experiments are limited to a few models, and several settings are only tested on Gemini-3.8-Flash. As the authors acknowledge in Section 5, they did not test the largest frontier models.
  4. I would also like to see more analysis of the mistakes made by the models. For example, Section 4.1 reports high disqualification rates before retries, but it is unclear how many failures come from misunderstanding chess rules versus output formatting issues.
Questions for authors
  1. How do you distinguish memorization from difficulty reasoning about unfamiliar rules? Could the performance drop have other explanations?
  2. For the FEN experiment, could the lack of move history explain the difference in performance?
  3. Have you looked at the types of mistakes models make in different variants? In particular, how often do they misunderstand the rules versus choose strategically poor moves?
  4. How sensitive are the results to the Fairy Stockfish evaluation settings and the conversion to win probabilities?
  5. Do you think the conclusions would still hold for larger or more recent models?
  6. Since games in different variants may end at different move numbers, could survivorship bias affect the later parts of the curves in Figure 1? It would be helpful to see the number of remaining games at each move.
2 comments

Thank you for the detailed review. We’ve uploaded a revised document with all changes marked in blue. We respectfully ask reviewers to read our responses and to let us know if there are any further questions or concerns, so that we have the opportunity to further improve the paper. We address each concern below:

W1 + Q1: Memorization Framing

We agree that our claim regarding memorization in the original framing was too strong, and have modified our claims from asserting that models are memorizing the training data to asserting that models are narrowly generalizing from the training data, which covers a broader range of potential mechanisms. Our main edits in this regard are as follows. In the Abstract on Lines 13 and 17:

However, recent work has indicated that many of those capabilities may come from narrow generalization of skills from training data rather than broad generalization of reasoning ability. [...] We find that standard chess performance is strong in the opening moves and degrades in the midgame, while performance on chess variants remains weak throughout the game, providing support for the claim that narrow generalization on the training distribution accounts for a sizable portion of the high performance on reasoning benchmarks.

In the Introduction (Section 1, Line 40):

However, recent academic work has indicated that many of those capabilities may result from narrow generalization of skills from training data rather than broad generalization of reasoning ability.

In the Conclusion (Section 6, Line 505):

This indicates that, at least in the domain of chess, a large share of the measured gains in LLM reasoning capability are due to narrow generalization in specific domains rather than broad generalization across reasoning.

Additionally, in Section 5, we have provided further evidence of training exposure to different variants by scanning the Dolma-3-Mix-150B-1025 dataset for chess variants. This dataset is a subset of the full Dolma-3-Mix-6T dataset with the same upsampling method, thereby making it representative of a standard pretraining dataset. The first stage of our scan was a prefilter, which identified text following any of the following three patterns: any number of numbered moves in a variety of notations, runs of at least six moves of unnumbered SAN or UCI notation, and FEN strings. The second stage of our scan classified each record according to whether its moves or positions were consistent with standard chess or some variant, as well as if a variant name was mentioned in the surrounding text. The vast majority of records were not detected as variants. The full results of the scan are as follows, as adapted from Table 1 (Line 433) of the paper.

Table: Labels of records in the dolma3_mix-150B-1025 dataset

Text label
Moves labelNoneStandardArmageddonChess960OtherOut of scopeTotal
Move records
Standard29,127831036576029,454
Chess96050013900144
Rook-bishop0000000
Bishop-extinction0000000
Leap-pawn0000000
Other variant30008011
Uncategorized74,001025839184774,489
Total103,136833612432687104,098
FEN positions
Standard7,83930341807,894
Chess9604003007
Rook-bishop0000000
Bishop-extinction0000000
Leap-pawn0000000
Other variant0000000
Uncategorized46300060469
Total8,30630372408,370

W2 + Q2: Notation and Move History Availability

We’ve added additional experiments in Section 4.2 to distinguish notation familiarity from move history and board-state availability. These experiments are as follows.

In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 295):

We create the FEN List notation, which consists of a list of FEN strings over the entire history of the game, and compare it to a single FEN string in the current position. Average move quality over the course of play in both settings is shown in Figure 7. This results in no significant improvement over a single FEN string, suggesting that providing the move history does not drive the differences in performance observed on algebraic and FEN notation.

In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 302):

We create the FEN + Algebraic notation, which prepends the current board state in FEN to the algebraic move history. Average move quality over the course of play in both settings is shown in Figure 7. This performs roughly equivalent to the performance on algebraic notation, suggesting that having access to the board state provides no advantage.

In LLMs Play Slightly Better with Familiar Notation (Section 4.2, Line 308):

Finally, we compare performance on a single FEN string to performance on an ASCII representation of the board, which is not commonly used for chess notation. Average move quality over the course of play in both settings is shown in Figure 7. This performs significantly worse throughout most of the game, further suggesting the effects of narrow generalization. All notations are tested on Gemini-3.8-Flash playing standard chess.

W3 + Q5: Additional Models

In order to test the latest frontier models, we have added Gemini-4-Argon to Sections 4.1 (LLMs Require Several Retries) and 4.3 (LLMs Play Worse on Chess Variants), and find its performance to be quite similar to Gemini-3.8-Flash. Additionally, we have added GPT-5.6-Luna to all experiments in Sections 4.7 (Similarity of Play to Humans Does Not Affect LLM Performance) and 4.8 (Conversation Persistence Does Not Affect LLM Performance). Furthermore, we added GPT-5.6-Luna to the FEN vs Algebraic experiment in Section 4.2 (LLMs Play Slightly Better with Familiar Notation) and to the no-reasoning experiments in Section 4.6 (Reasoning Effort Does Not Significantly Affect LLM Chess Skill). We found that most findings on Gemini-3.8-Flash generalize to GPT-5.6-Luna.

W4 + Q3: Analysis of Model Mistakes

We agree that an analysis of LLM mistakes would be a useful addition to the paper, and commit to adding that in the camera-ready version.

Q4: Result Sensitivity

We agree this is a concern, and commit to measuring the robustness of our metric to changes in evaluation depth and different fits of win probability parameters for the camera-ready. In the meantime, we have added a sentence to our limitations paragraph in Section 6 addressing this.

Q6: Survivorship Bias

We agree this could affect results, and have added survivorship curves to Figure 1.

We hope these revisions address your concerns, and we are happy to discuss further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their clear and constructive feedback. We were happy to see that reviewers found the variant selection meaningful (bGzX, Teza, RHJ7), and that the additional experiments were useful for contextualizing the main results (bGzX, RHJ7). We’ve uploaded a revised document with all changes marked in blue. We respectfully ask reviewers to read our responses and to let us know if there are any further questions or concerns, so that we have the opportunity to further improve the paper. The main concerns and our responses are as follows.

  1. Memorization framing (bGzX, RHJ7). We’ve added an analysis of the Dolma-3-Mix-150B-1025 training data to verify that chess variants have low prevalence in the training data (Section 5). We agree that our claim regarding memorization in the original framing was too strong, and have modified our claims from asserting that models are memorizing the training data to asserting that models are narrowly generalizing from the training data, which covers a broader range of potential mechanisms.
  2. Metric validity (Teza, RHJ7). We’ve recalibrated our scoring metric on Fairy Stockfish self-play separately for each variant, and rescored each experiment using these new metrics (Section 3.5, Appendix F).
  3. Comparison of additional notations (bGzX, RHJ7). We’ve added additional notation experiments to test the effects of move history and board state availability (Section 4.2). Those experiments are as follows: (a) FEN List, where we provide the history of board positions as FEN strings; (b) FEN + Algebraic, where we prepend the current FEN state to the Algebraic move history; and (c) ASCII, where we present the current board state in a non-standardized ASCII format. We find that FEN List performs similarly to FEN, FEN + Algebraic performs similarly to Algebraic, and ASCII performs worse than either.
  4. Additional models (bGzX, RHJ7). We’ve added GPT-5.6-Luna to Sections 4.2, 4.6, 4.7, and 4.8. We’ve also added Gemini-4-Argon to Sections 4.1 and 4.3. Our findings generalize to the new models as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.