acceptodds
Under review as a conference paper at ICLR 2027

Language Models that Play Chess and Explain their Moves

Abstract

Modern chess engines are silent experts: they play at a superhuman level, but do not offer explanations for their play. On the other hand, language models (LMs) can generate plausible-sounding explanations, but their weak playing strength limits the utility of their explanations. We introduce Queen, a 4B-parameter chess-language model that can explain its moves and plans while playing at the level of a typical Grandmaster. Our novel framework enables domain-specific reasoning through complementary components: an encoder-decoder architecture and an iterative distillation algorithm. This architecture integrates a silent expert chess encoder with an instruction-tuned LM through cross-attention, which we train via a question-answering curriculum to extract chess concepts from the encoder's representations. Building on this domain-adapted model, we iteratively improve its explanations with a natural-language analog of the Bellman update: the model analyzes the positions after its top candidate moves and consolidates them into an explanation of the current position, which is then distilled back into the model. Over seven iterations, our model gains over 900 Elo points (1782 2697), substantially surpassing all frontier models on both playing strength and puzzle accuracy, despite containing three orders of magnitude fewer parameters. Furthermore, LM-based evaluations show that our explanations are fluent and approach GPT-5.6-Sol (high) in coherence. The generality of our architecture and training procedure suggests a recipe for applying language models to domains where silent expert encoders are available, like games, robotics, and computer use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model, reasoning, chess, self-improvement, domain adaptation, search

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ksZiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces QUEEN, a ~4B-parameter chess-language model meant to both play strong chess and explain its moves in natural language. The architecture couples a frozen Lc0 BT5 encoder (240M, ~2987 Elo without search) with a frozen SmolLM3-3B decoder. They are linked by Flamingo-style gated cross-attention, where encoder layer i feeds decoder layer 2i (~470M bridge parameters).

A four-stage QA curriculum (static/dynamic × current/future positions) trains only the bridge and new square/piece tokens, yielding PAWN. The model is then seeded with ~8.4K filtered GPT-5.6-Sol (low) explanations (PAWN-1). After that, it is improved over seven rounds of “natural-language Bellman” search distillation. In each round, the model expands three candidate moves, recurses into children where it errs, consolidates the child explanations with Qwen3.8-27B, filters with Stockfish, and runs SFT.

Elo rises from 1782 to 2697, measured over 32 games against 8 engines, above GPT-5.6-Sol (2071) and Gemini-3.1-Pro (2201). QUEEN also has the best no-mistake rates on 1K tactical and 1K general positions. On a GPT-5.6-Sol judge, QUEEN ties frontier models on structural coherence but scores lowest among the frontier models on conceptual coherence (2.76) and fluency (4.45). Ablations remove the encoder or the curriculum, measure disagreement with Leela’s policy, and replace the frontier seed with Stockfish-HCE templated explanations.

Critical strengths
  1. Multi-layer Flamingo cross-attention between a frozen expert encoder and a frozen LM, with a staged curriculum that reaches near-perfect board-state extraction. The architecture and decoder ablations justify the design choices.

  2. A 3B-scale LM reaching ~2700 Elo and beating frontier LMs in full games, not just isolated puzzles, is a substantial result. Evaluating on full games is a sensible choice over static best-move accuracy.

  3. Recursing into the model’s own mistakes and consolidating child analyses is a creative way to generate training data at the edge of the model’s ability.

  4. Both the encoder and the curriculum are necessary. The HCE-seeded variant shows that frontier-LM seeding is not needed for strong play.

  5. Prompts, costs, data composition, an AI-use statement, and a promised code release.

Critical weaknesses
  1. Explanation quality, the paper’s central claim, is weakly supported. QUEEN has the lowest conceptual coherence of all frontier models and the lowest fluency of all models, including C1-4B. The abstract (“approach GPT-5.6-Sol in coherence”) and the conclusion (“fluent, high-quality explanations”) overstate the results. The authors’ own showcase seem to contain conceptual errors: it calls a bishop and a knight “the bishop pair” and refers to a knight “on its starting square” on f8.

  2. The method may not distil strategic understanding. The method improves move selection and PV correctness, but nothing in the training signal rewards correct conceptual reasoning. For example, nothing teaches the model to correctly weigh a pawn sacrifice against piece activity. Consolidation explicitly adds no new content, and hallucinated motifs can propagate, as the authors acknowledge. The pattern of higher Elo with lower conceptual coherence is consistent with a stronger move predictor attaching post-hoc prose, rather than a model that verbalizes Leela’s evaluation.

  3. No faithfulness analysis. It is never tested whether the prose causally determines the move, or whether the move is decided first and the prose rationalizes it. Table 5 (disagreement with Leela on positions with ≥5 near-equal moves) does not address this: disagreement there is expected and says nothing about whether the explanation reflects the reasoning.

  4. Circular, unvalidated LM judging. GPT-5.6-Sol is the seed teacher, a baseline, and the judge, which risks self-preference bias. There is no human (e.g., titled-player) evaluation, no inter-rater agreement, and no calibration of the judge on chess concepts. The judge also receives Stockfish PVs and a blundercheck, so “structural coherence” largely re-measures substantiation.

  5. Heavy dependence on Stockfish weakens the self-improvement framing. Stockfish injects oracle moves, triggers the recursion, filters the data, and overwrites evaluations. The gains may come largely from Stockfish/search distillation rather than a natural-language Bellman update. The fact that HCE seeding starts higher than Sol seeding supports this reading.

  6. Missing strong, cheap baselines. One such baseline is a frontier or small LM given tool or engine access: Leela/Stockfish top lines, an MCTS tree with the most-visited branches, and python-chess-extracted features, then narrated. Another is SFT on Stockfish PVs with templated prose and no iteration. These are needed to justify the architecture versus a pipeline approach. Frontier models are compared with FEN-only input, while QUEEN has access to a 2987-Elo encoder.

  7. Statistical fragility. Elo is based on 32 games against opponents spanning ~900 Elo, with no confidence intervals. The non-monotone trajectory suggests noise on possibly non negligible order. NMR differences against Gemini seem within ~1–2 standard errors.

  8. Incomplete related work. Concurrent encoder–decoder approaches that integrate chess-engine representations into LMs are only briefly acknowledged. The positioning against these and other concurrent work (including OpenReview-only submissions) should be expanded.

  9. Ablations are evaluated only at PAWN-1. It is unknown whether the encoder and curriculum still matter after iterative distillation, and the coherence of the HCE variant is not reported.

Questions for authors

Q0) Clarify each of the concerns identified above.

Q1) Please reconcile PAWN-1 Elo (1014 in Table 4 vs. 1782 in Sec. 4.1/Table 6), the median-GM figure (2697 vs. 2730), and the expected win-rate computation. Can you report Elo with bootstrap or BayesElo confidence intervals, ideally from more games?

Q2) Can you provide a human evaluation (e.g., FM/IM/GM annotators) on a subset, with agreement statistics against the GPT-5.6-Sol judge? Can you report results with a non-OpenAI judge to rule out self-preference?

Q3) Faithfulness: if the prose is truncated, replaced, or perturbed before the BEST_MOVE field, how often does the move change? Is the move recoverable from the prose alone?

Q4) How does QUEEN compare to an engine-in-the-loop pipeline, in which an LM narrates Leela MCTS principal branches plus programmatically extracted features, on coherence and on cost?

Q5) What fraction of the final training data’s best moves originate from Stockfish injection or relabeling rather than the model’s own candidates? How does Elo evolve if oracle injection is disabled?

Q6) Does conceptual coherence change across iterations (PAWN-1 -> QUEEN)? If it stays flat or decreases, how do you reconcile this with the claim of improved explanation quality?

Q7) Can you evaluate whether explanations mention concepts that are verifiably present (e.g., HCE feature deltas, or probe-detectable concepts in Leela, especially in positions where the best move involves material sacrifice for positional compensation?

Q8) How is consolidation fidelity verified, i.e., that Qwen3.8-27B does not add content? What is the rate of new or unsupported claims in consolidated outputs?

Q9) What are the inference latency and cost of QUEEN compared with the frontier baselines? Figure 1 claims cost-efficiency without numbers.

Q10) Do the encoder and curriculum ablations still matter after several distillation iterations? What are the coherence scores of QUEEN (HCE)?

Q11) Why does the trajectory drop at iteration 5 (2539 -> 2434), and how was the checkpoint selected at each iteration?

9 comments
AuthorsOpenReview ↗

Response to Reviewer ksZi (1/6)

Thank you for your detailed review and for noting that recursing into the model's own mistakes and consolidating child analyses is "a creative way to generate training data at the edge of the model's ability."

Explanation quality, the paper’s central claim, is weakly supported. QUEEN has the lowest conceptual coherence of all frontier models and the lowest fluency of all models, including C1-4B.

On fluency, please see General Response P2.

Conceptual coherence is only one of three notions of coherence (Section 4.3), which is itself one of three dimensions along which we evaluate explanations. We claim our explanations are high quality based on a holistic evaluation: QUEEN is first on accuracy by 305 Elo over the strongest frontier baseline, first on all four substantiation metrics, and tied with Sol on structural coherence at 3.51. Conceptual coherence at 2.77 is the one axis on which we trail the frontier models, though we remain well ahead of C1-4B at 2.16, the only other model at our scale. Our claim that the explanations are high quality should be read as relative to prior work and frontier models, not to a human expert.

The abstract (“approach GPT-5.6-Sol in coherence”) and the conclusion (“fluent, high-quality explanations”) overstate the results.

Table 3 now reports coherence as the mean of structural coherence, conceptual coherence, and fluency. After the fluency correction, QUEEN scores 3.72, above Luna (3.57) and below Gemini (3.94) and Sol (4.04). We agree that “approaches GPT-5.6-Sol” was an inaccurate representation of our results, and we have revised the abstract and introduction to state that QUEEN exceeds GPT-5.6-Luna in coherence. We retain the description of “fluent, high-quality explanations”: after the correction, QUEEN’s fluency is 4.89, and human annotators score it 5.00 (General Response P3). Explanation quality in our framework also spans accuracy and substantiation, on which QUEEN leads all models.

The authors’ own showcase seem to contain conceptual errors: it calls a bishop and a knight “the bishop pair” and refers to a knight “on its starting square” on f8.

In the example (Appendix E.1), our model does make a few mistakes as you correctly identify. Both instances are conceptual coherence errors, which we analyse in Section 4.3: hallucinated motifs propagate through consolidation and are distilled back into the model across iterations. We agree this is a genuine limitation of our method and it is among our highest priorities for future work.

The method may not distill strategic understanding. The method improves move selection and PV correctness, but nothing in the training signal rewards correct conceptual reasoning. For example, nothing teaches the model to correctly weigh a pawn sacrifice against piece activity. Consolidation explicitly adds no new content, and hallucinated motifs can propagate, as the authors acknowledge.

We agree with all your points. We acknowledge these points in the paper. That said, the loop does improve conceptual coherence, from 2.20 at PAWN-1 to 2.77 for QUEEN (Section 4.3), though less than structural coherence (2.57 →\to 3.51), since nothing in the verifier rewards naming motifs correctly.

AuthorsOpenReview ↗

Response to Reviewer ksZi (2/6)

No faithfulness analysis. It is never tested whether the prose causally determines the move, or whether the move is decided first and the prose rationalizes it. Table 5 (disagreement with Leela on positions with ≥5\geq 5 near-equal moves) does not address this: disagreement there is expected and says nothing about whether the explanation reflects the reasoning. Q3) Faithfulness: if the prose is truncated, replaced, or perturbed before the BEST_MOVE field, how often does the move change? Is the move recoverable from the prose alone?

You are right that Table 5 does not bear on faithfulness; its purpose was narrower, testing whether the model reconstructs Leela’s policy head rather than using the representations to make its own selection, and we will state that scope explicitly.

We have run the intervention you described and added it to Section 5. On 200 random positions from the general benchmark, we intervene on QUEEN's prose before the BEST_MOVE field and record the move it then recommends, using greedy decoding:

  • Replaced: We prompt GPT-5.6-Sol (high) to rewrite the prose so that it recommends a move QUEEN had considered but discarded (the primed move). QUEEN then recommends the primed move in 100% of positions, so the prose causally determines the move rather than the move being fixed in advance.
  • Truncated: In the most extreme truncation, we remove the prose entirely and decode a move immediately after the begin-of-turn marker. QUEEN changes its recommendation in 26.5% of positions, so in over a quarter of cases it is the prose that leads to the final move.
  • Unperturbed: With its own prose, QUEEN's recommended move always matches the move its prose recommends (100%), so the move is always recoverable from, and consistent with, the prose.

If the move were decided first and the prose written afterwards to rationalize it, neither replacing nor removing the prose would change the recommendation. We therefore conclude that QUEEN uses its prose to inform its recommendation, and that the prose is not a post-hoc rationalization.

ProseOriginal move (%)Primed move (%)Other move (%)
Original100.00.00.0
Swapped0.0100.00.0
None73.53.523.0

Prose interventions on 200 general positions: the move QUEEN recommends under each prose condition.

… The pattern of higher Elo with lower conceptual coherence is consistent with a stronger move predictor attaching post-hoc prose, rather than a model that verbalizes Leela’s evaluation.

On the pattern itself, Elo and conceptual coherence in fact rise together across iterations (conceptual coherence 2.20 →\to 2.77 from PAWN-1 to QUEEN, while Elo rises 1724 →\to 2514), so the pattern is a gap relative to frontier models rather than a trade-off within our training. That remaining gap is consistent with correct analysis expressed in imprecise vocabulary, since the Appendix F rubric penalizes mislabeled motifs rather than incorrect lines, and the interventions above show that the prose is not attached post hoc.

There is no human (e.g., titled-player) evaluation, no inter-rater agreement, and no calibration of the judge on chess concepts. Q2) Can you provide a human evaluation (e.g., FM/IM/GM annotators) on a subset, with agreement statistics against the GPT-5.6-Sol judge? Can you report results with a non-OpenAI judge to rule out self-preference?

Please see P3 of the General Response, where two expert human players, blind to model identity, rate explanations using the same rubric as our LLM judge. We report both their inter-annotator agreement metrics and their agreement with the LLM judge.

Rather than adding a judge from another model family, which would exchange one model’s preferences for another’s, we address potential self-preference bias with this human study.

Circular, unvalidated LM judging. GPT-5.6-Sol is the seed teacher, a baseline, and the judge, which risks self-preference bias.

We agree the judge is not neutral, but the evidence suggests the bias runs against us rather than for us. As a baseline, GPT-5.6-Sol is scoring its own outputs, which are naturally closer to the judge's preferences than QUEEN's: GPT-5.6-Sol (low) seeded only the first of our eight training rounds, and QUEEN's explanations were thereafter shaped by search distillation. Our human study (General Response P3) is consistent with this. On the same 50 positions, the LM judge places Sol further ahead of QUEEN than human annotators do: by 0.76 versus 0.48 on conceptual coherence, and 0.24 versus 0.13 on overall coherence. Any self-preference therefore inflates Sol's scores relative to ours, making our reported coherence comparisons conservative.

AuthorsOpenReview ↗

Response to Reviewer ksZi (3/6)

The judge also receives Stockfish PVs and a blundercheck, so “structural coherence” largely re-measures substantiation.

The judge receives Stockfish reference lines because it is considerably weaker than the analyses it scores. At 2071 Elo, it cannot otherwise distinguish a sound variation from an unsound one, and every model is judged with the same information, so none is advantaged by it. We would push back on the second half. Substantiation asks whether a single predicted line is mistake-free; structural coherence asks whether the explanation considers reasonable alternatives and the critical variations that depend on the recommended move, such as the refutations of 13. e4 and 13. a3 in Appendix E.1, which substantiation never examines. The two also rank models differently: QUEEN leads Gemini by nine points of FNMR on tactical positions but trails it on structural coherence, 3.51 to 3.55. If the second measure were a restatement of the first, that inversion could not occur.

Heavy dependence on Stockfish weakens the self-improvement framing. Stockfish injects oracle moves, triggers the recursion, filters the data, and overwrites evaluations. The gains may come largely from Stockfish/search distillation rather than a natural-language Bellman update. The fact that HCE seeding starts higher than Sol seeding supports this reading.

We have replaced all mentions of "self-improvement" with the more accurate "search distillation." See General Response P4 for more details.

On the attribution, we don’t think search distillation and a Bellman update are alternatives: any value-iteration method could be described as distilling from the reward signal, and the description is true without displacing the update. Stockfish supplies the values; the Bellman analogy is to how they are propagated.

We would push back on the HCE reading. Both arms of Table 9 run the identical loop with the same Stockfish components and differ only in their seed explanations, so the comparison isolates seed quality rather than Stockfish’s role in the loop. HCE seeds start higher because they are built from engine search, whereas the GPT-5.6-Sol (low) seeds contain illegal moves and mistakes (Section 3.3). Both arms then improve substantially under the same loop, by 575 and 382 Elo from P1 to P4, which points to the loop as the source of the gains.

Missing strong, cheap baselines. One such baseline is a frontier or small LM given tool or engine access: Leela/Stockfish top lines, an MCTS tree with the most-visited branches, and python-chess-extracted features, then narrated. Q4) How does QUEEN compare to an engine-in-the-loop pipeline, in which an LM narrates Leela MCTS principal branches plus programmatically extracted features, on coherence and on cost?

We would push back on describing this system as a baseline. GPT-5.6-Sol given Stockfish's top lines is the configuration of our LLM judge, and we report it as a reference Oracle in our human evaluation (General Response P3), where it scores highest (human coherence 4.70, against 4.36 for QUEEN and 4.49 for Sol). This is expected by construction: its moves and principal variations come from Stockfish at 1M nodes, a stronger engine than Leela, so its accuracy and substantiation are those of a superhuman engine rather than of the language model, whose role reduces to narrating analysis it is handed. It therefore plays the same role as Leela in Table 1: an upper reference rather than a competing method.

For a smaller open-weight model, we give Qwen3.8-27B (roughly 7$\times$ the size of QUEEN) the top-3 Stockfish PVs at 1M nodes, ask it to explain the position in our explanation format, and score its explanations with the same GPT-5.6-Sol judge on the 200 positions of Table 3 (100 tactical, 100 general). Qwen3.8-27B is somewhat better than QUEEN on conceptual coherence (3.02 vs. 2.77), but despite having the engine's best lines in hand, it is worse on structural coherence (3.21 vs. 3.51), as well as on fluency (4.67 vs. 4.89) and mean coherence (3.63 vs. 3.72). At QUEEN's own scale, Qwen3.5-4B given the same lines trails QUEEN on every axis (structural 3.09, conceptual 2.28, fluency 4.58). Being given Stockfish's lines is therefore not enough: to explain a position, the model must still choose and illustrate the variations that show why the recommended move is best and why the alternatives fail.

ModelStructuralConceptualFluencyCoherence
QUEEN3.512.774.893.72
Qwen3.8-27B + Stockfish PVs3.213.024.673.63
Qwen3.5-4B + Stockfish PVs3.092.284.583.32

Qwen3.8-27B given the top-3 Stockfish PVs, on our tactical and general benchmarks.

See below for a discussion on cost.

Reviewer_vCt2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors contribute QUEEN, a 4B LLM that plays chess and explains its moves. The model is a Leela encoder coupled to a small LM with cross-attention. It is trained with a QA curriculum then seeded with frontier explanations and then Stockfish-filtered distilled back into the model. The model achieves strong (for an LLM) puzzle rating and has the ability to generate text that resembles chess explanations.

Critical strengths

The curriculum strategy is clearly useful and clearly demonstrated through ablating it out. The bridge types and studies are also very useful and potentially transferable. The model results in probably some currently-SOTA chess puzzle rating and model size in an LLM model. This is an interesting contribution.

Critical weaknesses

The paper is marketed as a chess model that produces explanations, and it can do that, but the explanations are low quality. The paper's own measure of conceptual coherence shows that QUEEN is lower than every frontier model tested. The examples in the paper contain obvious mistakes and hallucinations that would completely invalidate the explanations for a chess player. The only evaluation is a single LLM judge, which also happens to be the seed teacher and a baseline. There is no human evaluation, which would have caught the major problems with the explanations.

The description of "self-improvement" seemed overstated to me. Stockfish is heavily used in the methodology—which is fine, but it's not self-improvement. It filters candidates, overwrites evaluations, etc. This is more like Stockfish-flavored distillation than a Bellman update.

Questions for authors

How does the obvious missing baseline do: Leela move/PV and a frontier LLM explaining it?

How do humans rate explanation coherence and quality against strong baselines?

What if you run without the Stockfish replacement and filtering steps (i.e. can you defend the self-improvement claim)?

4 comments
AuthorsOpenReview ↗

Response to reviewer vCt2 (1/2)

Thank you for your review and for noting that our curriculum strategy is "clearly useful" and that our bridge studies are "very useful and potentially transferable."

The paper is marketed as a chess model that produces explanations,

Our aim is not to ship a system for end users; our paper’s goal is to demonstrate how a silent expert encoder can be connected with a language model using chess as a case study domain.

... and it can do that, but the explanations are low quality. The paper's own measure of conceptual coherence shows that QUEEN is lower than every frontier model tested.

Conceptual coherence is only one of three notions of coherence (Section 4.3), which is itself one of three dimensions along which we evaluate explanations. We claim our explanations are high quality based on a holistic evaluation: QUEEN is first on accuracy by 305 Elo over the strongest frontier baseline, first on all four substantiation metrics, and tied with Sol on structural coherence at 3.51. Conceptual coherence at 2.77 is the one axis on which we trail the frontier models, though we remain well ahead of C1-4B at 2.16, the only other model at our scale. Our claim that the explanations are high quality should be read as relative to prior work and frontier models, not to a human expert.

The examples in the paper contain obvious mistakes and hallucinations that would completely invalidate the explanations for a chess player.

In the example (Appendix E.1), our model does make a few mistakes as you correctly identify: it describes the knight on f8 as being on its starting square, and implies that a bishop and knight form a bishop pair. Both instances are conceptual coherence errors, which we analyse in Section 4.3: hallucinated motifs propagate through consolidation and are distilled back into the model across iterations. We agree this is a genuine limitation of our method and it is among our highest priorities for future work.

We would like to push back on “completely invalidate.” The errors are only in the descriptive prose rather than in the board state, which the model recovers reliably (Table 13). Moreover, the recommended move and the principal variation in that example are independently verifiable, and across 1,000 tactical positions our recommended move is not a mistake 91.6% of the time, against 82.5% for the strongest frontier baseline. A mislabelled piece is a real defect in the prose, but it does not void a correct recommendation supported by a mistake-free continuation.

AuthorsOpenReview ↗

Response to reviewer vCt2 (2/2)

The description of "self-improvement" seemed overstated to me. Stockfish is heavily used in the methodology—which is fine, but it's not self-improvement. It filters candidates, overwrites evaluations, etc. This is more like Stockfish-flavored distillation than a Bellman update.

We have replaced all mentions of "self-improvement" with the more accurate "search distillation." See General Response P4 for more details.

On the Bellman characterization, any value-iteration method could equally be described as distilling from the reward signal. Stockfish simply provides the values; the Bellman analogy is to how they are propagated: recursion on the nodes whose estimates are mistaken while its children’s are not, and composing the three child analyses into a single root explanation (Figure 5).

How do humans rate explanation coherence and quality against strong baselines?

Please see P3 of the General Response, where two expert human players, blind to model identity, rate explanations using the same rubric as our LLM judge. We report both their inter-annotator agreement metrics and their agreement with the LLM judge.

How does the obvious missing baseline do: Leela move/PV and a frontier LLM explaining it?

We would push back on describing this system as a baseline. GPT-5.6-Sol given Stockfish's top lines is the configuration of our LLM judge, and we report it as a reference Oracle in our human evaluation (General Response P3), where it scores highest (human coherence 4.70, against 4.36 for QUEEN and 4.49 for Sol). This is expected by construction: its moves and principal variations come from Stockfish at 1M nodes, a stronger engine than Leela, so its accuracy and substantiation are those of a superhuman engine rather than of the language model, whose role reduces to narrating analysis it is handed. It therefore plays the same role as Leela in Table 1: an upper reference rather than a competing method.

What if you run without the Stockfish replacement and filtering steps (i.e. can you defend the self-improvement claim)?

We are not defending the self-improvement claim; we agree it was inaccurate and have replaced it throughout. What we claim instead is a Bellman-style search distillation. The replacement step fires only when all three candidates are mistakes (a drop of 5% or more in win rate), and without it those positions yield no usable update, since the model cannot supply a move it does not have. PAWN-1’s explanations recommend illegal moves and mistakes (Section 3.3), so at the start of the loop the model cannot recognise its own errors, and we would expect improvement to stall near the competence of the seeded model. The mistake filtering is not separable from the consolidation: an update is only informative where the initial explanation is wrong but its successors are not, and updating from children that themselves contain mistakes would propagate those errors into training: the mechanism we identify behind our low conceptual coherence. Removing it would not isolate the Bellman update but remove the condition under which it carries any signal. The PV filter is separable, but it can only discard instances and never author an explanation; removing it would admit consolidations worse than the original. We are happy to report its rejection rate if that would be useful.

Thank you again for your time and thoughtful review. With the rebuttal period nearing its end, we would be grateful if you could take a look at our response when your schedule permits. We would be happy to clarify any remaining questions.

The paper is called "Language Models that Play Chess and Explain their Moves". In that context, I think it's fair to say that this paper is claiming to have contributed a system that explains moves well, and it's clear from the examples shown and from the results that this is not the case. Beating LLMs, which are known to not do this well, and C1, a seemingly weak baseline, is necessary but not sufficient for establishing that a move-explanation system actually works. The authors acknowledge that the hallucinating "is a genuine limitation of our method". I appreciate their candor, but think of this as a dealbreaker rather than as just a limitation. An explanation model that can't even get the pieces right, let alone the higher level concepts that educational and helpful explanations demand, is not there yet. I maintain that this "completely invalidates" the current methodology as a finished, working method. Also, in the paper and in the response, the authors repeatedly measure explanation quality by accuracy. This is a type error, especially when Stockfish is used, because the correctness of the thing being explained, and the quality of the explanation, are two different things. The work is obviously promising, but it is not mature enough for publication yet. I'm excited to see the next iteration of this research.

Reviewer_H5ooOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents QUEEN, a chess-language model that couples a frozen Lc0 BT5 encoder (240M) with SmolLM3-3B through Flamingo-style gated cross-attention. Training has two phases. First, a four-stage, programmatically verified QA curriculum trains only the bridge parameters and newly added square and piece tokens. Second, after seeding on filtered GPT-5.6-Sol explanations, the model is refined over seven rounds of iterative distillation. In each round, the model proposes three candidate moves and explains the resulting child positions. Qwen3.8-27B then consolidates these child explanations into an explanation of the root position, and the Stockfish-filtered outputs are used for supervised fine-tuning. The authors frame this procedure as a natural-language analogue of the Bellman update.

Critical strengths
  1. The paper convincingly argues that an explanation is only as useful as the move it explains. Decomposing explanation quality into accuracy, substantiation, and coherence is a reasonable and useful evaluation framework.
  2. The iterative procedure yields a gain of over 900 Elo, and the final model clearly outperforms frontier LMs in full-game play.
Critical weaknesses
  1. Stockfish is involved throughout the "self-improvement" loop, it injects best moves, filters consolidated explanations, and overwrites evaluation values. No ablation separates the contribution of the Bellman-style consolidation from that of Stockfish filtering, so it is unclear how much of the gain can be attributed to the proposed mechanism itself.

  2. The ablations are evaluated only at PAWN-1, so it remains unknown whether the gaps persist through iterative distillation.

  3. In Table 3, QUEEN scores 2.76 on conceptual coherence, compared with 3.61 for SOL, and has the lowest fluency score of all models (4.45). These numbers do not support the abstract's claim that QUEEN "approaches GPT-5.6-Sol in coherence," nor the conclusion's description of its outputs as "fluent, high-quality explanations."

  4. Inconsistent key numbers.

    • The Elo of PAWN-1 is 1014 in Table 4 but 1782 in the §4.1 trajectory and in Table 6 (P1).
    • The introduction gives the median Grandmaster rating as 2697, which is identical to QUEEN's rating, yet describes QUEEN as "approaching" it. Table 1 lists the median Grandmaster rating as 2730.

    These numbers should be carefully checked and reconciled.

  5. The cost-efficiency axis in Fig. 1 is not supported by any reported compute figures. Data generation appears substantial: roughly 400K root positions per iteration, recursive child generation, and consolidation with a 27B model. The total training and data-generation cost should be reported.

  6. The Elo trajectory is non-monotonic, rating drops from 2539 (PAWN-5) to 2434 (PAWN-6), and this regression is not discussed. Since each estimate is based on only 32 games without confidence intervals, it is unclear whether the drop reflects a genuine regression or evaluation noise.

  7. Minor issues.

    • "QUEEN-2" and "PAWN-2" are used inconsistently.
    • The size of the per-iteration training set is reported variously as 283K, 285K, and "~250K."
    • The caption of Table 3 reads "F models."
    • Appendix A contains several typos ("paris," "strea," "midtrianing").
Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer H5oo (1/2)

Thank you for your review and for noting that our paper "convincingly argues that an explanation is only as useful as the move it explains," and that our decomposition of explanation quality into accuracy, substantiation, and coherence is "a reasonable and useful evaluation framework."

The Elo trajectory is non-monotonic, rating drops from 2539 (PAWN-5) to 2434 (PAWN-6), and this regression is not discussed. Since each estimate is based on only 32 games without confidence intervals, it is unclear whether the drop reflects a genuine regression or evaluation noise.

Please see General Response P1 where we increase the number of games played by our PAWN and QUEEN models and provide confidence bounds for all reported Elo scores.

Stockfish is involved throughout the "self-improvement" loop, it injects best moves, filters consolidated explanations, and overwrites evaluation values. No ablation separates the contribution of the Bellman-style consolidation from that of Stockfish filtering, so it is unclear how much of the gain can be attributed to the proposed mechanism itself.

We have replaced all mentions of "self-improvement" with the more accurate "search distillation." See General Response P4 for more details.

Stockfish filters at a few distinct points in our search distillation algorithm:

  • If you mean the mistake filtering that identifies positions from which to perform the Bellman update, this is not separable from the consolidation. An update is only informative at positions where the initial explanation contains errors but whose successors don’t. If we performed our Bellman updates from children that contained mistakes, all errors would propagate. Ablating this filtering step would not isolate the Bellman update; it would remove the condition under which the update carries any signal at all.
  • If instead you mean the PV filter applied before training, this is separable, but it can only discard instances and cannot author an explanation. Stockfish produces no natural language, while the quantities in Tables 2 and 3 measure explanation quality. We are happy to report the filter’s rejection rate if that would be helpful.

The ablations are evaluated only at PAWN-1, so it remains unknown whether the gaps persist through iterative distillation.

This is a fair concern! On the ablations, we evaluate at PAWN-1 because what the ablations test is the initialization, and the ablated models cannot serve as one. Our loop yields a training instance only where the root analysis contains an error but all three child analyses are mistake-free (Algorithm 1, l. 21). With a first-move no-mistake rate of 9.7% without the encoder and 1.3% without the curriculum, against 73.9% for PAWN-1, the chance that all three children are mistake-free is on the order of 10⁻³ and 10⁻⁶ respectively if we treat them as independent. From a pool of 400K roots that is a few hundred usable instances in the first case and effectively none in the second — the procedure does not run and fail, it has almost nothing to run on.

AuthorsOpenReview ↗

Response to Reviewer H5oo (2/2)

In Table 3, QUEEN scores 2.76 on conceptual coherence, compared with 3.61 for SOL, and has the lowest fluency score of all models (4.45). These numbers do not support the abstract's claim that QUEEN "approaches GPT-5.6-Sol in coherence," nor the conclusion's description of its outputs as "fluent, high-quality explanations."

On fluency, please see General Response P2.

Table 3 now reports coherence as the mean of structural coherence, conceptual coherence, and fluency. After the fluency correction, QUEEN scores 3.72, above Luna (3.57) and below Gemini (3.94) and Sol (4.04). We agree that “approaches GPT-5.6-Sol” was an inaccurate representation of our results, and we have revised the abstract and introduction to state that QUEEN exceeds GPT-5.6-Luna in coherence. We retain the description of “fluent, high-quality explanations”: after the correction, QUEEN’s fluency is 4.89, and human annotators score it 5.00 (General Response P3). Explanation quality in our framework also spans accuracy and substantiation, on which QUEEN leads all models.

The cost-efficiency axis in Fig. 1 is not supported by any reported compute figures.

The cost-efficiency axis is intended to denote inference cost. We have added an inference-cost analysis to the revision, reproduced below. A full game with QUEEN requires 0.0027 H100-hours with batching across games, or roughly 0.00675 dollars at 2.5 dollars/hour of H100 compute. On the other hand, a full game with GPT-5.6-Sol (high) costs $12.97, nearly 2,000 times more.

Model$ / game ↓Rel.
QUEEN0.006751×
Gemini7.911171×
Luna12.181804×
Sol12.971921×

Inference cost per game. QUEEN is estimated at $2.5/H100-hour. Rel.: cost relative to QUEEN.

Data generation appears substantial: roughly 400K root positions per iteration, recursive child generation, and consolidation with a 27B model. The total training and data-generation cost should be reported.

Thank you for raising this point! We have revised the paper to report these costs in Appendix B.3 (Computational details), and quote them here for convenience. All experiments were conducted on H100 GPUs. The initial four-stage domain adaptation took roughly 300 H100-hours, and seeding with GPT-5.6-Sol took roughly 20 H100-hours. Data preparation for search distillation is the most computationally intensive part of our pipeline, though it is parallelizable: it took about 1,000 H100-hours per iteration, while SFT on the resulting data took around 30 H100-hours. In total, the critical path of training QUEEN took roughly 7,500 H100-hours. Prompting GPT-5.6-Sol (low) for the initial seed data cost 652.35 dollars, and API costs across all of our evaluations came to roughly $1,800.

Inconsistent key numbers.

Minor issues.

Thanks for raising these points! They have been fixed in the revision. Please note that the training set size varies by iteration, from 150K on the low end to 283K examples on the higher end. We have revised the paper to clarify this point.

Thank you again for your time and thoughtful review. With the rebuttal period nearing its end, we would be grateful if you could take a look at our response when your schedule permits. We would be happy to clarify any remaining questions.

Thanks for your detailed response. It addresses most of my concerns, and I will raise my score accordingly.

Reviewer_VydyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines Leela, which is a transformer-based encoder that plays chess optimally, with a small LLM to be able to generate natural language moves and explanations about these moves, so the system can be used to train humans and analyze the moves. The system uses both SFT and RL to train the model using a frontier LLM or feature extraction to generate the dataset for explanation. The results show that the trained model reaches the same ELO in accuracy, and the explanation judged by another frontier model makes sense.

Critical strengths

Combining these two models (a good transformer chess player and a small LLM) and training them was a challenging task that the author tackled in different ways. This combination opens the door to more explainable AI models in the future.

Critical weaknesses

The paper has a lot of different areas that could be improved:

  • Not using any user studies to evaluate the explanations makes it fall short, as the explanation is just stating the moves and the board state (like a commentator). Looking at the example given, it is equivalent to seeing the moves rolled up after each other in a video.
  • No explanation of why C1-4B did really badly (lost all games) since it was from previous work; does that align with previous work?
  • Why did we just give the model the internal representation and never test it by giving the board and the suggested moves from the encoder? Since the model plays well, we just need to explain the moves and future moves. Why are we avoiding what Leela learned as a suggestion, and why is being different from Leela a good thing?
  • The work should focus on the training and the connecting part and do the ablation study on that, not just a small part of it.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer Vydy (1/2)

Thank you for your review and noting how our methods can be used to develop “more explainable AI models in the future”. As a small clarification of the summary, QUEEN is trained entirely with supervised fine-tuning (domain adaptation followed by iterative search distillation); we do not use any RL.

Not using any user studies to evaluate the explanations makes it fall short, as the explanation is just stating the moves and the board state (like a commentator). Looking at the example given, it is equivalent to seeing the moves rolled up after each other in a video.

Regarding the example in Appendix E.1, we first clarify that explaining a move in chess necessarily involves describing relevant board features and possible continuations; these observations are not merely commentary, but the reasoning that explains why a move is good. Moreover, our model not only provides a potential sequence of moves and board states, but also constructs and evaluates counterfactual branches that illustrate why the chosen sequence of moves is preferable. In that example, the model rejects 13. e4 because 13... dxe4 14. fxe4 Ng4 hands the opponent a strong knight and immediate tactical threats, and rejects 13. a3 as a quiet flank move that does not address the central tension.

You bring up a great point regarding user studies. Please see P3 of the General Response, where two expert human players, blind to model identity, rate explanations using the same rubric as our LLM judge. We report both their inter-annotator agreement metrics and their agreement with the LLM judge.

No explanation of why C1-4B did really badly (lost all games) since it was from previous work; does that align with previous work?

C1-4B’s poor performance in our full-game evaluation is consistent with prior work, which primarily trained and evaluated models on static positions rather than sustained play across complete games; Tang et al. (2026) report no full-game results for C1-4B. While C1-4B can produce explanations alongside move predictions, the accuracy of those predictions is substantially lower than QUEEN’s (43.2% vs. 91.6% FNMR on tactical positions and 38.8% vs. 97.1% on general positions), reflecting how far our work advances beyond this earlier generation of chess language models.

AuthorsOpenReview ↗

Response to Reviewer Vydy (2/2)

Why did we just give the model the internal representation and never test it by giving the board and the suggested moves from the encoder? Since the model plays well, we just need to explain the moves and future moves.

Great question! Our encoder ablation in Table 4 tests the first half of this directly: it gives the language model the complete board state as text instead of Leela's representations, with the same domain-adaptation curriculum. This model loses all of its games (514 Elo) and avoids a mistake on only 9.7% of tactical positions, compared with 1724 Elo and 73.9% for PAWN-1 with the encoder. Hwang et al. (2025) observe a similar ceiling when supervising an LLM's move predictions with a Stockfish-derived action-value network, finding that models plateau far below expert-level play (around 1900 Elo). Together, these suggest that seeing the board, or imitating expert moves, is not enough for a language model to acquire the underlying chess understanding, which motivates our encoder–decoder architecture that directly transfers the expert's learned representations.

We explored using Leela's suggested moves as a supervision target in early experiments. Rather than learning features from Leela's representations, the cross-attention learned to reconstruct Leela's policy head and remained stuck in this local minimum. After this training, the model could not learn even the simplest task in our curriculum, identifying the piece on a given square (Stage 1), which our curriculum-trained model solves with 99.97% accuracy (Table 13). Supervising on the expert's moves teaches the bridge a shortcut to the expert's output rather than a readout of the position, which is why we instead transfer Leela's representations through the curriculum.

Why are we avoiding what Leela learned as a suggestion, and why is being different from Leela a good thing?

We are not avoiding what Leela has learned; rather, we directly leverage its learned hidden representations to provide our model with expert chess knowledge at the representation level, instead of only supervising it with Leela's suggested moves. As described above, supervising on Leela's best moves leads the bridge to simply reconstruct Leela's policy head, rather than develop its own move-selection and reasoning capabilities.

Being different from Leela is not a goal in itself; rather, it indicates that the model selects moves through its own analysis, which is what its explanation describes, instead of copying Leela's choice and attaching a post-hoc justification to a move it did not reason about. Table 5 confirms that our final model avoids this shortcut: on positions where at least five moves are within 3% win rate of the best move, it selects a move different from Leela's 53.8% of the time.

The work should focus on the training and the connecting part and do the ablation study on that, not just a small part of it.

Could you please clarify this statement? Our ablations cover both the connecting architecture and training:

Connecting architecture

  • Bridge design (Appendix A.3, Table 14): We compare our Flamingo-style gated cross-attention (~470M parameters) against a LLaVA-style prefix projection (~83M, with LoRA on the decoder) and a direct KV-cache projection (~612M). KV projection trailed both alternatives by 5–10 points on the static tasks and was dropped; Flamingo outperforms LLaVA, most clearly on the hardest stage, dynamic-future (96.07% vs. 93.20% with SmolLM3).
  • Decoder family (Appendix A.3, Table 14): With the Flamingo bridge fixed, we compare SmolLM3-3B, Qwen3-4B, and Gemma3-4B. Gemma3 is clearly weakest (79.78% on dynamic-future), and SmolLM3 is both the smallest and the most accurate on three of the four stages.
  • Encoder (Table 4): Replacing Leela with a text description of the full board, under the same curriculum, drops playing strength to 514 Elo (losing all games) and first-move accuracy on tactical positions from 73.9% to 9.7%.

Training

  • Curriculum (Table 4): Skipping domain adaptation while keeping the encoder also drops playing strength to 514 Elo, and first-move accuracy on tactical positions to 1.3%.
  • Supervision on Leela's moves: As described above, supervising the bridge on Leela's best moves leads it to reconstruct Leela's policy head, after which it cannot learn even the first stage of the curriculum.
  • Seed source (Table 9): Replacing the GPT-5.6-Sol seed explanations with templated explanations built from Stockfish's handcrafted evaluation yields a similar trajectory of improvement, showing that frontier-model seeding is not necessary.

Thank you again for your time and thoughtful review. With the rebuttal period nearing its end, we would be grateful if you could take a look at our response when your schedule permits. We would be happy to clarify any remaining questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to reviewers (1/2)

We thank all reviewers for their feedback. We address shared points below; individual points are answered in each reviewer's thread. All major changes are highlighted in blue in the revised PDF.

1. Elo confidence intervals

We compute Elo confidence intervals for all models and add them to Table 1. They are also reported below for completeness.

ModelElo95% CI
QUEEN (ours)25142486–2543
Gemini22012023–2379
Sol20711890–2246
Luna18221563–2025
C1-4B514*—

Estimated ratings anchored to Lichess Elos. QUEEN is rated over 1,024 games and all other models over 32 games. All frontier models use high reasoning effort. *C1-4B lost all 32 games.

We can only play 32 games per frontier model due to prohibitive costs. However, our intermediate PAWN checkpoints and final QUEEN model are not cost-constrained. We thus evaluate the Elo of our models over 1,024 games (64 openings × 2 sides of play × 8 opponents), substantially narrowing the 95% confidence intervals. This revises QUEEN's rating from the high-variance 32-game estimate of 2697 to 2514, and we update the abstract and introduction accordingly. Our main conclusions still hold:

  • Every iteration improves on the last except PAWN-5 → PAWN-6 (−18 Elo), a regression within overlapping confidence intervals, for a total gain of 790 Elo (1724 → 2514).
  • QUEEN is rated 305 points above Gemini and 443 points above Sol, corresponding to expected win rates of 85.3% and 92.8% respectively. The lower bound of QUEEN's interval (2486) is still over 100 points above the upper bound of Gemini's (2383).
ModelElo95% CI
PAWN-117241676–1770
PAWN-220462014–2077
PAWN-322392208–2269
PAWN-422992268–2329
PAWN-523772347–2407
PAWN-623592329–2389
PAWN-724342405–2464
QUEEN25142486–2543

Estimated ratings anchored to Lichess Elos over 1,024 games (128 per opponent, 64 opening lines).

2. Fluency Corrections

There are several concerns about the low fluency score of QUEEN. We investigated this phenomenon and discovered that although our prompt in Appendix F instructs the LM judge not to penalize the special chess tokens our model generates, the judge did not consistently follow this instruction. When we rescored the same explanations with our special tokens rewritten into plain text, our fluency score increased from 4.45 to 4.89. Our structural coherence score remained unchanged after the rewrite (3.51), and our conceptual coherence score rose by an insignificant amount, from 2.76 to 2.77. We make these changes in the revised Table 3 and also add a column reporting each model's average across structural coherence, conceptual coherence, and fluency.

AuthorsOpenReview ↗

General response to reviewers (2/2)

3. Human Evaluations

To validate our LLM-as-a-judge evaluation, two expert human players (~2200 Lichess blitz) annotated explanations from QUEEN, C1-4B, GPT-5.6-Sol, and GPT-5.6-Luna on 50 random positions from the general benchmark, using the same rubric as the judge (Appendix F). Annotators did not know which model produced each explanation. We also include the judge's own configuration (GPT-5.6-Sol given the top-3 Stockfish PVs at 1M nodes) in the comparison as a validation of the judge ("Oracle").

ModelStructural (Human)Structural (LM)Conceptual (Human)Conceptual (LM)Fluency (Human)Fluency (LM)Coherence (Human)Coherence (LM)
QUEEN4.143.343.952.585.004.944.363.62
Sol4.053.244.433.344.985.004.493.86
Luna3.582.923.993.044.975.004.183.65
C1-4B2.242.621.411.584.984.822.883.01
Oracle4.534.884.594.324.984.984.704.73

Human and LM-judge scores (1–5) on 50 random positions. Human is the mean of two annotators (≈2200 Lichess blitz), and Oracle is Sol given the top-3 Stockfish PVs at 1M nodes. Coherence is the mean of the three scores.

Humans generally place QUEEN slightly above Sol on structural coherence, and slightly below Luna on conceptual coherence. We measured a weighted Cohen's κ of 0.78 (substantial) between the two human annotators, and 0.59 (moderate, approaching substantial) between the average human score and the Sol judge. These agreement scores were calculated only on structural and conceptual coherence, as fluency was uniformly high for all models.

4. "Self-improvement" Terminology and Other Inconsistencies

We agree that "self-improvement" is not an accurate description of our algorithm, given how heavily Stockfish is used in the loop. We have replaced all instances of "self-improvement" with search distillation.

We have also fixed the following inconsistencies:

  • PAWN-1 Elo (1014 in Table 4): 1014 was our internal Elo before anchoring to Lichess blitz ratings. Table 4 now uses the correct new value, 1724.
  • Median Grandmaster rating (2697 in the introduction vs. 2730 in Table 1): 2730 is the correct Elo. However, due to the updated ratings, we are changing all occurrences of Grandmaster-level play to near International Master strength (median Lichess rating 2560).
  • Expected win rates: these are computed as 11+10−D/400\frac{1}{1 + 10^{−D/400}}, where DD is the Elo difference between two opponents.

Dear Reviewers, as the rebuttal deadline is approaching, please actively engage in the discussion based on the authors' responses. Thank You!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.