- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces QUEEN, a ~4B-parameter chess-language model meant to both play strong chess and explain its moves in natural language. The architecture couples a frozen Lc0 BT5 encoder (240M, ~2987 Elo without search) with a frozen SmolLM3-3B decoder. They are linked by Flamingo-style gated cross-attention, where encoder layer i feeds decoder layer 2i (~470M bridge parameters).
A four-stage QA curriculum (static/dynamic × current/future positions) trains only the bridge and new square/piece tokens, yielding PAWN. The model is then seeded with ~8.4K filtered GPT-5.6-Sol (low) explanations (PAWN-1). After that, it is improved over seven rounds of “natural-language Bellman” search distillation. In each round, the model expands three candidate moves, recurses into children where it errs, consolidates the child explanations with Qwen3.8-27B, filters with Stockfish, and runs SFT.
Elo rises from 1782 to 2697, measured over 32 games against 8 engines, above GPT-5.6-Sol (2071) and Gemini-3.1-Pro (2201). QUEEN also has the best no-mistake rates on 1K tactical and 1K general positions. On a GPT-5.6-Sol judge, QUEEN ties frontier models on structural coherence but scores lowest among the frontier models on conceptual coherence (2.76) and fluency (4.45). Ablations remove the encoder or the curriculum, measure disagreement with Leela’s policy, and replace the frontier seed with Stockfish-HCE templated explanations.
Critical strengths
-
Multi-layer Flamingo cross-attention between a frozen expert encoder and a frozen LM, with a staged curriculum that reaches near-perfect board-state extraction. The architecture and decoder ablations justify the design choices.
-
A 3B-scale LM reaching ~2700 Elo and beating frontier LMs in full games, not just isolated puzzles, is a substantial result. Evaluating on full games is a sensible choice over static best-move accuracy.
-
Recursing into the model’s own mistakes and consolidating child analyses is a creative way to generate training data at the edge of the model’s ability.
-
Both the encoder and the curriculum are necessary. The HCE-seeded variant shows that frontier-LM seeding is not needed for strong play.
-
Prompts, costs, data composition, an AI-use statement, and a promised code release.
Critical weaknesses
-
Explanation quality, the paper’s central claim, is weakly supported. QUEEN has the lowest conceptual coherence of all frontier models and the lowest fluency of all models, including C1-4B. The abstract (“approach GPT-5.6-Sol in coherence”) and the conclusion (“fluent, high-quality explanations”) overstate the results. The authors’ own showcase seem to contain conceptual errors: it calls a bishop and a knight “the bishop pair” and refers to a knight “on its starting square” on f8.
-
The method may not distil strategic understanding. The method improves move selection and PV correctness, but nothing in the training signal rewards correct conceptual reasoning. For example, nothing teaches the model to correctly weigh a pawn sacrifice against piece activity. Consolidation explicitly adds no new content, and hallucinated motifs can propagate, as the authors acknowledge. The pattern of higher Elo with lower conceptual coherence is consistent with a stronger move predictor attaching post-hoc prose, rather than a model that verbalizes Leela’s evaluation.
-
No faithfulness analysis. It is never tested whether the prose causally determines the move, or whether the move is decided first and the prose rationalizes it. Table 5 (disagreement with Leela on positions with ≥5 near-equal moves) does not address this: disagreement there is expected and says nothing about whether the explanation reflects the reasoning.
-
Circular, unvalidated LM judging. GPT-5.6-Sol is the seed teacher, a baseline, and the judge, which risks self-preference bias. There is no human (e.g., titled-player) evaluation, no inter-rater agreement, and no calibration of the judge on chess concepts. The judge also receives Stockfish PVs and a blundercheck, so “structural coherence” largely re-measures substantiation.
-
Heavy dependence on Stockfish weakens the self-improvement framing. Stockfish injects oracle moves, triggers the recursion, filters the data, and overwrites evaluations. The gains may come largely from Stockfish/search distillation rather than a natural-language Bellman update. The fact that HCE seeding starts higher than Sol seeding supports this reading.
-
Missing strong, cheap baselines. One such baseline is a frontier or small LM given tool or engine access: Leela/Stockfish top lines, an MCTS tree with the most-visited branches, and python-chess-extracted features, then narrated. Another is SFT on Stockfish PVs with templated prose and no iteration. These are needed to justify the architecture versus a pipeline approach. Frontier models are compared with FEN-only input, while QUEEN has access to a 2987-Elo encoder.
-
Statistical fragility. Elo is based on 32 games against opponents spanning ~900 Elo, with no confidence intervals. The non-monotone trajectory suggests noise on possibly non negligible order. NMR differences against Gemini seem within ~1–2 standard errors.
-
Incomplete related work. Concurrent encoder–decoder approaches that integrate chess-engine representations into LMs are only briefly acknowledged. The positioning against these and other concurrent work (including OpenReview-only submissions) should be expanded.
-
Ablations are evaluated only at PAWN-1. It is unknown whether the encoder and curriculum still matter after iterative distillation, and the coherence of the HCE variant is not reported.
Questions for authors
Q0) Clarify each of the concerns identified above.
Q1) Please reconcile PAWN-1 Elo (1014 in Table 4 vs. 1782 in Sec. 4.1/Table 6), the median-GM figure (2697 vs. 2730), and the expected win-rate computation. Can you report Elo with bootstrap or BayesElo confidence intervals, ideally from more games?
Q2) Can you provide a human evaluation (e.g., FM/IM/GM annotators) on a subset, with agreement statistics against the GPT-5.6-Sol judge? Can you report results with a non-OpenAI judge to rule out self-preference?
Q3) Faithfulness: if the prose is truncated, replaced, or perturbed before the BEST_MOVE field, how often does the move change? Is the move recoverable from the prose alone?
Q4) How does QUEEN compare to an engine-in-the-loop pipeline, in which an LM narrates Leela MCTS principal branches plus programmatically extracted features, on coherence and on cost?
Q5) What fraction of the final training data’s best moves originate from Stockfish injection or relabeling rather than the model’s own candidates? How does Elo evolve if oracle injection is disabled?
Q6) Does conceptual coherence change across iterations (PAWN-1 -> QUEEN)? If it stays flat or decreases, how do you reconcile this with the claim of improved explanation quality?
Q7) Can you evaluate whether explanations mention concepts that are verifiably present (e.g., HCE feature deltas, or probe-detectable concepts in Leela, especially in positions where the best move involves material sacrifice for positional compensation?
Q8) How is consolidation fidelity verified, i.e., that Qwen3.8-27B does not add content? What is the rate of new or unsupported claims in consolidated outputs?
Q9) What are the inference latency and cost of QUEEN compared with the frontier baselines? Figure 1 claims cost-efficiency without numbers.
Q10) Do the encoder and curriculum ablations still matter after several distillation iterations? What are the coherence scores of QUEEN (HCE)?
Q11) Why does the trajectory drop at iteration 5 (2539 -> 2434), and how was the checkpoint selected at each iteration?
9 comments
Response to Reviewer ksZi (2/6)
No faithfulness analysis. It is never tested whether the prose causally determines the move, or whether the move is decided first and the prose rationalizes it. Table 5 (disagreement with Leela on positions with near-equal moves) does not address this: disagreement there is expected and says nothing about whether the explanation reflects the reasoning. Q3) Faithfulness: if the prose is truncated, replaced, or perturbed before the BEST_MOVE field, how often does the move change? Is the move recoverable from the prose alone?
You are right that Table 5 does not bear on faithfulness; its purpose was narrower, testing whether the model reconstructs Leela’s policy head rather than using the representations to make its own selection, and we will state that scope explicitly.
We have run the intervention you described and added it to Section 5. On 200 random positions from the general benchmark, we intervene on QUEEN's prose before the BEST_MOVE field and record the move it then recommends, using greedy decoding:
- Replaced: We prompt GPT-5.6-Sol (high) to rewrite the prose so that it recommends a move QUEEN had considered but discarded (the primed move). QUEEN then recommends the primed move in 100% of positions, so the prose causally determines the move rather than the move being fixed in advance.
- Truncated: In the most extreme truncation, we remove the prose entirely and decode a move immediately after the begin-of-turn marker. QUEEN changes its recommendation in 26.5% of positions, so in over a quarter of cases it is the prose that leads to the final move.
- Unperturbed: With its own prose, QUEEN's recommended move always matches the move its prose recommends (100%), so the move is always recoverable from, and consistent with, the prose.
If the move were decided first and the prose written afterwards to rationalize it, neither replacing nor removing the prose would change the recommendation. We therefore conclude that QUEEN uses its prose to inform its recommendation, and that the prose is not a post-hoc rationalization.
| Prose | Original move (%) | Primed move (%) | Other move (%) |
|---|---|---|---|
| Original | 100.0 | 0.0 | 0.0 |
| Swapped | 0.0 | 100.0 | 0.0 |
| None | 73.5 | 3.5 | 23.0 |
Prose interventions on 200 general positions: the move QUEEN recommends under each prose condition.
… The pattern of higher Elo with lower conceptual coherence is consistent with a stronger move predictor attaching post-hoc prose, rather than a model that verbalizes Leela’s evaluation.
On the pattern itself, Elo and conceptual coherence in fact rise together across iterations (conceptual coherence 2.20 2.77 from PAWN-1 to QUEEN, while Elo rises 1724 2514), so the pattern is a gap relative to frontier models rather than a trade-off within our training. That remaining gap is consistent with correct analysis expressed in imprecise vocabulary, since the Appendix F rubric penalizes mislabeled motifs rather than incorrect lines, and the interventions above show that the prose is not attached post hoc.
There is no human (e.g., titled-player) evaluation, no inter-rater agreement, and no calibration of the judge on chess concepts. Q2) Can you provide a human evaluation (e.g., FM/IM/GM annotators) on a subset, with agreement statistics against the GPT-5.6-Sol judge? Can you report results with a non-OpenAI judge to rule out self-preference?
Please see P3 of the General Response, where two expert human players, blind to model identity, rate explanations using the same rubric as our LLM judge. We report both their inter-annotator agreement metrics and their agreement with the LLM judge.
Rather than adding a judge from another model family, which would exchange one model’s preferences for another’s, we address potential self-preference bias with this human study.
Circular, unvalidated LM judging. GPT-5.6-Sol is the seed teacher, a baseline, and the judge, which risks self-preference bias.
We agree the judge is not neutral, but the evidence suggests the bias runs against us rather than for us. As a baseline, GPT-5.6-Sol is scoring its own outputs, which are naturally closer to the judge's preferences than QUEEN's: GPT-5.6-Sol (low) seeded only the first of our eight training rounds, and QUEEN's explanations were thereafter shaped by search distillation. Our human study (General Response P3) is consistent with this. On the same 50 positions, the LM judge places Sol further ahead of QUEEN than human annotators do: by 0.76 versus 0.48 on conceptual coherence, and 0.24 versus 0.13 on overall coherence. Any self-preference therefore inflates Sol's scores relative to ours, making our reported coherence comparisons conservative.
Response to Reviewer ksZi (3/6)
The judge also receives Stockfish PVs and a blundercheck, so “structural coherence” largely re-measures substantiation.
The judge receives Stockfish reference lines because it is considerably weaker than the analyses it scores. At 2071 Elo, it cannot otherwise distinguish a sound variation from an unsound one, and every model is judged with the same information, so none is advantaged by it. We would push back on the second half. Substantiation asks whether a single predicted line is mistake-free; structural coherence asks whether the explanation considers reasonable alternatives and the critical variations that depend on the recommended move, such as the refutations of 13. e4 and 13. a3 in Appendix E.1, which substantiation never examines. The two also rank models differently: QUEEN leads Gemini by nine points of FNMR on tactical positions but trails it on structural coherence, 3.51 to 3.55. If the second measure were a restatement of the first, that inversion could not occur.
Heavy dependence on Stockfish weakens the self-improvement framing. Stockfish injects oracle moves, triggers the recursion, filters the data, and overwrites evaluations. The gains may come largely from Stockfish/search distillation rather than a natural-language Bellman update. The fact that HCE seeding starts higher than Sol seeding supports this reading.
We have replaced all mentions of "self-improvement" with the more accurate "search distillation." See General Response P4 for more details.
On the attribution, we don’t think search distillation and a Bellman update are alternatives: any value-iteration method could be described as distilling from the reward signal, and the description is true without displacing the update. Stockfish supplies the values; the Bellman analogy is to how they are propagated.
We would push back on the HCE reading. Both arms of Table 9 run the identical loop with the same Stockfish components and differ only in their seed explanations, so the comparison isolates seed quality rather than Stockfish’s role in the loop. HCE seeds start higher because they are built from engine search, whereas the GPT-5.6-Sol (low) seeds contain illegal moves and mistakes (Section 3.3). Both arms then improve substantially under the same loop, by 575 and 382 Elo from P1 to P4, which points to the loop as the source of the gains.
Missing strong, cheap baselines. One such baseline is a frontier or small LM given tool or engine access: Leela/Stockfish top lines, an MCTS tree with the most-visited branches, and python-chess-extracted features, then narrated. Q4) How does QUEEN compare to an engine-in-the-loop pipeline, in which an LM narrates Leela MCTS principal branches plus programmatically extracted features, on coherence and on cost?
We would push back on describing this system as a baseline. GPT-5.6-Sol given Stockfish's top lines is the configuration of our LLM judge, and we report it as a reference Oracle in our human evaluation (General Response P3), where it scores highest (human coherence 4.70, against 4.36 for QUEEN and 4.49 for Sol). This is expected by construction: its moves and principal variations come from Stockfish at 1M nodes, a stronger engine than Leela, so its accuracy and substantiation are those of a superhuman engine rather than of the language model, whose role reduces to narrating analysis it is handed. It therefore plays the same role as Leela in Table 1: an upper reference rather than a competing method.
For a smaller open-weight model, we give Qwen3.8-27B (roughly 7$\times$ the size of QUEEN) the top-3 Stockfish PVs at 1M nodes, ask it to explain the position in our explanation format, and score its explanations with the same GPT-5.6-Sol judge on the 200 positions of Table 3 (100 tactical, 100 general). Qwen3.8-27B is somewhat better than QUEEN on conceptual coherence (3.02 vs. 2.77), but despite having the engine's best lines in hand, it is worse on structural coherence (3.21 vs. 3.51), as well as on fluency (4.67 vs. 4.89) and mean coherence (3.63 vs. 3.72). At QUEEN's own scale, Qwen3.5-4B given the same lines trails QUEEN on every axis (structural 3.09, conceptual 2.28, fluency 4.58). Being given Stockfish's lines is therefore not enough: to explain a position, the model must still choose and illustrate the variations that show why the recommended move is best and why the alternatives fail.
| Model | Structural | Conceptual | Fluency | Coherence |
|---|---|---|---|---|
| QUEEN | 3.51 | 2.77 | 4.89 | 3.72 |
| Qwen3.8-27B + Stockfish PVs | 3.21 | 3.02 | 4.67 | 3.63 |
| Qwen3.5-4B + Stockfish PVs | 3.09 | 2.28 | 4.58 | 3.32 |
Qwen3.8-27B given the top-3 Stockfish PVs, on our tactical and general benchmarks.
See below for a discussion on cost.
Response to Reviewer ksZi (1/6)
Thank you for your detailed review and for noting that recursing into the model's own mistakes and consolidating child analyses is "a creative way to generate training data at the edge of the model's ability."
On fluency, please see General Response P2.
Conceptual coherence is only one of three notions of coherence (Section 4.3), which is itself one of three dimensions along which we evaluate explanations. We claim our explanations are high quality based on a holistic evaluation: QUEEN is first on accuracy by 305 Elo over the strongest frontier baseline, first on all four substantiation metrics, and tied with Sol on structural coherence at 3.51. Conceptual coherence at 2.77 is the one axis on which we trail the frontier models, though we remain well ahead of C1-4B at 2.16, the only other model at our scale. Our claim that the explanations are high quality should be read as relative to prior work and frontier models, not to a human expert.
Table 3 now reports coherence as the mean of structural coherence, conceptual coherence, and fluency. After the fluency correction, QUEEN scores 3.72, above Luna (3.57) and below Gemini (3.94) and Sol (4.04). We agree that “approaches GPT-5.6-Sol” was an inaccurate representation of our results, and we have revised the abstract and introduction to state that QUEEN exceeds GPT-5.6-Luna in coherence. We retain the description of “fluent, high-quality explanations”: after the correction, QUEEN’s fluency is 4.89, and human annotators score it 5.00 (General Response P3). Explanation quality in our framework also spans accuracy and substantiation, on which QUEEN leads all models.
In the example (Appendix E.1), our model does make a few mistakes as you correctly identify. Both instances are conceptual coherence errors, which we analyse in Section 4.3: hallucinated motifs propagate through consolidation and are distilled back into the model across iterations. We agree this is a genuine limitation of our method and it is among our highest priorities for future work.
We agree with all your points. We acknowledge these points in the paper. That said, the loop does improve conceptual coherence, from 2.20 at PAWN-1 to 2.77 for QUEEN (Section 4.3), though less than structural coherence (2.57 → 3.51), since nothing in the verifier rewards naming motifs correctly.