acceptodds
Under review as a conference paper at ICLR 2027

Explaining Chess and Go in Natural Language

Abstract

Modern chess engines play far beyond human ability, often choosing moves whose underlying ideas are difficult for humans to understand. We propose an automatic metric for evaluating chess explanations, grounded in chess-engine evaluations. Evaluating state-of-the-art language models using this metric, we find that they perform surprisingly poorly, often relying on surface-level board observations rather than robust, transferable chess concepts. We then introduce BinProp, a training-free method that enables these same models to discover reusable chess concepts and use them to explain moves in natural language. BinProp improves explainability for every model tested, nearly tripling the average score compared with direct prompting. Beyond chess, the same method discovers useful concepts in Go, draughts, and shogi without additional tuning.

Primary area:
interpretability and explainable AI
Keywords:
explainability, automatic explanations, grounded explanations, explaining chess, explaining go, chess, go

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o1ngOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a contrastive approach to generating explanations for moves in board games. It develops BinProp, an algorithm for discovering motifs that can later be integrated into natural language explanations. Evaluations show a substantial increase in explainability when integrated with frontier LLMs.

Critical strengths

The technique is remarkably simple and extracts relevant motifs cheaply and without training.

Critical weaknesses

My primary concerns with the paper are evaluation and generality.

The evaluation is somewhat circular in that the BinProp is both designed for and evaluated by its ability to contrastively select out future positional properties. The method would be more convincing if its generated explanations were evaluated by human players, following arxiv.org/abs/2212.08195. In addition, the comparison to other techniques is limited to just the Concept-Guided Chess Commentary paper.

I am also somewhat concerned about the generality of the method. It is only evaluated in unbalanced positions where there is one unique best move. For the method to be useful as analysis tool, one would prefer that it can be applied to arbitrary positions.

Questions for authors

Would a human evaluation also find that BinProp explanations are superior? Could the method be extended to general positions? How cherry-picked are the explanations in the paper?

3 comments

Dear Reviewer,

As the rebuttal period draws to a close, we wanted to ask whether you have any remaining questions or concerns we could address.

Thank you again for your time and feedback.

Reviewer_o1ngOpenReview ↗

Response to Rebuttal

I thank the authors for their detailed response. The human study is a welcome addition, and the other responses address my other concerns. I have raised my score accordingly.

A couple more points before the reviewing period ends:

The "Generalization" column label in Figure 8 appears to be garbled. Table 2 would be clearer with confidence intervals.

Thank you for updating your score. We have addressed both remaining issues and additionally added a breakdown of each chess master’s choices in Appendix F.

Should you have any further questions or concerns, we would be happy to address them.

Reviewer_oWtgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BinProp, which uses an LLM to propose reusable board features as executable functions. The system tests these features on engine-scored positions, then turns selected features into natural-language explanations of two candidate moves. These features describe board advantages rather than the rules of the game.

Critical strengths

It turns model-proposed chess concepts into executable, testable conditions and uses feedback to improve the feature set. This allows the concepts to be checked and reused across positions. BinProp improves the proposed explanation score for every tested model on chess.

Critical weaknesses
  • The natural-language explanations are not directly evaluated. The score measures function outputs, and no expert or beginner evaluation is reported. It remains unclear whether the explanations are accurate, relevant, and easy to understand.

  • Separating better and worse positions does not establish that a feature explains how a move creates an advantage. The paper should clarify which aspects of explanation quality its score measures and which it does not.

  • The paper includes a comparison with CCC, but lacks controls with matched search budgets and execution feedback. It also lacks systematic ablations, making the source of the improvement unclear.

  • The paper shows examples of useful and weak features, but provides limited analysis of why models produce them. It also does not report results by game stage or decision difficulty.

  • The test setup requires a single clearly favorable move, leaving equally reasonable choices untested. Comparing two moves is a reasonable task setting, but its benefits over other explanation formats are not evaluated.

  • With the same board information and compute budget, does generating functions before writing explanations produce better explanations than generating text directly?

Questions for authors

(lines 058-060) The paper aims to explain moves in natural language, but its evaluation scores function outputs rather than the explanations themselves. What evidence from the current experiments shows that higher scores also mean the explanations better help people understand moves?

5 comments

Dear Reviewer,

As the rebuttal period draws to a close, we wanted to ask whether you have any remaining questions or concerns we could address.

Thank you again for your time and feedback.

Thank for the response. Many parts have been addressed. I'm more interested in natural language explanation and its impact on people. I think this paper would be better if the authors had included discussions of the rebuttal stage. However, this might mean I'm biased, and it may not represent the authors' original research goal. Regarding the "analysis of why models produce them," I think it's currently only partially addressed, but I understand the difficulty. I have raised my score.

Thank you for updating your score. The revised paper includes a study with 15 titled chess masters. Masters preferred BinProp explanations over those from direct prompting and favored explanations that scored higher on our metric. Both preferences were stronger on questions where the masters agreed more often.

We additionally added analysis on why some models did better than others. Of the 20 chess concepts we checked, 17 appeared in a larger share of motifs from the top two models than from the bottom two. The top two also combined multiple concepts in 76% of their motifs, compared with 32% for the bottom two. This analysis is summarized in Section 5.3, with the full breakdown by chess concept in Appendix K.9.

Should you have any further questions or concerns, we would be happy to address them.

Reviewer_CJpcOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an engine-grounded metric for evaluating chess move explanations based on how well binary motifs distinguish favorable and unfavorable game continuations, including nearby negatives. The authors evaluate eight LLMs using this metric and report that direct prompting often produces brittle, position-specific motifs. To address this, they introduce BinProp, a training-free method that iteratively discovers reusable motifs represented as Python functions, validates them against engine evaluations, and selects relevant motifs to generate natural-language explanations. Experiments show improvements over direct prompting across eight LLMs in chess and Go, with additional results on draughts and shogi. The authors also report a correlation between the proposed explainability metric and agreement with Stockfish move preferences.

Critical strengths

S1. Clear Motivation

The paper addresses a well-motivated problem of explaining decisions made by superhuman game engines in human-understandable terms. Engine evaluations provide reliable assessments of move quality but do not directly explain the strategic concepts behind these decisions. The paper clearly identifies this gap and motivates the need for concept-based explanations.

S2. Engine-Grounded Motif Discovery

The proposed BinProp method provides a systematic approach to discovering and validating chess motifs using engine evaluations. By expressing motifs as Python functions and evaluating them on winning and losing positions, the method grounds concept discovery in objective game outcomes rather than relying solely on LLM judgments. The iterative feedback and redundancy filtering further support the discovery of reusable motifs.

S3. Generalizability Leveraging LLM Knowledge

The approach leverages the domain knowledge of LLMs to discover game-specific motifs without requiring a predefined concept library. This allows the same discovery framework to be applied across different games, including chess, Go, draughts, and shogi, without additional model training. The consistent improvements across most evaluated models and games demonstrate the flexibility of this approach.

S4. Clear Writing and Presentation

The paper is clearly written and well organized, making the motivation, proposed method, and evaluation procedure easy to follow. The figures and qualitative examples effectively illustrate the differences between direct prompting and BinProp. The appendices provide detailed experimental procedures, prompts, and discovered motif descriptions, supporting reproducibility.

Critical weaknesses

W1. Unclear explanatory faithfulness of post-hoc concept attribution BinProp primarily establishes statistical associations between engine evaluations and interpretable board properties, rather than explaining why the engine actually prefers a particular move. A concept that frequently appears in favorable positions may correlate with good outcomes without being a decisive factor in the engine's decision. The paper does not distinguish such post-hoc associations from faithful explanations of the underlying decisions. This raises a fundamental question about whether the proposed method explains engine decisions or merely describes properties associated with favorable outcomes.

W2. Limited diversity and questionable validity of nearby negatives The proposed explainability metric relies on nearby negatives to distinguish generalizable concepts from brittle ones. However, nearby negatives are generated through a narrowly defined procedure involving a late suboptimal move and the engine's best reply, with only eight such positions per test instance. It remains unclear whether concepts that perform well under this sampling procedure would generalize to more diverse counterfactuals. More fundamentally, a later mistake may lead to an unfavorable position without invalidating the strategic rationale behind the original move, raising concerns about whether these examples are appropriate counterfactuals for evaluating explanations.

W3. Insufficient calibration of the explainability metric The proposed metric is the minimum classification accuracy across three groups of eight held-out positions, but its absolute scores are not adequately calibrated against simple non-explanatory controls. For example, a predicate that independently produces random binary outputs has an expected score of approximately 35.3% under this metric, exceeding the best reported BinProp score of 29.6%, although such a predicate would not necessarily pass BinProp's discovery and selection criteria. This illustrates that the metric alone does not provide an interpretable measure of explanation quality. Additional controls using random, trivial, or deliberately spurious predicates would help establish what the reported scores actually signify.

W4. Limited evaluation dimensions and absence of human evaluation The evaluation relies primarily on a single engine-grounded explainability metric that measures how well binary predicates distinguish favorable and unfavorable positions. While this captures one aspect of generalization, it does not establish whether the explanations are comprehensive, understandable, informative, or useful to human players. The correlation with Stockfish agreement in Section 5.1 provides indirect validation but does not address these complementary dimensions. Human evaluations of explanation correctness and usefulness, or experiments measuring whether the explanations improve subsequent decision-making, would substantially strengthen the paper's claims.

Questions for authors

Q1. Scope of Superhuman Knowledge Discovery

The paper motivates its approach through the difficulty of understanding superhuman game engines. To what extent can BinProp discover concepts beyond established human chess knowledge, rather than rediscovering known concepts in executable form? A discussion of this distinction, particularly in relation to Schut et al. (2025), would help clarify the scope and potential of the proposed approach.

Q2. Naming of the Explainability Metric

The proposed explainability metric primarily measures how well a concept distinguishes favorable and unfavorable continuations, including nearby negatives. Would a term such as contrastivity or contrastive discriminability better reflect what the metric measures, while distinguishing it from broader notions of explainability?

Q3. Naming of the Paper

Given that the main technical contribution is the discovery and selection of executable concepts, while natural-language explanations are generated afterward, would the authors consider a title that more directly reflects this contribution? For example, "Explaining Chess and Go Moves through Binary Motif Discovery" might better specify the scope of the work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear Reviewers,

Thank you for your time and feedback!

The main point you raised was the need for a human evaluation. To address this, we conducted a study with 15 titled chess masters. The masters preferred BinProp explanations over those from direct prompting. They also tended to choose the explanation our metric scored higher.

We also added ablations, compared the concepts found by stronger and weaker models, and tested BinProp against Cui et al.’s strategy verbalization method. No further concerns were raised after our final response.

BinProp helps explain the decisions of black-box chess engines in terms humans can understand, without any training or fine-tuning. We hope future work can build on this approach to explain the decisions of more general AI systems.

Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.