- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a contrastive approach to generating explanations for moves in board games. It develops BinProp, an algorithm for discovering motifs that can later be integrated into natural language explanations. Evaluations show a substantial increase in explainability when integrated with frontier LLMs.
Critical strengths
The technique is remarkably simple and extracts relevant motifs cheaply and without training.
Critical weaknesses
My primary concerns with the paper are evaluation and generality.
The evaluation is somewhat circular in that the BinProp is both designed for and evaluated by its ability to contrastively select out future positional properties. The method would be more convincing if its generated explanations were evaluated by human players, following arxiv.org/abs/2212.08195. In addition, the comparison to other techniques is limited to just the Concept-Guided Chess Commentary paper.
I am also somewhat concerned about the generality of the method. It is only evaluated in unbalanced positions where there is one unique best move. For the method to be useful as analysis tool, one would prefer that it can be applied to arbitrary positions.
Questions for authors
Would a human evaluation also find that BinProp explanations are superior? Could the method be extended to general positions? How cherry-picked are the explanations in the paper?
3 comments
Response to Rebuttal
I thank the authors for their detailed response. The human study is a welcome addition, and the other responses address my other concerns. I have raised my score accordingly.
A couple more points before the reviewing period ends:
The "Generalization" column label in Figure 8 appears to be garbled. Table 2 would be clearer with confidence intervals.
Thank you for updating your score. We have addressed both remaining issues and additionally added a breakdown of each chess master’s choices in Appendix F.
Should you have any further questions or concerns, we would be happy to address them.
Dear Reviewer,
As the rebuttal period draws to a close, we wanted to ask whether you have any remaining questions or concerns we could address.
Thank you again for your time and feedback.