- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
To interpret Go value networks, the authors extract symbolic stone patterns corresponding to QiGan—the intuitive evaluation of board positions. Their framework models network outputs via AND interactions (active when all target stones are present) and OR interactions (active when any stone is present), optimizing for sparsity while penalizing high-order rules. Recurring stone configurations (common coalitions) are assigned attribution scores and analyzed alongside professional Go players. Across 50 board states with 10 expert-selected target stones, the method produces highly sparse representations that faithfully reconstruct network evaluations.
Critical strengths
Extracting interpretable knowledge from a superhuman AI model, such as a Go-playing model, could advance both interpretability research and human learning.
The paper demonstrates that representing a single OR relation may require exponentially many AND terms, motivating the explicit inclusion of OR interactions to obtain more compact explanations.
The universal matching theorem establishes that the full interaction representation can reproduce the model’s outputs across all subsets of the selected stones, while the connection to Shapley values provides a theoretical foundation for the attribution framework.
Consultation with five professional Go players helps relate numerical interactions to meaningful Go concepts.
The empirical evaluation covers sparsity, reconstruction accuracy, interaction order, and transferability across board contexts, providing evidence beyond illustrative visual examples.
Critical weaknesses
The theorem guarantees a representation of model behaviour on a particular family of masked inputs. It does not establish that the extracted patterns are uniquely identified concepts or mechanisms actually used internally. The paper’s claims of recovering “exact inference patterns” are stronger than this guarantee supports.
The examples cover relatively few contexts. Moreover, Appendix P.2 counts an interaction as generalising when its magnitude exceeds a threshold in both contexts; this does not require preservation of its sign or numerical effect. Consequently, the metric does not fully establish consistent strategic meaning.
About writing, the text is not easy to read.
Questions for authors
What is an "interaction", "AND interaction", "OR interaction"?
Lines 044 to 046 introduce "AND interaction". From lines 079 to 082 jump out "OR interaction". Then, it seems that "AND interaction" and "OR interaction" are enough. Why? The line of arguement is not clear.