acceptodds
Under review as a conference paper at ICLR 2027

Mining and Quantifying Knowledge Learned by Go Models via Harsanyi Interactions

Abstract

Given a deep neural network (DNN) that has surpassed human beings in a task, disentangling the explicit knowledge encoded by the DNN to gain new insights into the task is a promising yet challenging regime in explainable AI. In this paper, we aim to disentangle the "QiGan" encoded by the DNN for Go, which has beaten top human players. Specifically, we disentangle primitive shape patterns of stones memorized by the value network, and these shape patterns represent the "QiGan" used to conduct a fast situation assessment of the current board state. To this end, we propose to use both AND interactions and OR interactions to obtain a more concise explanation of shape patterns. We further prove the universal matching property of AND-OR interactions to ensure the faithfulness and verifiability of the explained shape patterns. In experiments, our method finds many novel shape patterns beyond the traditional shape patterns in human knowledge.

Primary area:
interpretability and explainable AI
Keywords:
Knowledge Discovery, Interpretability, Game AI

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_jsEtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To interpret Go value networks, the authors extract symbolic stone patterns corresponding to QiGan—the intuitive evaluation of board positions. Their framework models network outputs via AND interactions (active when all target stones are present) and OR interactions (active when any stone is present), optimizing for sparsity while penalizing high-order rules. Recurring stone configurations (common coalitions) are assigned attribution scores and analyzed alongside professional Go players. Across 50 board states with 10 expert-selected target stones, the method produces highly sparse representations that faithfully reconstruct network evaluations.

Critical strengths

Extracting interpretable knowledge from a superhuman AI model, such as a Go-playing model, could advance both interpretability research and human learning.

The paper demonstrates that representing a single OR relation may require exponentially many AND terms, motivating the explicit inclusion of OR interactions to obtain more compact explanations.

The universal matching theorem establishes that the full interaction representation can reproduce the model’s outputs across all subsets of the selected stones, while the connection to Shapley values provides a theoretical foundation for the attribution framework.

Consultation with five professional Go players helps relate numerical interactions to meaningful Go concepts.

The empirical evaluation covers sparsity, reconstruction accuracy, interaction order, and transferability across board contexts, providing evidence beyond illustrative visual examples.

Critical weaknesses

The theorem guarantees a representation of model behaviour on a particular family of masked inputs. It does not establish that the extracted patterns are uniquely identified concepts or mechanisms actually used internally. The paper’s claims of recovering “exact inference patterns” are stronger than this guarantee supports.

The examples cover relatively few contexts. Moreover, Appendix P.2 counts an interaction as generalising when its magnitude exceeds a threshold in both contexts; this does not require preservation of its sign or numerical effect. Consequently, the metric does not fully establish consistent strategic meaning.

About writing, the text is not easy to read.

Questions for authors

What is an "interaction", "AND interaction", "OR interaction"?

Lines 044 to 046 introduce "AND interaction". From lines 079 to 082 jump out "OR interaction". Then, it seems that "AND interaction" and "OR interaction" are enough. Why? The line of arguement is not clear.

Reviewer_eGAmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the knowledge encoded by the value network of a Go AI, aiming to extract interpretable shape patterns referred to as “QiGan.” The authors extend AND interactions to include OR interactions, allowing a more concise decomposition of the network’s outputs. They establish a universal matching property for the AND–OR representation and introduce additional penalties on high-order interactions to encourage simpler and more transferable patterns. Common coalitions shared across salient interactions are then identified and assigned attribution scores. Experiments on KataGo examine the fidelity, sparsity, and transferability of the extracted interactions. Professional Go players interpret selected patterns, identifying both agreements and conflicts with conventional human intuition.

Critical strengths
  • The research question is interesting. AND–OR interactions provide an intuitive way to summarize the knowledge encoded by the model.

  • The paper presents interesting observations, particularly patterns that conflict with professional players’ intuition.

Critical weaknesses
  • The extracted “QiGan” lacks sufficient validation. Matching the value network’s outputs does not necessarily indicate that the identified patterns represent correct Go knowledge. For cases that disagree with human intuition, it would be helpful to provide further search or game continuations to verify the judgments and explain the underlying reasons.

  • The generalization results are not convincing enough. The metric in Appendix P.2 only checks whether interactions remain salient across contexts, rather than whether their signs and magnitudes are consistent. It is unclear whether the extracted patterns remain useful beyond the specific positions analyzed.

  • The criteria for selecting the ten stones are unclear. How does the professional player determine which stones are representative? Do they form a complete local tactical configuration? It would be helpful to provide specific selection rules and examine the sensitivity of the results to different selections.

Questions for authors

Are there any potential methods to validate the newly discovered Qigan and transfer them into knowledge that humans, or AI models can use.

Reviewer_yVQ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an 'AND-OR' interaction framework for interpreting the knowledge encoded in the value network of a Go game model. It extends existing explanations based on Harsanyi interactions by introducing 'OR' interactions, which aim to represent alternative patterns more concisely than 'AND' interactions alone. The method also introduces a regularisation strategy designed to reduce the influence of higher-order interactions and identifies common coalitions of stones in order to summarise recurring patterns.

Critical strengths
  1. The problem is interesting and well-motivated: it concerns extracting explicit rules from a superhuman network. Moreover, involving professional players to interpret the results is a key strength.

  2. The theoretical justification of the OR extension is clear: Thm. 2.2 shows that an OR over m variables requires 2^m - 1 AND interactions, and Thms. E.1 and I.1 link the framework to Shapley. The ablation on the high-order penalty is sound (Fig. 4, 11).

Critical weaknesses
  1. The paper claims to extract genuine properties of the network, but the fidelity evidence largely stems from construction rather than proving these patterns aren't artifacts of the optimizer. Specifically, Thm. 2.3 provides an exact identity for any weights (App. G), meaning g matches v across masked states regardless of what the network actually learns. Since the fit relies on a slack parameter (|ε_T| < η = 0.38) and an ad-hoc correction a_k (Eq. 15), achieving an R² of 0.990–0.997 (Fig. 5) doesn't rigorously test true fidelity. Furthermore, no lower-order baseline (e.g., order ≤ 2) is provided, η is tuned directly from data without a proper sensitivity analysis, and the text leaves ambiguous whether Fig. 5 compares g to v or to v + ε_T. Additionally, the AND-OR decomposition lacks uniqueness due to free parameters under an L1 objective. It remains unclear whether the AND-only baseline in Fig. 2 uses identical optimization settings, and comprehensive stability checks across random seeds, η, τ, or r are missing (robustness to stone selection is merely claimed in App. Q without supporting data). Finally, because interactions rely on 10 hand-picked stones with a frozen background, and isolating salient patterns still requires manual filtering, claims about what the value network broadly "knows" are overstated.

  2. The discovery of novel Go knowledge is insufficiently disentangled from masking artifacts, baseline choices, or value-network errors. Masked boards likely push inputs out of distribution, yet the paper omits crucial implementation details on how they are fed into KataGo (such as history frames, the side to move, or output heads). App. J attributes saturation to data shift, but since b = v(x_∅), the metric φ(T) captures the presence versus absence of stones rather than a comparative tactical alternative. High attribution scores (e.g., φ = 1.00 in Fig. 6) might simply reflect this distribution shift rather than valid strategic patterns, and no independent validation (like move-replaying, deep MCTS search, or held-out boards) is provided. Finally, the human evaluation reads as anecdotal: one co-author is among the evaluators (footnote 10), there is a discrepancy in the reported number of experts (one vs. five in App. L), and key protocol details like player level, blinding, and inter-annotator agreement are omitted. Out of 50 annotated coalitions, only 16 are shown without a breakdown of how many were deemed consistent, novel, or conflicting.

Questions for authors
  1. In Fig. 5, is g evaluated against v or v + ε_T, and how is the correction term a_k handled? What performance do an order ≤ 2 model and a model without the ε_T slack achieve, and how sensitive are these results to η?
  2. How stable are the extracted interactions and coalitions across different random seeds, hyper-parameters (η, τ, r), and choices of the 10 target stones / frozen background? Does the baseline in Fig. 2 use the exact same optimization pipeline?
  3. What are the exact inference details when feeding masked states into KataGo (e.g., history stack, turn indicator, specific checkpoint)? Can you provide independent validation (such as move replaying or deep searches) to confirm that patterns conflicting with human knowledge reflect true strategy rather than out-of-distribution artifacts?
  4. Could you clarify the human evaluation setup (exact number and rank of players, blinding protocols, and agreement metrics)? Please also provide the full breakdown and status of all 50 annotated coalitions (consistent, novel, conflicting).
Reviewer_GwjdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an AND-OR interaction framework to disentangle the Go shape patterns encoded by the value network of a Go-playing AI, extending conventional AND interactions with OR interactions, introducing regularization for high-order interactions and common coalitions, and theoretically establishing a universal matching property, with experiments on Go, Gobang, and object detection to demonstrate its potential for interpreting and discovering DNN inference patterns.

Critical strengths
  1. The paper addresses an interesting problem in explainable AI, namely disentangling implicit knowledge encoded in high-performing DNNs, using Go-playing AI as an illustrative application.
  2. Introducing OR interactions extends conventional AND-based interaction representations and potentially improves the expressiveness and conciseness of interaction-based explanations.
  3. The paper attempts to combine theoretical analysis with empirical validation, including a universal matching property and applications to multiple tasks.
Critical weaknesses
  1. The relationship between the extracted AND-OR interactions and the actual QiGan encoded by the value network is insufficiently established. Local stone configurations associated with network outputs do not necessarily represent meaningful strategic knowledge.
  2. The methodological novelty of AND-OR interactions is unclear. The proposed extension appears relatively intuitive, and the manuscript does not adequately distinguish it from existing Boolean interaction representations, rule-based explanations, and feature interaction methods.
  3. The theoretical significance of the universal matching property is questionable. The ability to reconstruct or match DNN outputs does not necessarily guarantee that the extracted interactions faithfully represent the underlying inference mechanisms or knowledge of the model.
  4. The explanation faithfulness criterion is insufficiently justified. Predictive fidelity and sparsity are useful properties, but they do not automatically establish that the extracted interactions correspond to genuine knowledge representations learned by the DNN.
  5. The motivation and effectiveness of high-order interaction regularization remain unclear. The manuscript does not sufficiently demonstrate whether the proposed constraints improve the transferability of meaningful patterns rather than simply suppressing complex interactions.
  6. The mathematical significance of common coalitions is insufficiently explained. Frequent co-occurrence of stone combinations across extracted interactions does not necessarily imply that these coalitions constitute coherent or meaningful strategic patterns.
  7. The claim that the method discovers novel Go knowledge beyond human understanding is not convincingly supported. Qualitative comparisons and feedback from professional players are insufficient to establish the strategic validity, practical usefulness, and genuine novelty of the discovered patterns.
  8. The claimed general applicability of the framework is inadequately validated. The experiments on Gobang and object detection are insufficient to establish that the proposed interaction definitions, extraction mechanisms, and theoretical guarantees generalize across fundamentally different input representations and tasks.
Questions for authors
  1. How do the authors rigorously establish that the extracted AND-OR interactions represent the actual QiGan or strategic knowledge encoded by the value network, rather than merely identifying local feature combinations associated with changes in network outputs?
  2. Why does the universal matching property necessarily imply explanation faithfulness? How do the authors distinguish the reconstruction of model outputs from the disentanglement of the actual knowledge encoded by the model?
  3. How are the claimed novel shape patterns systematically validated beyond qualitative comparisons with traditional Go knowledge? Have the authors conducted controlled evaluations with professional Go players to establish the strategic validity and practical usefulness of these patterns?
  4. What fundamental properties ensure that the proposed AND-OR interaction framework and its theoretical guarantees remain valid across fundamentally different tasks, such as Go, Gobang, and object detection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.