- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes combining mechanistic interpretability (MI) and neurosymbolic AI to extract logical representations from neural networks. The paper creates an exhaustive dataset of connected non-isomorphic graphs with 5 nodes and 2 possible node colour and trains 2-layer GNN on the dataset. Sparse autoencoders (SAEs) are then used to identify latent features corresponding to these predicates, followed by causal interventions to evaluate their functional roles. The collected results show evidence that GNN activations can behave consistently with discrete relational logic programs.
Critical strengths
The direction of combining MI and formal logic is meaningful and novel to the best of my knowledge.
The proposal and definition of "feature ablation, counterfactual injection, and feature swapping" are helpful tools that provide stronger evidence for the logic interpretations that correlation-based analysis alone.
Critical weaknesses
Large gap between the paper's motivation and its actual contribution. The paper motivates its approach by highlighting the lack of a formal language for specification and verification in mechanistic interpretability ("Mechanistic interpretability (MI) aims to reverse-engineer subgraphs of neural networks into human-understandable circuits; yet, it lacks a formal language for specification and verification."). The proposed approach is evaluated only on a highly controlled synthetic dataset consisting of five-node graphs with binary node colors and a small set of predefined logical rules. Also, the candidate predicates and their satisfaction conditions are already known and used to align the extracted SAE features with logical interpretations. The experiments demonstrate that the proposed method can identify and causally test neural features corresponding to known logical predicates, rather than discovering previously unknown mechanisms.
While this work is positioned as a feasibility study, the extremely restricted experimental setting makes it difficult to assess whether the proposed framework meaningfully addresses the broader problem identified in the motivation. In particular, it remains unclear how the approach would generalize to settings where the underlying logical structure is unknown, or where the relevant mechanisms cannot be easily expressed using predefined logical templates. The toy problem studied here is too specific in nature to claim neurosymbolic MI for GNN, even as a feasibility study.
The paper only executes 10 random seeds for the experiments, and coupled with the large standard deviations, I think more random seeds should be tested, at least 30. The results in Table 1 can also be misleading, with near perfect necessity score in Table 1 for RBB, but a much poorer score in Table 6.
Questions for authors
Please address the weaknesses above.
I am also interested to know if the paper would like to scale back their claims on the problem they are tackling, in which rather than tackling neurosymbolic MI for GNN, they study a specific task, which is non-isomorphic graphs with coloured nodes.
2 comments
I thank the authors for revision and rebuttal. I have also read the other reviews and rebuttals. The manuscript is improved, in terms of clarity. On that front, I do not have any further query. However, the major concern still remains, about the scope of the claims and the experiments conducted. As the rebuttal mentioned, this specific task of colored non-isomorphic graphic is a testbed. It is, in my reviewing opinion, a highly restrictive testbed and am I unable to see how this would extend to actual applications. I would be interested to see results beyond this testbed. Thus, the paper has improved in clarity, but my score remains the same.
Scope and seed variance
We thank the reviewer for the detailed review and for recognizing the combination of MI and formal logic as a meaningful and novel direction, as well as the value of the proposed interventions over correlation-based analysis.
Re W1, W2 (motivation and contribution): We would like to clarify how the motivation and contribution are connected. The motivation identifies the lack of a formal specification language for MI interpretations. Our contribution is to show that, for model classes with known logical expressiveness bounds, such a language exists and can be operationalized: the bound defines the vocabulary, SAEs provide the grounding, and causal interventions provide the verification. The feasibility study demonstrates that this pipeline works end-to-end in a setting where every step can be checked against ground truth—a necessary first step before applying it where ground truth is unavailable. We are explicit about what is supplied and what is discovered (Sec. 4), and we have newly sharpened the wording at the end of the introduction (Sec. 1) so that the scope of the claims is unambiguous.
Re W3 (number of seeds): We note that the variance we report is not uniformly distributed but concentrated in a minority of runs (App. B.3): for ΦR,B,B, 7 of 10 runs achieve necessity ≥ 0.95 and specificity of 100%, and the two low-specificity runs coincide with the low-necessity runs, indicating isolated poor SAE fits rather than a systematic failure of the method. Since the GNN and SAE are seeded jointly, a poor SAE initialization cannot be compensated within a run; selecting over several SAE seeds per GNN would remove most of this spread.
Re W3 (Table 1 vs. Table 6): The two tables report different quantities, which we now make explicit in the revised captions. Table 1 reports the selected primary SAE per progression from a single-seed run (the best-λ configuration chosen by the criterion in App. B.1), whereas Table 6 reports the mean ± std across all seeds, including the isolated poor runs discussed above. The near-perfect RBB necessity in Table 1 and the lower mean in Table 6 are therefore consistent, and the two captions now cross-reference each other to prevent this misreading.
Re Q1 (scope of the claims): Our claim is not that neurosymbolic MI for GNNs is solved, but that the proposed synthesis is feasible and verifiable, with GNNs as a principled first instantiation because their logical expressiveness is formally characterized. The specific task—colored non-isomorphic graphs—is the testbed on which this is demonstrated, not the object of the contribution itself. We now state this explicitly at the end of the introduction (Sec. 1).