- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper concerns connecting concepts (SAEs and their activation) between layers of a model through co-occurrence. The ensuing "Conceptome" is argued to serve as surrogate model "explanation".
Critical strengths
Important problem. Well argued attempt at combining SAEs into more global metacognitive framework.
Critical weaknesses
Unclear whether the resulting explanations are understandable, comprehensive and actionable. How are users supposed to make sense of the complex wiring patterns shown in the appendix C?
Questions for authors
Please give concrete use cases and worked examples of usability
2 comments
Not entirely convinced about the usability
Thank you for the response and attempt to address usability, however I still struggle with the complexity of the output and I am not yet convinced. I still think it is a quite cool idea and maintain my positive score.
We thank the reviewer for recognizing the importance of the problem and the merits of our approach. We appreciate the insightful feedback regarding the usability of the CONCEPTOME and address this concern through concrete use cases and worked examples, which we have added to Appendix E (p. 20; Figures 12–17, pp. 24–27) of the revised manuscript.
W1 & Q1: Understanding and using the CONCEPTOME
The global structure illustrated in Appendix C allows users to explore individual rules and trace how concepts interact and propagate across layers. An important advantage of our approach, however, is that the CONCEPTOME can be queried for an individual sample, a class, or an arbitrary collection of samples. Users can thus extract subgraphs relevant to their questions and localize their analysis to specific layers or paths within the network. We illustrate this through the following use cases.
Understanding concept reuse and class differentiation. In addition to the CONCEPTOME example in Figure 6 of the original manuscript, we present class-specific paths for tabby (Figure 12, p. 24) and tiger cat (Figure 13, p. 24), illustrating how concepts combine across layers to reach their respective predictions. We further show a shared path that branches into distinct rules at later layers, leading to the two classes (Figure 14, p. 25). These examples demonstrate how users can investigate which concept interactions are reused across classes and where their computational pathways diverge, all within a common global structure.
Debugging prediction errors. By querying a collection of tiger cat images misclassified as tabby (Figure 15, p. 25), we identify a shared path associated with these incorrect predictions. This allows users to move beyond individual mistakes and investigate recurring concept interactions across samples exhibiting the same failure mode.
We further query collections combining correctly and incorrectly classified samples: balloon images misclassified as parachute together with correctly classified parachutes (Figure 16, p. 26), and rock beauty images misclassified as tench together with correctly classified tenches (Figure 17, p. 27). In both cases, we identify shared paths associated with correct and incorrect predictions. For instance, the balloon-to-parachute path contains concepts associated with both balloons and parachutes, providing insight into shared visual representations that may contribute to this confusion. These examples also illustrate how users can localize their perspective to particular portions of the network.
From explanation to intervention. The recovered paths provide starting points for investigating model behavior beyond visual inspection. Users can identify concepts and rules shared across failure cases, formulate hypotheses about their role in model predictions, and test these hypotheses through targeted interventions, as demonstrated in our interventional experiments in Section 5.
Together, these examples illustrate the CONCEPTOME as a queryable representation of model computation, enabling users to explore how vision models organize and reuse concepts, investigate shared and divergent computational pathways, and gain insights into the model's behavior across both successful and unsuccessful predictions.