- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of constructing a global surrogate for a frozen model using logical rules defined over a set of features. These features may correspond to concepts in datasets such as CUB, concepts in CBMs, or features extracted by SAEs. The proposed framework applies thresholding to these features, resulting in binary indicators (selected/unselected), and then seeks logical rules over these indicators that capture the behavior of the frozen model. The authors refer to these logical rules as an "explanation". The paper identifies three fundamental representational limitations of such explanations: (1) information loss (B1), (2) unavailable curved structure (B2), and (3) missing flat direction (B3). These limitations characterize cases in which logical rules over thresholded features cannot faithfully represent the underlying model. The authors then conduct experiments across several benchmarks, including CUB, PTB-XL, and WikiText, using models such as GPT-2 and concept-bottleneck architectures for image and ECG classification. The experimental evaluation mainly focuses on demonstrating information loss (B1). Specifically, for each benchmark, the authors identify inputs that produce identical thresholded feature representations but different model predictions. Therefore, any logical rule defined only over these thresholded features must necessarily fail to represent the model correctly on at least one of these inputs, which is basically the limitation described by B1. The authors also show that refining the representation can remove this information loss. For example, on CUB, moving from binary concept thresholds to 32 levels per concept eliminates all such conflicting inputs on the evaluated sample.
Critical strengths
-
The idea of constructing a global surrogate for a frozen model using logical rules over a set of interpretable features could be useful. However, I believe the paper would benefit from a clearer motivation and justification for why this particular form of surrogate is desirable (e.g., what other more standard forms are missing).
-
The paper considers a relatively diverse set of settings, including CUB, CBMs, and SAEs, which helps illustrate applicability across different interpretability settings.
-
I appreciate the inclusion of formal theoretical results, which I believe are important and still relatively uncommon in interpretability research. That said, the specific results presented here seem somewhat expected and, in my view, do not provide particularly significant new insights into interpretability (see weaknesses below).
Critical weaknesses
-
Writing needs improvement. The paper is currently (very) difficult to follow. It took me several readings to fully understand the setting, the main objectives, and the contributions. Part of this difficulty is due to many paper-specific definitions and terms, such as "Joint1" (which, as far as I could tell, is never even formally defined), B1, B2, B3, Gate 1, and Gate 2. These make the overall framework unnecessarily complicated to follow. Additionally, several technical sections are written in a way that makes the arguments really difficult to follow. More fundamentally, the paper introduces the term "explanation" to refer to a logical rule intended to serve as a global surrogate for a frozen model over thresholded features. This is not necessarily the conventional meaning of an explanation in interpretability research, and the choice is not motivated. In particular, the paper should explain much earlier and more explicitly what setting it considers, why this setting is important, what problem it aims to solve, and how the proposed contributions address that problem. Overall, the motivation, problem formulation, and main contributions are presented very vaguely, making it difficult to understand what the paper actually does. I strongly encourage the authors to substantially revise the presentation and improve the accessibility of the paper.
-
The “explanation” definition is not well motivated, and it seems to be the main source of the limitations the paper identifies. The paper shows that thresholding concepts into binary indicators discards information that the CBM head relies on, and that using finer thresholds recovers it. This is a fairly expected result, but my main concern is with the setup itself. The information loss comes from the paper’s own choice to define an explanation as a logical rule over thresholded features. Standard concept-based and mechanistic interpretability methods typically work with continuous activations or directions instead. A CBM head, for example, is a weighted combination of concept scores, so it keeps this information by construction. The paper does not explain why its more restrictive explanation definition should be preferred. In fact, under the paper’s own theory, this standard CBM explanation cannot be expressed at all: a head that is linear in continuous concept scores produces decision boundaries that are tilted or curved relative to per-concept thresholds, so the budget K is infinite no matter how many bins are used. In other words, the framework treats the most common way of explaining a CBM as inexpressible, and the 32-bin “repair” on CUB holds only for the finite set of evaluated images, not in the sense the theory defines. Finally, thresholding is not the common thread across the experiments either. In the ECG concept bottleneck, the concepts are already binary inside the model, and the failures arise only because three unnamed units that the head explicitly uses are left out of the explanation. Which diseases fail is therefore known in advance from the architecture, which makes this experiment close to trivial.
-
Lack of motivation for using global rather than local surrogates. Interpretability methods often aim to explain individual predictions through local explanations or local surrogates, rather than attempting to capture the behavior of an entire model using a single global surrogate. Such a local formulation may be more realistic and could potentially avoid some of the representational limitations identified here. Overall, I believe the paper needs to provide a much stronger justification for its particular choice of explanation framework and clarify how it relates to existing interpretability approaches.
-
Theoretical results are expected. While I appreciate the theoretical component of the paper, the main results seem relatively expected given the requirement that logical rules over thresholded features should perfectly represent the original model. In particular, the three limitations (B1, B2, and B3) appear to characterize fairly natural representational mismatches between the restricted logical-rule framework and the original model. Therefore, I am not convinced that these results offer important new insights into interpretability or reveal surprising limitations that would not already be anticipated from the formulation.
-
Experiments are mainly on B1 and not on B2 and B3. The experimental evaluation does not seem to fully align the breadth of the theoretical analysis. While the theoretical section introduces three distinct representational limitations (B1, B2, and B3), the experiments focus mainly on demonstrating B1. In particular, the experiments show that two inputs may have identical thresholded feature representations while producing different model predictions. Although this is a valid demonstration of the way the authors defined information loss, it leaves the practical relevance of B2 and B3 largely unexplored.
-
Mismatch between the theory and experiments definition. The theoretical framework requires the logical-rule representation to agree with the original model almost surely over the full data distribution. This seems like a rather strong requirement, particularly for a global surrogate intended to approximate a potentially complex model. Moreover, there appears to be a mismatch between the theoretical and experimental settings: while the theoretical analysis considers agreement over the full (continuous) data distribution, the experiments demonstrate the limitations using examples drawn from a finite sample of points.
-
Missing related work. Concept leakage is a well established phenomena which seems to align closely with the ideas discussed here and is not discussed/compared to at all in the paper.
Questions for authors
-
In settings such as CBMs and SAEs, explanations are often expressed through directions, continuous activations, or mappings in latent space, rather than binary thresholded features. Can the proposed framework accommodate such representations? It seems possible to me that a significant portion of the information loss identified in the paper arises specifically from thresholding, which is not necessarily the standard way these features are interpreted in mechanistic interpretability. How do the authors view this limitation?
-
Would it make sense to consider a local version of the proposed explanation framework, where different logical rules are constructed for different inputs or local regions of the input space? This seems more closely aligned with several common interpretability approaches and could potentially improve some of the representational limitations discussed in the paper.
-
Some of the assumptions in the theoretical analysis, particularly the requirement of exact (almost-sure) agreement between the logical-rule representation and the original model over the full data distribution, seem quite strong. Could these assumptions be relaxed, for instance, by considering approximate agreement? Would the main theoretical conclusions still hold under such relaxations?
-
Is there a way to experimentally evaluate the limitations associated with B2 and B3, similarly to the evaluation of B1?