- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a concept-based model's predictions can be reproduced without the input-specific information carried by a separate, uninterpretable channel. It tests this by replacing that channel with a learned distribution shared across inputs and measuring how often the predicted class stays the same. A training penalty encourages the original and replacement predictions to agree. Across several architectures and datasets, the experiments show that this penalty can increase agreement while retaining high accuracy, and can improve accuracy after concept corrections.
Critical strengths
-
The paper gives sidechannel reliance a precise definition and a common measurement framework across several concept-based model architectures.
-
The experiments show that high SIS can coexist with good predictive accuracy, with the clearest improvement over the tested baselines on CUB.
Critical weaknesses
-
A low measured SIS may reflect a poor replacement-fitting procedure rather than genuine dependence on the sidechannel. The paper needs stronger fitting checks before interpreting low SIS as evidence that the sidechannel is necessary.
-
For CMR, removing the sidechannel also changes prediction from selecting one rule to combining all rules. The authors should show whether their result holds without this change.
-
The paper lacks a matched comparison with the disagreement regularization in Decoupling Concept Bottleneck Model. Comparing fixed and learned replacement distributions would clarify what the proposed method adds.
-
The paper does not establish that minimizing sidechannel reliance offers a better practical accuracy–interpretability trade-off than minimizing concept–residual overlap. Its theoretical distinction between these objectives does not replace a matched empirical comparison with mutual-information minimization.
-
The evaluation does not report accuracy under random concept interventions and random residual interventions, as evaluated in Disentangled Concept-Residual Models. These tests would complement SIS by showing whether its improvements correspond to greater sensitivity to concepts and less sensitivity to residuals.
Questions for authors
- With the classifier fixed, how stable are low SIS estimates across more restarts, longer fitting, and richer replacement distributions? Can a tractable example establish how close the fitted score is to the optimum?
- How do CMR's results change when concept-only inference mixes over the original individual rules rather than allowing sets of rules?
- Does the method outperform a matched disagreement-regularization baseline, and how much does learning the replacement distribution help?
- How does concept–residual mutual-information minimization compare on the accuracy–SIS trade-off?
- How do random concept interventions and random residual interventions affect accuracy, and do these changes track improvements in SIS?
4 comments
The paper lacks a matched comparison with the disagreement regularization in Decoupling Concept Bottleneck Model. Comparing fixed and learned replacement distributions would clarify what the proposed method adds.
Does the method outperform a matched disagreement-regularization baseline, and how much does learning the replacement distribution help?
SIS regularization is more general than disagreement regularization. Specifically, disagreement regularization is SIS regularization for a concrete additive architecture (DCBM, y = f(c) + g(z)) with as divergence measure JS divergence and a fixed prior of g(z) = 0. Disagreement regularization can only be applied to specific additive architectures, unlike SIS, which is model-agnostic.
As a response, we did an experiment with the additive CRM architecture comparing disagreement regularization with SIS regularization and ranging their importance weight beta. The hypervolumes (computed as in Table 1) are similar:
| Approach | CelebA | MNIST-Add | CUB |
|---|---|---|---|
| Dis. Reg | 22.00 ± 0.12 | 87.76 ± 0.23 | 62.23 ± 0.18 |
| SIS Reg | 22.05 ± 0.05 | 87.50 ± 0.02 | 61.25 ± 0.20 |
This shows that SIS regularization works equally well as disagreement regularization when both can be applied, while SIS regularization is more general as it works for any CBM, unlike disagreement regularization. We have included the full results in the revised paper (Table 4, Appendix D.2).
To further show the effect of learning the replacement distribution, we performed additional experiments: for all unregularized (β = 0) models, we compare SIS under as fixed replacement the marginal and under our learned prior. Note that the marginal is the natural fixed replacement: unlike ad-hoc choices such as z = 0 or a random categorical distribution, it is KL-optimal among input-independent priors (Theorem E.5). However, note that technically for embedding-based sidechannels like CRM and CEM, the marginal's distribution is a different, more expressive distribution than the original posterior and learned prior (as it is a mixture), which helps explain the few settings where it seems to perform slightly better than the learnable prior.
| SIS (β = 0), fixed → learned | CelebA | MNIST-Add | CUB |
|---|---|---|---|
| CRM | 0.68 → 0.81 | 0.06 → 0.01 | 0.01 → 0.01 |
| CEM | 0.68 → 0.92 | 0.07 → 0.01 | 0.04 → 0.13 |
| LRM | 0.67 → 0.71 | 0.02 → 0.02 | 0.01 → 0.01 |
| CMR | 0.60 → 0.70 | 0.00 → 0.04 | – |
| CMR* | 0.60 → 0.80 | 0.00 → 0.05 | – |
| DCR | 0.68 → 0.85 | 0.05 → 0.01 | – |
Learning the replacement increases SIS for every model on CelebA (up to +0.24) at no cost in accuracy. On MNIST-Add and CUB, most unregularized models have near-zero SIS under any input-independent replacement, because their predictions are carried by the residual. We have included the full results in Table 5, Appendix D.2.
The paper does not establish that minimizing sidechannel reliance offers a better practical accuracy–interpretability trade-off than minimizing concept–residual overlap. Its theoretical distinction between these objectives does not replace a matched empirical comparison with mutual-information minimization.
How does concept–residual mutual-information minimization compare on the accuracy–SIS trade-off?
To empirically support the argument we made (that even I(C;Z) = 0 does not guarantee that predictions are reproducible without the sidechannel), we ran experiments with mutual information regularization, for the full results see Appendix D.2 (Tables 6 and 7, Figures 14-16). For all models and datasets, we replaced the SIS regularizer with a CLUB estimate of I(C;Z) (Cheng et al., 2020), and varied the regularization strength.
Accuracy-SIS hypervolume (as in Table 1), when using MI reg. / when using SIS reg.:
| CelebA | MNIST-Add | CUB | |
|---|---|---|---|
| CRM | 20.6 / 22.3 | 25.4 / 87.3 | 4.9 / 61.7 |
| CEM | 22.0 / 22.3 | 84.7 / 87.5 | 9.2 / 63.1 |
| LRM | 20.6 / 22.2 | 12.5 / 4.9 | 3.6 / 51.8 |
| CMR | 21.8 / 22.3 | 8.0 / 33.7 | – |
| DCR | 21.6 / 22.2 | 11.9 / 10.8 | – |
SIS regularization wins in 11 of 13 settings, often by a large margin. Importantly: MI-minimizing models may sacrifice concept accuracy, which is problematic: concept accuracy ranges 0.17-0.59 on MNIST-Add (including the two settings where MI wins: LRM and DCR) and is on average 0.77 for CUB CEM, against 0.92-0.97 for SIS regularization, which does not affect concept accuracy. For CEM, MI minimization is even ill-posed: concepts are predicted from the residual embeddings, so I(C;Z) can only be reduced by degrading the concepts.
The evaluation does not report accuracy under random concept interventions and random residual interventions, as evaluated in Disentangled Concept-Residual Models. These tests would complement SIS by showing whether its improvements correspond to greater sensitivity to concepts and less sensitivity to residuals.
How do random concept interventions and random residual interventions affect accuracy, and do these changes track improvements in SIS?
We thank the reviewer for the interesting suggestion and have run these tests. Following Zabounidis et al., a random concept (residual) intervention replaces the predicted concepts (residual) of each test input with those of another randomly chosen test input, leaving the other part unchanged (5 random permutations). We evaluated over 3000 trained models (all models, datasets, regularization strengths and seeds). Below, we provide some examples, full results on all models are in Appendix D.2 (Tables 8 and 9, Figure 17). We can draw the following conclusions:
- Without SIS regularization, random residual interventions reduce accuracy to chance for all models, while random concept interventions affect it much less: the models rely mostly on their residual. (random chance = accuracy when both concepts and residual come from a different random example).
- SIS regularization shifts this dependence towards the concepts, completely so for models that reach high SIS.
The table below shows an example for CRM where the SIS-regularized model is the one with the highest SIS within 2 points of the unregularized accuracy. Accuracies are in %, with chance (explained earlier) in brackets.
| Dataset | β (SIS reg. strength) | Acc. | Acc under rand. concepts | Acc under rand. residual | SIS |
|---|---|---|---|---|---|
| CelebA (chance 57) | 0 | 84.7 | 84.7 | 57.4 | 81.0 |
| CelebA (chance 57) | 1 | 83.2 | 59.2 | 82.4 | 98.8 |
| MNIST-Add (chance 6.5) | 0 | 96.2 | 96.2 | 6.5 | 0.7 |
| MNIST-Add (chance 6.5) | 1.5 | 96.1 | 6.6 | 96.1 | 100 |
| CUB (chance 0.5) | 0 | 66.7 | 66.6 | 0.5 | 0.6 |
| CUB (chance 0.5) | 0.5 | 66.6 | 15.2 | 26.3 | 53.0 |
Additionally, if we define concept (residual) sensitivity as the fraction of above-chance accuracy lost under random concept (residual) interventions (see Tables 8, 9 and Figure 17 in Appendix D.2 for all results), we can draw the following conclusions:
- On CelebA and CUB, SIS is strongly rank-correlated with concept sensitivity (Spearman ρ = +0.92 to +0.99) and anti-correlated with residual sensitivity (ρ = −0.95 to −0.99) for all models, across all configurations. The same holds when only β varies at a fixed architecture (+0.80 to +1.00 and −0.80 to −1.00).
- On MNIST-Add (has complete concepts), the trend has the same sign but is weaker.
Dear reviewer,
Please let us know if you have any further questions or things we could clarify further. If not, we would appreciate if you could consider updating your review based on our replies.
We thank the reviewer for the interesting questions. Answering them led us to run several new comparisons, which we found very valuable, as they consistently support the design choices behind SIS and strengthen the paper’s empirical evidence. All new results are included in the revised paper (changes in blue) and summarized below.
To test the stability of the fitted prior, we performed new experiments where we refit p(z) for every β=0 model (all datasets, all seeds) for five random initializations and for twice the original number of epochs. SIS proves highly stable: for most of the model-dataset pairs, the standard deviation across restarts is below 0.001, and longer fitting generally changes SIS by less than 0.005 (an example exception is CEM on CUB, 0.13 -> 0.20 after convergence), which remains far below the SIS achieved with SIS regularization. This confirms that the low SIS estimates are stable. We have included the full results in the revised paper (Table 2, Appendix D.2).
We chose to use the same replacement type of distribution (e.g. categorical) as the original sidechannel distribution, as otherwise one cannot really claim it is still the same model. However, our results further below compare the learned prior with the marginal prior, and the latter is a more expressive distribution than the former for embedding-based sidechannels (as it is a mixture of embeddings instead of a single embedding).
More fundamentally, in practice a CSM is only as representation-interpretable as the best concept-only predictor one can "actually obtain", not one that merely exists". Below, we argue that, while SIS captures what exists in principle, the approximated SIS captures what can be obtained in practice, and it is the latter that determines how well a model can actually be interpreted. If no input-independent replacement can be found that reproduces the model's predictions, the model cannot be used or explained through its concepts, even if such a replacement exists in principle. The same standard is common elsewhere: a network may contain a small subnetwork that matches its performance (Frankle & Carbin, 2019), but it only becomes simpler once such a subnetwork is found. In mechanistic interpretability, a model is explained to the extent that a discovered circuit reproduces its behavior, typically measured by the fraction of performance the circuit recovers (Wang et al., 2023; Conmy et al., 2023). SIS follows this view: it certifies the agreement achieved by a concrete concept-only predictor, the bottleneck mode with the fitted prior. If follow-up work can find better ways to fit the prior, CSMs will become more representation-interpretable in practice, and the approximated SIS will reflect this.
Frankle & Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ICLR 2019.
Wang, Variengien, Conmy, Shlegeris & Steinhardt, Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023.
Conmy, Mavor-Parker, Lynch, Heimersheim & Garriga-Alonso, Towards Automated Circuit Discovery for Mechanistic Interpretability, NeurIPS 2023.
We performed an extra experiment where CMR's bottleneck mode uses only a single rule: the conclusions we can draw are the same. For instance, on CelebA, the hypervolume with SIS regularization is nearly identical (22.32 ± 0.05 vs. 22.11 ± 0.04), and on MNIST-Addition it is even higher (33.72 ± 1.37 vs. 4.74 ± 0.45). The latter is quite interesting considering CMR's theoretical expressivity is strictly higher with multiple rules, which shows again the limitation of CMR's gradient-based rule learning which is hard to optimize (which we already previously addressed in Appendix D.2 by equipping it with a better rule learner). In both cases, SIS regularization strongly improves on the unregularized model, and CMR on MNIST-Addition still trades accuracy almost linearly for SIS. We included the full results in the revised paper (Table 3 and Figure 13, Appendix D.2).