acceptodds
Under review as a conference paper at ICLR 2027

Quantifying the Accuracy-Interpretability Trade-off in Concept Sidechannel Models

15 views

Abstract

Concept bottleneck models make predictions through human-understandable concepts, providing interpretability. However, this bottleneck limits task accuracy when the concept set is incomplete or the concept-to-task map is insufficiently expressive. Recent concept-based models address this limitation by adding sidechannels that transmit additional information around the bottleneck. These sidechannels often recover accuracy, but they also raise a basic diagnostic question: how much can the final prediction still be interpreted using the concept pathway, and how much interpretability is lost? We study this question through a unified probabilistic view of concept sidechannel models. The framework exposes two inference modes: the default inference mode of the model, and a bottleneck mode, in which the sidechannel is replaced by an input-independent distribution. Comparing these modes yields an architecture-agnostic invariance metric, and naturally leads to a regularizer that penalizes discrepancies between the two modes. Across several architectures and datasets, we find that accuracy-only training often produces models whose predictions cannot be reproduced without the sidechannel, even when concept-only prediction is feasible. SIS regularization allows users to trace a controlled accuracy-interpretability frontier, improves responsiveness to concept interventions, and can yield task predictors that rely more visibly on concepts. Our results provide practical tools for auditing and tuning the role of sidechannels in concept-based models.

Primary area:
interpretability and explainable AI
Keywords:
concept-based models, interpretability, explainable AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EWPmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a concept-based model's predictions can be reproduced without the input-specific information carried by a separate, uninterpretable channel. It tests this by replacing that channel with a learned distribution shared across inputs and measuring how often the predicted class stays the same. A training penalty encourages the original and replacement predictions to agree. Across several architectures and datasets, the experiments show that this penalty can increase agreement while retaining high accuracy, and can improve accuracy after concept corrections.

Critical strengths
  • The paper gives sidechannel reliance a precise definition and a common measurement framework across several concept-based model architectures.

  • The experiments show that high SIS can coexist with good predictive accuracy, with the clearest improvement over the tested baselines on CUB.

Critical weaknesses
  • A low measured SIS may reflect a poor replacement-fitting procedure rather than genuine dependence on the sidechannel. The paper needs stronger fitting checks before interpreting low SIS as evidence that the sidechannel is necessary.

  • For CMR, removing the sidechannel also changes prediction from selecting one rule to combining all rules. The authors should show whether their result holds without this change.

  • The paper lacks a matched comparison with the disagreement regularization in Decoupling Concept Bottleneck Model. Comparing fixed and learned replacement distributions would clarify what the proposed method adds.

  • The paper does not establish that minimizing sidechannel reliance offers a better practical accuracy–interpretability trade-off than minimizing concept–residual overlap. Its theoretical distinction between these objectives does not replace a matched empirical comparison with mutual-information minimization.

  • The evaluation does not report accuracy under random concept interventions and random residual interventions, as evaluated in Disentangled Concept-Residual Models. These tests would complement SIS by showing whether its improvements correspond to greater sensitivity to concepts and less sensitivity to residuals.

Questions for authors
  1. With the classifier fixed, how stable are low SIS estimates across more restarts, longer fitting, and richer replacement distributions? Can a tractable example establish how close the fitted score is to the optimum?
  2. How do CMR's results change when concept-only inference mixes over the original individual rules rather than allowing sets of rules?
  3. Does the method outperform a matched disagreement-regularization baseline, and how much does learning the replacement distribution help?
  4. How does concept–residual mutual-information minimization compare on the accuracy–SIS trade-off?
  5. How do random concept interventions and random residual interventions affect accuracy, and do these changes track improvements in SIS?
4 comments

We thank the reviewer for the interesting questions. Answering them led us to run several new comparisons, which we found very valuable, as they consistently support the design choices behind SIS and strengthen the paper’s empirical evidence. All new results are included in the revised paper (changes in blue) and summarized below.

 

A low measured SIS may reflect a poor replacement-fitting procedure rather than genuine dependence on the sidechannel. The paper needs stronger fitting checks before interpreting low SIS as evidence that the sidechannel is necessary.

With the classifier fixed, how stable are low SIS estimates across more restarts, longer fitting, and richer replacement distributions? Can a tractable example establish how close the fitted score is to the optimum?

To test the stability of the fitted prior, we performed new experiments where we refit p(z) for every β=0 model (all datasets, all seeds) for five random initializations and for twice the original number of epochs. SIS proves highly stable: for most of the model-dataset pairs, the standard deviation across restarts is below 0.001, and longer fitting generally changes SIS by less than 0.005 (an example exception is CEM on CUB, 0.13 -> 0.20 after convergence), which remains far below the SIS achieved with SIS regularization. This confirms that the low SIS estimates are stable. We have included the full results in the revised paper (Table 2, Appendix D.2).

We chose to use the same replacement type of distribution (e.g. categorical) as the original sidechannel distribution, as otherwise one cannot really claim it is still the same model. However, our results further below compare the learned prior with the marginal prior, and the latter is a more expressive distribution than the former for embedding-based sidechannels (as it is a mixture of embeddings instead of a single embedding).

More fundamentally, in practice a CSM is only as representation-interpretable as the best concept-only predictor one can "actually obtain", not one that merely exists". Below, we argue that, while SIS captures what exists in principle, the approximated SIS captures what can be obtained in practice, and it is the latter that determines how well a model can actually be interpreted. If no input-independent replacement can be found that reproduces the model's predictions, the model cannot be used or explained through its concepts, even if such a replacement exists in principle. The same standard is common elsewhere: a network may contain a small subnetwork that matches its performance (Frankle & Carbin, 2019), but it only becomes simpler once such a subnetwork is found. In mechanistic interpretability, a model is explained to the extent that a discovered circuit reproduces its behavior, typically measured by the fraction of performance the circuit recovers (Wang et al., 2023; Conmy et al., 2023). SIS follows this view: it certifies the agreement achieved by a concrete concept-only predictor, the bottleneck mode with the fitted prior. If follow-up work can find better ways to fit the prior, CSMs will become more representation-interpretable in practice, and the approximated SIS will reflect this.

 

Frankle & Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ICLR 2019.

Wang, Variengien, Conmy, Shlegeris & Steinhardt, Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023.

Conmy, Mavor-Parker, Lynch, Heimersheim & Garriga-Alonso, Towards Automated Circuit Discovery for Mechanistic Interpretability, NeurIPS 2023.

 

For CMR, removing the sidechannel also changes prediction from selecting one rule to combining all rules. The authors should show whether their result holds without this change.

How do CMR's results change when concept-only inference mixes over the original individual rules rather than allowing sets of rules?

We performed an extra experiment where CMR's bottleneck mode uses only a single rule: the conclusions we can draw are the same. For instance, on CelebA, the hypervolume with SIS regularization is nearly identical (22.32 ± 0.05 vs. 22.11 ± 0.04), and on MNIST-Addition it is even higher (33.72 ± 1.37 vs. 4.74 ± 0.45). The latter is quite interesting considering CMR's theoretical expressivity is strictly higher with multiple rules, which shows again the limitation of CMR's gradient-based rule learning which is hard to optimize (which we already previously addressed in Appendix D.2 by equipping it with a better rule learner). In both cases, SIS regularization strongly improves on the unregularized model, and CMR on MNIST-Addition still trades accuracy almost linearly for SIS. We included the full results in the revised paper (Table 3 and Figure 13, Appendix D.2).

The paper lacks a matched comparison with the disagreement regularization in Decoupling Concept Bottleneck Model. Comparing fixed and learned replacement distributions would clarify what the proposed method adds.

Does the method outperform a matched disagreement-regularization baseline, and how much does learning the replacement distribution help?

SIS regularization is more general than disagreement regularization. Specifically, disagreement regularization is SIS regularization for a concrete additive architecture (DCBM, y = f(c) + g(z)) with as divergence measure JS divergence and a fixed prior of g(z) = 0. Disagreement regularization can only be applied to specific additive architectures, unlike SIS, which is model-agnostic.

As a response, we did an experiment with the additive CRM architecture comparing disagreement regularization with SIS regularization and ranging their importance weight beta. The hypervolumes (computed as in Table 1) are similar:

ApproachCelebAMNIST-AddCUB
Dis. Reg22.00 ± 0.1287.76 ± 0.2362.23 ± 0.18
SIS Reg22.05 ± 0.0587.50 ± 0.0261.25 ± 0.20

This shows that SIS regularization works equally well as disagreement regularization when both can be applied, while SIS regularization is more general as it works for any CBM, unlike disagreement regularization. We have included the full results in the revised paper (Table 4, Appendix D.2).

To further show the effect of learning the replacement distribution, we performed additional experiments: for all unregularized (β = 0) models, we compare SIS under as fixed replacement the marginal and under our learned prior. Note that the marginal is the natural fixed replacement: unlike ad-hoc choices such as z = 0 or a random categorical distribution, it is KL-optimal among input-independent priors (Theorem E.5). However, note that technically for embedding-based sidechannels like CRM and CEM, the marginal's distribution is a different, more expressive distribution than the original posterior and learned prior (as it is a mixture), which helps explain the few settings where it seems to perform slightly better than the learnable prior.

SIS (β = 0), fixed → learnedCelebAMNIST-AddCUB
CRM0.68 → 0.810.06 → 0.010.01 → 0.01
CEM0.68 → 0.920.07 → 0.010.04 → 0.13
LRM0.67 → 0.710.02 → 0.020.01 → 0.01
CMR0.60 → 0.700.00 → 0.04–
CMR*0.60 → 0.800.00 → 0.05–
DCR0.68 → 0.850.05 → 0.01–

Learning the replacement increases SIS for every model on CelebA (up to +0.24) at no cost in accuracy. On MNIST-Add and CUB, most unregularized models have near-zero SIS under any input-independent replacement, because their predictions are carried by the residual. We have included the full results in Table 5, Appendix D.2.

 

The paper does not establish that minimizing sidechannel reliance offers a better practical accuracy–interpretability trade-off than minimizing concept–residual overlap. Its theoretical distinction between these objectives does not replace a matched empirical comparison with mutual-information minimization.

How does concept–residual mutual-information minimization compare on the accuracy–SIS trade-off?

To empirically support the argument we made (that even I(C;Z) = 0 does not guarantee that predictions are reproducible without the sidechannel), we ran experiments with mutual information regularization, for the full results see Appendix D.2 (Tables 6 and 7, Figures 14-16). For all models and datasets, we replaced the SIS regularizer with a CLUB estimate of I(C;Z) (Cheng et al., 2020), and varied the regularization strength.

Accuracy-SIS hypervolume (as in Table 1), when using MI reg. / when using SIS reg.:

CelebAMNIST-AddCUB
CRM20.6 / 22.325.4 / 87.34.9 / 61.7
CEM22.0 / 22.384.7 / 87.59.2 / 63.1
LRM20.6 / 22.212.5 / 4.93.6 / 51.8
CMR21.8 / 22.38.0 / 33.7–
DCR21.6 / 22.211.9 / 10.8–

SIS regularization wins in 11 of 13 settings, often by a large margin. Importantly: MI-minimizing models may sacrifice concept accuracy, which is problematic: concept accuracy ranges 0.17-0.59 on MNIST-Add (including the two settings where MI wins: LRM and DCR) and is on average 0.77 for CUB CEM, against 0.92-0.97 for SIS regularization, which does not affect concept accuracy. For CEM, MI minimization is even ill-posed: concepts are predicted from the residual embeddings, so I(C;Z) can only be reduced by degrading the concepts.

The evaluation does not report accuracy under random concept interventions and random residual interventions, as evaluated in Disentangled Concept-Residual Models. These tests would complement SIS by showing whether its improvements correspond to greater sensitivity to concepts and less sensitivity to residuals.

How do random concept interventions and random residual interventions affect accuracy, and do these changes track improvements in SIS?

We thank the reviewer for the interesting suggestion and have run these tests. Following Zabounidis et al., a random concept (residual) intervention replaces the predicted concepts (residual) of each test input with those of another randomly chosen test input, leaving the other part unchanged (5 random permutations). We evaluated over 3000 trained models (all models, datasets, regularization strengths and seeds). Below, we provide some examples, full results on all models are in Appendix D.2 (Tables 8 and 9, Figure 17). We can draw the following conclusions:

  • Without SIS regularization, random residual interventions reduce accuracy to chance for all models, while random concept interventions affect it much less: the models rely mostly on their residual. (random chance = accuracy when both concepts and residual come from a different random example).
  • SIS regularization shifts this dependence towards the concepts, completely so for models that reach high SIS.

The table below shows an example for CRM where the SIS-regularized model is the one with the highest SIS within 2 points of the unregularized accuracy. Accuracies are in %, with chance (explained earlier) in brackets.

Datasetβ (SIS reg. strength)Acc.Acc under rand. conceptsAcc under rand. residualSIS
CelebA (chance 57)084.784.757.481.0
CelebA (chance 57)183.259.282.498.8
MNIST-Add (chance 6.5)096.296.26.50.7
MNIST-Add (chance 6.5)1.596.16.696.1100
CUB (chance 0.5)066.766.60.50.6
CUB (chance 0.5)0.566.615.226.353.0

Additionally, if we define concept (residual) sensitivity as the fraction of above-chance accuracy lost under random concept (residual) interventions (see Tables 8, 9 and Figure 17 in Appendix D.2 for all results), we can draw the following conclusions:

  • On CelebA and CUB, SIS is strongly rank-correlated with concept sensitivity (Spearman ρ = +0.92 to +0.99) and anti-correlated with residual sensitivity (ρ = −0.95 to −0.99) for all models, across all configurations. The same holds when only β varies at a fixed architecture (+0.80 to +1.00 and −0.80 to −1.00).
  • On MNIST-Add (has complete concepts), the trend has the same sign but is weaker.

Dear reviewer,

Please let us know if you have any further questions or things we could clarify further. If not, we would appreciate if you could consider updating your review based on our replies.

Reviewer_jLKhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the accuracy-interpretability tradeoff in concept-based sidechannel models by proposing (1) a metric (SIS) that quantifies how much the final prediction depends on concepts based on probabilistic graphical model formulation of sidechannel models and (2) a method (SIS regularization) that can control how much the prediction depends on the concepts. Experiment results show that SIS regularization (1) improves interpretability by reducing dependence on sidechannels and (2) improves intervenability.

Critical strengths
  1. Proposal of a training technique (SIS regularization) that improves the accuracy-interpretability frontier of concept-based sidechannel models seems to be a good contribution. I think the paper provides good motivation for the regularization term based on the PGM formulation of sidechannel models and sufficient experimental results to show the usefulness of SIS regularization in yielding better interpretable models without compromising performance.
Critical weaknesses
  1. The framework the authors propose (functional interpretability vs representation interpretability, CSM meta-model) looks a bit weak. Section 3.1 introduces the notions, functional interpretability and representation interpretability using Definition, Assumption, Proposition format, but the actual statements made seem to be less rigorous (notion of interpretability in Definition 3.1 and 3.2 is bit vague, Proposition 3.1 is not really a mathematical proposition). Also CSM meta-model seems to be a straight-forward PGM formulation of sidechannel models. Though I think it is a good basis for introducing SIS, I’m not sure whether this can count as a standalone contribution.
Questions for authors
  1. While I do think \hat{SIS} is a good intuitive approximation to SIS, does Theorem 3.1 rigorously supports that \hat{SIS} is a good proxy for SIS? For now, it describes only the lower bound of SIS - \hat{SIS} and not the upper bound, so it seems like Theorem 3.1 has little value. Examining the proof of Theorem 3.1, I don't think upper bound can be easily attained (e.g. expanding two-sided Hoeffding inequality doesn't seem to work since the information SIS \geq SIS_{\hat{q}} is only useful for the lower bound). Can you also elaborate on what “Generalization of SIS” means in Theorem 3.1?
  2. How are the hyperparameters \alpha and \beta in SIS regularized training objective set? How do these hyperparameters affect the training? The paper seems to lack details on training configurations when using SIS regularization.
  3. The authors state that “we show that this is a weaker criterion than minimizing sidechannel reliance, as a model could fully maximize its reliance on concepts while still relying heavily on the sidechannel to form the final prediction.” (line 80-82), but I’m not sure where this point is shown. Can you clarify where your point was shown?
5 comments

We first want to thank the reviewer for taking the time to read our paper. Below, we respond to the reviewer's concerns. In the revised paper pdf, we have included additional results that are also briefly highlighted below (changes are in blue).

 

The framework the authors propose (functional interpretability vs representation interpretability, CSM meta-model) looks a bit weak. Section 3.1 introduces the notions, functional interpretability and representation interpretability using Definition, Assumption, Proposition format, but the actual statements made seem to be less rigorous (notion of interpretability in Definition 3.1 and 3.2 is a bit vague, Proposition 3.1 is not really a mathematical proposition). Also CSM meta-model seems to be a straight-forward PGM formulation of sidechannel models. Though I think it is a good basis for introducing SIS, I’m not sure whether this can count as a standalone contribution.

Our proposed framework is only part of our contributions, which are also required to make SIS definable. In the current literature, CBM papers generally do not explain what "interpretability" they mean, while we want to avoid confusion by making the distinction between functional and representation interpretability concrete. The CSM meta-model is indeed a simple abstraction of CSMs, but making it concrete allows to define SIS and its regularization in a model-agnostic way, i.e. they can be measured / used for any CSM that implements the meta-model (which is currently all CSMs that we are aware of).

While I do think \hat{SIS} is a good intuitive approximation to SIS, does Theorem 3.1 rigorously supports that \hat{SIS} is a good proxy for SIS? For now, it describes only the lower bound of SIS - \hat{SIS} and not the upper bound, so it seems like Theorem 3.1 has little value. Examining the proof of Theorem 3.1, I don't think upper bound can be easily attained (e.g. expanding two-sided Hoeffding inequality doesn't seem to work since the information SIS \geq SIS_{\hat{q}} is only useful for the lower bound). Can you also elaborate on what “Generalization of SIS” means in Theorem 3.1?

The gap between \hat{SIS} and SIS has two sources: (i) the finite test sample, and (ii) using the fitted prior q instead of the supremum over all input-independent priors.

  • Theorem 3.1 primarily addresses (i): it guarantees that the agreement measured on a finite test set generalizes to unseen data from the same distribution. This is what the "generalization" in its name refers to. If we write SIS_q for the population agreement of the fitted bottleneck mode, Hoeffding's inequality in fact holds in both directions for SIS_q (but not for SIS). So \hat{SIS} is a two-sided estimate of the agreement of the concept-only predictor we actually obtain. We ask the reviewer whether they would prefer to see this in the paper: in that case, we can state a two-sided version explicitly in the revised pdf by making the theorem give Hoeffding bounds on SIS_q, with the current lower bound on SIS as a corollary, to make this more clear.
  • Source (ii) is one-directional by definition: q is one admissible prior, so SIS >= SIS_q. This also means that SIS >= SIS_q's lower bound (which is a function of \hat{SIS}). This is why the theorem currently gives a lower bound on SIS.

As the reviewer notes, an upper bound on SIS cannot follow from test data alone. It would require certifying that *no* input-independent prior does better, i.e. solving the supremum globally. SIS with a fitted prior is a certificate: it guarantees that a concrete concept-only predictor reproduces the model's decisions on at least this fraction of inputs (up to the finite-sample term of Theorem 3.1), and a better prior can only increase it. Empirically, we performed a new stability analysis that shows that \hat{SIS} generally barely changes across refits (see Appendix D.2).

More fundamentally, in practice a CSM is only as representation-interpretable as the best concept-only predictor one can "actually obtain", not one that "merely exists". Below, we argue that, while SIS captures what exists in principle, the approximated SIS captures what can be obtained in practice, and it is the latter that determines how well a model can actually be interpreted. If no input-independent replacement can be found that reproduces the model's predictions, the model cannot be used or explained through its concepts, even if such a replacement exists in principle. The same standard is common elsewhere: a network may contain a small subnetwork that matches its performance (Frankle & Carbin, 2019), but it only becomes simpler once such a subnetwork is found. In mechanistic interpretability, a model is explained to the extent that a discovered circuit reproduces its behavior, typically measured by the fraction of performance the circuit recovers (Wang et al., 2023; Conmy et al., 2023). SIS follows this view: it certifies the agreement achieved by a concrete concept-only predictor, the bottleneck mode with the fitted prior. A better fitting procedure can only increase the approximated SIS. If follow-up work can find better ways to fit the prior, CSMs will become more representation-interpretable in practice, and the approximated SIS will reflect this.

 

Frankle & Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ICLR 2019.

Wang, Variengien, Conmy, Shlegeris & Steinhardt, Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023.

Conmy, Mavor-Parker, Lynch, Heimersheim & Garriga-Alonso, Towards Automated Circuit Discovery for Mechanistic Interpretability, NeurIPS 2023.

How are the hyperparameters \alpha and \beta in SIS regularized training objective set? How do these hyperparameters affect the training? The paper seems to lack details on training configurations when using SIS regularization.

Appendix D.1 already states the details on training configurations, including beta. For instance, lines 929-935 describes the grid search over the values of beta for our SIS regularization, and in the same appendix we describe e.g. that we use the AdamW optimizer, our learning rate, number of training epochs, etc. We apologize as we forgot to write that we chose alpha=1, and we've added this in the revised version. With this choice, the first two terms become similar to the joint log-likelihood log p(y, c | x) of the CSM (except that we use c_hat, the predicted concepts, as common in CBMs). Note that the gradient from the task supervision does not backpropagate through the concepts, which otherwise would let small alpha's encourage concept leakage (like with joint training).

 

The authors state that “we show that this is a weaker criterion than minimizing sidechannel reliance, as a model could fully maximize its reliance on concepts while still relying heavily on the sidechannel to form the final prediction.” (line 80-82), but I’m not sure where this point is shown. Can you clarify where your point was shown?

We discuss and provide evidence for this distinction in the following parts of the paper.

  • In Section 4 (lines 345-348), we explain why high intervenability does not necessarily imply low sidechannel reliance: a model can rely strongly on both concepts and the sidechannel simultaneously. Thus, encouraging concept usage does not guarantee that the sidechannel becomes unnecessary for making predictions.
  • In Section 5 (Table 1), we empirically compare SIS regularization against two approaches designed to encourage concept usage (CRM* and CRM**). While these methods improve SIS, they generally achieve smaller improvements than directly regularizing sidechannel reliance. As a response, we have also added the Pareto fronts comparing CRM* and CRM** vs CRM with SIS regularization in Appendix D.2 (Figure 18).

The central point is that concept reliance and sidechannel reliance are not mutually exclusive. Consequently, optimizing for concept usage alone does not necessarily minimize sidechannel reliance, whereas SIS regularization directly targets the latter.

To clarify, we have added a sentence in the introduction to explicitly reference Sections 4 and 5.

Reviewer_cpkEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a unified framework to unify and study various side channels used to augment conventional Concept Bottleneck Models. The paper presents the concepts of functional interpretability and representational interpretability. For CSM models, it then defines a "default" mode where the CBM channels and the side channels are used and a "bottleneck" mode when the side channels are disabled. They then define the "Sidechannel Invariance Score (SIS)" which captures how much the output classes of a CSM model differ between the default and bottleneck modes of operation. The paper proposes adding an SIS regularization term while training a CSM model to give it a tradeoff between accuracy and interpretability. Confirmatory or illustrative results are shown for CelebA and MNIST-Add datasets to show multiple options along an accuracy-interpretability frontier.

Critical strengths

The analysis of a CBM model using functional and representational interpretability is interesting. The unified framework for all CSMs appear useful too. The SIS framework is defined well and its utility is illustrated using a couple of datasets.

Critical weaknesses

While SIS and a regularisation loss using it are interesting, its definition is difficult to be very practical. As a result, the framework presented remains more of a theoretical framework that is interesting but of unknown practical use. The input-independent distribution p(z)p(z) also appears conceptual that could be difficult to estimate in a general situation. The experiments presented are also on rather simple datasets, perhaps inadequate these days.

I also don't understand how a model builder can choose an appropriate point in the accuracy-interpretability frontier. How can we settle at different points of the loss function given in L293? Figure 4 shows examples of "intervenability"; how does one use them? Is it by using different β\beta constant values during training? That appears strange.

Questions for authors

Please address the issues mentioned in the weakness section.

2 comments

We thank the reviewer for taking the time to read our paper. Below, we respond to the reviewer's concerns. In the revised paper pdf, we have included additional results that are also briefly highlighted below (changes are in blue).

 

While SIS and a regularisation loss using it are interesting, its definition is difficult to be very practical. As a result, the framework presented remains more of a theoretical framework that is interesting but of unknown practical use.

The input-independent distribution p(z) also appears conceptual that could be difficult to estimate in a general situation. The experiments presented are also on rather simple datasets, perhaps inadequate these days.

We respectfully but strongly disagree with the reviewer. The review does not identify a concrete aspect of SIS that would make it impractical, and our results show the opposite. SIS and SIS regularization are:

  • easy to implement, as SIS replaces z by an input-independent p(z) and counts agreeing predictions, and the regularizer adds one divergence term to the loss.
  • cheap, as it requires one extra pass through the task predictor per batch.
  • simple to estimate, as p(z) is not a density to be estimated (e.g. it does not need to approximate the true marginal). It is a single learned parameter (e.g. a vector, a categorical over rules, etc.).
  • generally stable: we performed additional experiments with random restarts of the learnable prior (see Appendix D.2).
  • general, as SIS and its regularization are model-agnostic, meaning that they work for any existing CSM.

Regarding the datasets: CelebA, MNIST-Addition and CUB (200 classes, 112 concepts) are standard benchmarks for concept-based models, and our evaluation spans over 3.000 trained models across five architectures and three seeds.

Thus, our experiments directly demonstrate the practicality of our approach. We would welcome any concrete concern about practicality and would be happy to address it.

 

I also don't understand how a model builder can choose an appropriate point in the accuracy-interpretability frontier. How can we settle at different points of the loss function given in L293? Figure 4 shows examples of "intervenability"; how does one use them? Is it by using different beta constant values during training? That appears strange.

Regarding choosing a point on the frontier: β is the same type of hyperparameter as for any regularization strength (e.g. the weight of an L1 penalty). In practice, one would train a small β sweep and select on validation data according to the application’s requirement. Typically, this would mean maximizing SIS while staying close to the best achievable accuracy. This region of the frontier is the most relevant one, as models that are interpretable but inaccurate already exist (e.g. standard CBMs), and the purpose of CSMs is to add accuracy without giving up (too much) interpretability. As Table 1 and Figure 3 show, SIS regularization moves models along this region, often reaching high SIS at a small accuracy cost.

SIS also has value on its own, independent of the frontier. SIS certifies, at the decision level, the fraction of inputs on which a concept-only predictor (bottleneck mode) makes the same decision as the model. This predictor can be inspected. On those inputs, the model’s decision can therefore be obtained and interpreted from the concepts alone. Theorem 3.1 assures bounds also on unseen data. This is directly relevant to trust: warranted trust requires that users can anticipate the model’s decisions (Jacovi et al., 2021), and effective human-AI collaboration requires knowing where the model’s behavior departs from the user’s understanding (Bansal et al., 2019). A model with SIS = 0.8 has decisions that a concept-only predictor reproduces on 80% of inputs. With SIS = 0.1, this holds for only 10% of inputs, even if both models are equally accurate.

Figure 4 shows how the regularization changes the model’s response to concept interventions, where a user corrects some predicted concepts at test time. With SIS regularization, such corrections propagate to the task prediction and improve accuracy much more, since the model relies on its concepts rather than on the residual. This is the practical use: a domain expert can fix the model’s mistakes by fixing its concepts. Different β curves are different trained models, trained with different regularization strengths β.

 

Jacovi, Marasović, Miller and Goldberg. Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. FAccT 2021.

Bansal, Nushi, Kamar, Lasecki, Weld, Horvitz. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. HCOMP 2019.

Dear reviewer,

Please let us know if you have any further questions or things we could clarify further. If not, we would appreciate if you could consider updating your review based on our replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.