- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Res-SCBM, a concept bottleneck model that models concept dependencies between observed concepts and unobserved residuals via their covariance. Thereby, the method combines the direction of intervention-aware CBMs with residual CBMs. By capturing the dependencies between concepts and the residual, the model can propagate intervention on known concepts to the unobserved but potentially correlated residuals, and thus has improved intervention effectiveness in settings with incomplete data.
Critical strengths
- The paper combines two mostly distinct directions in the area of CBMs: CBMs that handle interventions effectively and residual CBMs. How interventions can and should affect residuals is so far still underexplored in the CBM literature, and the paper takes a first step in this direction.
Critical weaknesses
- While the paper extends the existing SCBM with a residual and combines this with an intervention-aware training objective for CEMs, this makes quite strong assumptions about the way the residual connection is modeled and how the concepts are encoded. While this enables the propagation of interventions in the case of Res-SCBM, this modeling choice is not compared against other approaches with residual connections that are potentially more flexible.
Questions for authors
Major Questions/Suggestions:
Q1: Structure of the residual:
- As far as I understand, no part of the training particularly encourages the residual to capture different information from the concepts, is this correct? L. 254 states that the intervention-aware loss might encourage this, could the authors please explain this in more detail?
Q2: Exploration of the residual connection:
- I think the design choice and effect of the residual connection are not sufficiently explored. There are several works that include residual connections in CBMs, e.g. [1,2], see also [3, 4] for more general overviews. These are usually introduced exactly for the incomplete concept setting that is discussed in this work, but this work does not compare itself to these methods. A discussion of how the modeling choice affects the residual connection, and an experimental evaluation, is necessary in my opinion (I understand that the setting is not directly comparable to all residual CBM methods, but at least one or two other methods should be evaluated, in combination with a broader discussion).
- The width of the residual connection of 20 is far below the number of removed concepts in both datasets. Can the authors discuss why this number is relatively small, and also provide a bit more analysis on how the model behaves in this context? I would assume that a higher residual capacity (at least to some extent) would improve the performance of the model, but the experiment in Fig. 3 seems to indicate otherwise. In particular, in the context of these results, a more detailed analysis would be necessary (and also a comparison to how other residual methods work with different residual capacity).
Q2: Results in Tab. 1:
- Why is ResCBM better than the incomplete linear head on AWA, but substantially worse on CUB? (or the other way round, why is the linear head in CUB so good given that it has access only to very few concepts - are the CUB concepts used in the majority-class voting fashion that is often done?)
- On CUB incomplete, there is a substantial performance difference between Res-SCBM w/o and Res-SCBM, where does this come from, given that there are no interventions involved yet?
- I am a bit confused about the performance of the blackbox CBMs / baselines, in particular on CUB. Both the initial CBM paper [5] and CEM [6] report higher values for their baseline DNNs (83% and 74%, respectively). The reported DNN performance in this work of 68% is substantially lower. Where does this come from?
- Furthermore, CEM reports a performance on CUB of about 76%, which differs substantially from the performance of 70% reported in this work. Can the authors explain these differences?
Q3: Interpretation of results of Tab. 2 (l.356)
- Why does the high concept accuracy of Res-SCBM imply that the residual paths learn complementary information? They could just learn redundant concept information and encode the same information again (in particular in the complete tasks, where there is little other beneficial information to learn).
Q4: Intervention performance Fig.2:
- When adding interventions on AWA, the concept ROC goes down initially after intervening up to ~20 concepts. I assume this is the case due to not properly learned covariance between concepts? This is concerning given that interventions on about 20% of the data are necessary to not make the concept accuracy worse than before, which goes a bit against the idea of interventions.
- Also regarding the task accuracy, the interventions on AWA are less effective compared to all baselines. Can the others explain why this is the case and provide an idea of how to mitigate this?
- How does this effect connect to the evaluations in Fig. 3? (i.e. how would the curves of Fig. 3 look for the AwA dataset?) I would assume that the effect partially comes from the residual (as the effect is less pronounced for SCBM), and seeing how the residual capacity affects this would be helpful to understand these results better.
Q5: Analysis of the residual covariance/redundancy (RQ4): While the qualitative analysis of the residual covariance in Fig. 4 is interesting, I think the evaluations in this paper should go further, given that this is the main distinguishing factor of Res-SCBM compared to other residual methods. Therefore, the following evaluations would be of great interest to this work and would shed more light on how the residual functions and how the covariance analysis can bring benefits:
- How redundant is the information in the residuals given the concepts? Relating this to the learned covariance would be further interesting.
- How much does the residual information align with the concepts not present in the dataset (i.e., the removed but necessary concepts)? Both of these should also contain quantitative, not just qualitative, analysis.
Minor Points:
- As far as I understand, Interventions are performed on the logits, so that the conditional activations can be considered, and the actual values of the concepts are then sampled based on the conditional activations. Could this not in principle result in a concept being sampled as the opposite of the intervened value if the logits are the 5th or 95th percentile of its activations? If so, does this happen often in practice?
- For the intervention-aware objective: Why are all concepts intervened on during training? I would consider interventions rather sparse in practice, and thus it would be good if the model can handle sparse interventions as well?
- Related to that: What is in Eq. 6?
- Given that there are some works that started to incorporate specific structure into the concept space, for example in hierarchy [7] or object-centric [8] ways, how could Res-SCBM be adapted to such settings?
- Why do you think CEM performs so well in incomplete settings, despite not having explicit residual connections?
- It would also be cool to have an evaluation on some datasets with naturally incomplete concept sets, as opposed to datasets that were artificially reduced, but I understand that it is more difficult to find such datasets with sufficient concept annotations for the remaining concepts, so this is more a nice-to-have than a requirement.
[1] Shang, Chenming, et al. "Incremental residual concept bottleneck models." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024. [2] Martínez-García, María, et al. "A probabilistic hard concept bottleneck for steerable generative models." International Conference on Learning Representations. Vol. 2026. 2026. [3] Zabounidis, Renos, et al. "Benchmarking and enhancing disentanglement in concept-residual models." arXiv preprint arXiv:2312.00192 (2023). [4] Knab, Patrick, et al. "What’s in the bottle? a survey and roadmap of concept bottleneck models." Transactions on Machine Learning Research (2026). [5] Koh, Pang Wei, et al. "Concept bottleneck models." International conference on machine learning. Pmlr, 2020. [6] Espinosa Zarlenga, Mateo, et al. "Concept embedding models: Beyond the accuracy-explainability trade-off." Advances in neural information processing systems 35 (2022): 21400-21413. [7] Hill, Oscar, Mateo Espinosa Zarlenga, and Mateja Jamnik. "Hierarchical concept-based interpretable models." International Conference on Learning Representations. Vol. 2026. 2026. [8] Steinmann, David, et al. "Object-centric concept-bottlenecks." Advances in Neural Information Processing Systems 38 (2026): 68899-68924.
6 comments
[Q3.4] Differences in CEM Performance Compared to Prior Work
For fairness when evaluating against SCBMs, our CEM baseline uses the same hyperparameters and training procedure as those used in Vandenhirtz et al. [7]. We have clarified this in Appendix A.3. Our CEM results on CUB are comparable to those reported in that work.
[Q4.1] Complementary vs Redundant Information in the Residual Channel
We agree that high concept accuracy alone does not establish that residuals capture complementary rather than redundant information. Our intention was to show that improved task accuracy does not compromise concept prediction quality. On concept-incomplete datasets, Res-SCBM outperforms concept-only baselines with comparable concept accuracy and even surpasses a ground-truth concept classifier on AwA2-Incomplete, suggesting that residuals capture additional task-relevant information. We have clarified this distinction in Sections 5.1 and 5.3.
[Q5.1–Q5.3] Intervention Effectiveness and Residual Capacity
The smaller improvements in task accuracy on AwA2 may partly reflect the residual pathway, which captures additional predictive information and can reduce the influence of concept corrections. To investigate this, we evaluated an uncertainty-based intervention policy that corrects the most uncertain concepts first. This improves intervention performance on both AwA2 datasets, largely eliminating the initial concept ROC-AUC drop on AwA2 and reducing it on AwA2-Incomplete. This suggests that the initial drop partly reflects random concept selection rather than poorly learned dependencies. Results are shown in Section 5.2 (Figure 2), with original random-policy results retained in Appendix B.4.
To further investigate residual capacity, we compared and on AwA2-Incomplete, with and without (Appendix B.1). Res-SCBM remains responsive even with 500 residual dimensions, whereas the variant without shows little improvement under interventions. This demonstrates the importance of for preserving intervenability as residual capacity increases.
[Q6] Analysis of Residual Covariance and Redundancy
We thank the reviewer for these suggestions. We agree that quantifying residual redundancy and alignment with omitted concepts would provide valuable insights into the learned representations.
We explored several approaches to identifying omitted concepts in the residuals, including training linear probes to predict removed CUB attributes from the learned concept and residual representations, and comparing their performance against shuffled concept labels. While these experiments suggest that residuals contain information predictive of omitted concepts, distinguishing this from information associated with class identity remains challenging.
We therefore leave a more systematic quantitative analysis of residual redundancy, alignment with omitted concepts, and their relationship to the learned covariance structure for future work. We have added this to the Limitations & Future Work section.
Minor Comments
-
Reliability of Logit-Based Concept Interventions: The 5th/95th percentiles define intervention logits for Gaussian conditioning and propagation. Intervened concepts are fixed to ground-truth values (0 or 1) before Bernoulli sampling, while only non-intervened variables are sampled. Thus, concept accuracy under full intervention is exactly 100%.
-
Training with Full vs Sparse Concept Interventions: In , we intervene on all concepts during training, following [1], encouraging residuals to capture task-relevant information beyond the observed concepts. While partial-intervention training may better reflect practical settings, our results show that Res-SCBM remains responsive to sparse corrections. In Eq. 7 of the updated manuscript, denotes the task loss, implemented as cross-entropy.
-
Extending Res-SCBM to Structured Concept Spaces: Extending Res-SCBM to hierarchical representations such as HiCEMs [8] is challenging because our model uses scalar concepts rather than learned embeddings, and Gaussian conditioning does not enforce hierarchical consistency. Such an extension would require architectural changes, which we leave for future work.
-
CEM Performance in Concept-Incomplete Settings: CEM [6] uses high-dimensional, sample-dependent concept embeddings that can encode task-relevant information beyond supervised concepts, effectively providing implicit residual capacity. This allows CEM to maintain strong predictive performance under concept incompleteness without an explicit residual pathway.
References
[1] Espinosa Zarlenga, "Position: In Defense of Information Leakage in Concept-Based Models." ICML, 2026. [2] Havasi et al., "Addressing Leakage in Concept Bottleneck Models." NeurIPS, 2022. [3] Espinosa Zarlenga et al., "Learning to Receive Help: Intervention-Aware Concept Embedding Models." NeurIPS, 2023. [4] Koh et al., "Concept Bottleneck Models." ICML, 2020. [5] Espinosa Zarlenga et al., "Avoiding Leakage Poisoning: Concept Interventions Under Distribution Shifts." ICML, 2025. [6] Espinosa Zarlenga et al., "Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off." NeurIPS, 2022. [7] Vandenhirtz et al., "Stochastic Concept Bottleneck Models." NeurIPS, 2024. [8] Hill et al., "Hierarchical Concept-Based Interpretable Models." ICLR, 2026.
Thank you for your thoughtful feedback. We are glad you see value in exploring how concept interventions can affect residual variables, especially since this remains underexplored. We appreciate your suggestions and have addressed them below. Please let us know if you have any further concerns. Otherwise, we would sincerely appreciate it if you would consider updating your review in light of our replies below. For clarity, as the reviewer labels two sets of questions as Q2, we refer to the second Q2 as Q3 and increment the numbering of all subsequent questions accordingly.
[Q1] Structure of the Residual
We agree that our objective does not explicitly enforce non-redundancy. However, Lint implicitly encourages residuals to capture complementary information by minimising task loss using corrected ground-truth concepts and conditioned residuals. Since the observed concept information is already provided, residuals are encouraged to capture additional predictive information rather than duplicate it (see [1] for an information-theoretic justification).
[Q2.1] Comparison with Existing Residual CBM Methods
We thank the reviewer for this suggestion. We agree that additional baselines would strengthen the experimental evaluation. To address this, we have added two baselines, namely Hard Autoregressive CBM with a side-channel [2] and IntCEM [3], whose evaluations are now included in Tables 1 and 2 (Section 5.1), Figure 2 (Section 5.2), and Figure 11 (Appendix B.7). The former is particularly relevant as it combines explicit concept dependencies with a residual pathway, allowing us to compare against an alternative architecture incorporating both components. While IntCEM does not have an explicit residual channel, its high-dimensional concept embeddings provide additional representational capacity that can implicitly encode information beyond the supervised concepts, effectively acting as a form of residual information. Our updated results show that Residual SCBM outperforms both models in task accuracy while maintaining competitive mean concept ROC-AUC. In terms of interventions, Res-SCBM remains responsive to concept corrections and achieves competitive performance with Hard AR with side-channel. IntCEM generally achieves higher task accuracy as more concepts are intervened on, potentially due to its learned intervention policy. Moreover, we have expanded our related work discussion to better position our approach relative to existing residual-based CBMs.
[Q2.2] Choice of Residual Width and Effect of Residual Capacity
To assess the effect of residual capacity, we added an ablation on CUB-Incomplete with R∈{20,50,100,500} (Appendix B.6). Res-SCBM remains highly responsive to concept interventions across all widths, even when residual capacity greatly exceeds concept capacity. Comparing against a variant without Lint further shows that intervention-aware training becomes increasingly important as residual capacity grows.
[Q3.1] Differences Between Ground-Truth Concept Performance
The difference reflects how much predictive information the observed concepts retain. On CUB, we use majority-voted class-level annotations [4], meaning all images of a species share concept labels. The 22 retained concepts are highly informative, explaining the strong performance of a ground-truth concept classifier, whereas image-based models introduce concept prediction errors. In contrast, the 9 retained concepts in AwA2-Incomplete are less informative, allowing Res-SCBM to outperform the ground-truth concept classifier by capturing additional information through residuals.
Importantly, ground-truth concept accuracy serves as a reference for the information available in observed concepts, not an upper bound on Res-SCBM's performance.
[Q3.2] Effect of the Intervention-Aware Loss on Initial Task Accuracy
Although Lint is designed to encourage responsiveness to concept interventions, it still penalises task errors under full intervention, influencing the learned representations and classifier. This additional training signal may explain the improved initial accuracy on CUB-Incomplete, similar to effects observed with IntCEM [3]. Quantifying improvements in predictive performance independently of intervenability remains future work.
[Q3.3] Differences in DNN Performance Compared to Prior Work
Our DNN follows the architecture in [5] but uses a smaller hidden dimension of C+20 to match Res-SCBM's capacity. To address this difference, we added a less constrained DNN with C+200 hidden dimensions (Table 1), following [5]. Its performance is closer to their reported results, suggesting that the discrepancy largely stems from architectural differences.