- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a information theory-based mathematical formulation for leakage caused by the joint optimization of CMs. Based on their formulation, the authors prove that the joint training scheme optimizes concept to task leakage, a fact supported by theirs and previous work's empirical results. Then, they introduce the Gradient Reversal Layer into CBMs, yielding Gradient-Cancelled Concept Based Models (GC-CBMs), which provide an end-to-end equivalent of sequential CMs. Based on their analysis GC-CBM achieves the same minima as sequential CBMs, while being end-to-end and avoiding concept to task leakage. Lastly, they empirically show that GC-CBM outperforms jointly trained CBMs and CEMs in concept accuracy and intervenability, and performs similarly to sequential training while being end-to-end.
Critical strengths
Thank you so much for submitting this work! I enjoyed reading the analysis, and I appreciate the time taken to write it up and submit it to ICLR. Below are what I believe are this paper’s main strengths:
- S1: Mathematical analysis and proof of concept-to-task leakage created by joint training. Also the authors mathematically explain the interaction between concept incompleteness and leakage. Both of these phenomena have been observed in previous empirical works and this work complements them by adding mathematical rigor.
- S2: They propose new metrics RTL and RCL, to account for the bias of estimating MI in high dimensions seen in the previous metrics concept-to-task (CTL) and Interconcept Leakage (ICL) [1].
- S3: They show that it is possible to train sequential Concept-based Models in one go, avoiding the need for two steps. Then they show that sequential CBMs and GC-CBMs (and GC-CEMs) can outperform their jointly trained counterparts in concept accuracy and intervenability. Lastly, by tackling concept to task leakage, they also reduce interconcept leakage.
[1] Parisini, Enrico, et al. "Leakage and interpretability in concept-based models." Journal of Machine Learning Research 27.174 (2026): 1-40.
Critical weaknesses
In contrast, I believe the following are this work's major limitations:
GRL
- W1: Given that gradient cancellation happens only for and , both of which are strictly enforced, this leads to a zero gradient. Unless I am missing something, isn't is possible to just set that gradient to zero in the code (e.g. using detach in PyTorch) and achieve the same result? This eliminates the need of using GRL, thus avoiding software, architectural and optimization overhead and complexity of having two task heads (e.g., GC-CBM training takes longer to converge (Line 1375)). Also, given that the authors don't evaluate partially allowing for leakage (i.e. ), I really don't understand the use of GRL.
Experiments
-
W2: Unless I missed it, the authors did not empirically investigate GC-CBM in a concept incomplete dataset, even though the mathematical analysis of concept incompleteness and leakage. Toy concept incomplete datasets include incomplete FMNIST [2], and a synthetic tabular dataset [3,4].
-
W3: Given that one of the main comparisons in this work is GC training against sequential training, I would expect in the CEM part of the experiments to see a sequential CEM baseline.
Presentation
- W4: One of the main presentation issues is the fact that the figures in the main text are generally too small and difficult to read. Fig. 1, for example, could simply have been presented at a larger size to improve readability. Fig. 2 also leaves a substantial amount of unused space (due to wrapfigure). More generally, the figures could be decluttered. Finally, the resolution (DPI) of some of the figures appears to be rather low and should be improved for readability and presentation quality.
Minor Comments
- Please provide some interpretation of the RTL and RCL metric values and what they imply in practice. For example, in Fig. 6, CEM appears to have a lower RCL on CUB, but the absolute values are quite small. It would be helpful to discuss whether this difference is practically significant, or whether it is negligible and therefore does not represent a meaningful disadvantage of GC-CBM.
Clarity
- Please state in the main text that the interventions are group-interventions (as stated in App. J1). Also, for Fig. 4 and Fig. 7, I think it would help to show points instead of continuous lines since the x-axis is discrete.
- Line 108: "Hard CBMs by construction cannot encode any leaked information: they are trained independently, ..." there are works in which hard CBMs are being trained end-to-end (e.g., [2, 5]). Please specify only in the experiments that the hard CBMs are trained independently.
Experiments
- Lines 343-346. "Concept accuracy is competitive with joint and sequential CBM across all datasets and in some cases outperforms them (for AwA2 and CUB against both references for all λc). ". In all cases, Seq. CBM and GC-CBM seem to have the same concept accuracy (as also seen in the tables of App. K) and as also stated in Proposition 1. Please rephrase this to make the distinction between joint and sequential clearer.
- Lines 360-362: there is a bigger drop in CTL in TabularToy but accuracy remains almost the same. Please be more accurate in the wording, or further clarify on why this statement does not hold in TabularToy.
- Lines 375-377: As seen in Fig. 4, all but one joint CBM performs equally good to the rest of the models in interventions. Please rephrase and specify that the comparison is against the worst-perfoming Joint CBM.
Typos
- The captions of Figures 2,3,9 and Tables 2, 5, 6, 7 are missing a period at the end.
- Missing reference to Fig. 4 in paragraph "Gradient cancellation makes concept interventions more effective."
[2] Kalampalikis, N., Gupta, K., Vitanov, G., & Valera, I. (2026). Towards Reasonable Concept Bottleneck Models. Transactions on Machine Learning Research. Retrieved from https://openreview.net/forum?id=0TxwJMhXBY
[3] Dominici, G., Barbiero, P., Espinosa Zarlenga, M., Termine, A., Gjoreski, M., Marra, G., & Langheinrich, M. (2025, May). Causal concept graph models: Beyond causal opacity in deep learning. In International conference on learning representations (Vol. 2025, pp. 55102-55131).
[4] Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Chin-Cheong, K., Paschke, A., Zerres, J., ... & Vogt, J. E. (2024). Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitis. Medical image analysis, 91, 103042.
[5] Vandenhirtz, Moritz, et al. "Stochastic concept bottleneck models." Advances in Neural Information Processing Systems 37 (2024): 51787-51810.
Questions for authors
Clarity
- Unless I missed it, it is not stated what value is used. I assume it is . Regardless of its value, please state it in the main text.
- What is the concept representation used in GC-CBM? Fig. 1 shows that the logits are propagated to the task. Furthermore, what intervention values are used? The ground truth binary values, or (e.g.) 5th and 95th activation percentiles?
- In line 295, I assume is the concept embedding's dimensionality?
- In App. J.2 I noticed that different number of training epochs is used per model. I assume that this is due to early stopping or due to the models converging earlier?
Methodology
-
Why is there a clamp to zero for the RTL and RCL metrics? What would negative mean?
-
Why is GC-CBM's CTL lower than an independently trained hard CBM (and the Sequential CBM)? Given lines 108-109 "Hard CBMs by construction cannot encode any leaked information: they are trained independently...", and previous works, I would expect it to be equal at worst. On top of that, why aren't the CTL values of hard, sequential and GC-CBM zero?
Experiments
-
This work introduces alternatives to the CTL and ICL metrics intended for higher dimensionalities (e.g., concept embeddings). As a sanity check and as a comparison to the pre-existing metrics, can you provide both sets of metrics for both the CBM and CEM experiments?
-
An easy experiment, where I think GC-CBM would really shine is a comparison on TabularToy, against a joint CBM that achieves almost 100% concept accuracy, to show that leakage may still happen for the joint CBM.
-
An interesting ablation would be different values, and quantifying the benefit of allowing any leakage, and thus providing a use case of GRL other than complete gradient cancellation.
My final rating is based on the use of GRL, the overall clarity of the manuscript, and the incompleteness of the experimental evaluation. I believe the strongest aspect of this work is its mathematical formulation of concept-to-task leakage. I would be happy to increase my score if the authors address the concerns raised above.
4 comments
Concept-incomplete data. We added a concept-incomplete experiment on the synthetic tabular setting suggested by the reviewer. In TabularToy, , and we withhold (2 of 3 concepts observed) or (1 of 3); 3 seeds (Table R6, Fig. R5). Since concepts are binary, the best accuracy attainable from the observed ground-truth concepts is exact: 80.4% on our test set. Any accuracy above it requires the withheld concept to pass through the bottleneck.
- Jointly trained models exceed this ceiling by about 20 pp (99.5–99.8% with 2 of 3 concepts), through leakage. Joint CBM encodes the withheld concept in its probabilities: its concept accuracy drops to as low as 49.6%. CEM hides it in its embeddings at 99.8–99.9% concept accuracy: RTL 0.05–0.15.
- GC-CBM and GC-CEM stay at the ceiling (79.6–82.3%, within the ceiling's sampling error of about 1.3 pp), with GC-CEM's RTL , like Hard and Sequential CBMs.
Joint CBM with near-100% concept accuracy on TabularToy. We trained joint models at ( withheld, 3 seeds; Table R5, Fig. R4).
- Scalar CBMs: a high concept weight does suppress leakage. Joint CBM reaches 99.8% concept accuracy and stays at the ceiling, like GC-CBM.
- CEMs: it does not. CEM reaches 99.9% concept accuracy yet 93.9% task accuracy (13.5 pp above the ceiling), with RCL 0.17–0.20 that grows with .
- GC-CEM: RCL 0.003 and RTL 0.001.
High concept accuracy is therefore not evidence of low leakage for embedding models, which is exactly where RTL/RCL are needed.
Q. Both metric sets for CBMs and CEMs. Tables R5 and R6 report CTL and ICL for all models, and RTL and RCL for the CEMs. The two families agree in direction (CEM > GC-CEM), but CTL and ICL on CEM concept probabilities vary far more across seeds: at , CEM's CTL ranges from 0.02 to 0.27 over 3 seeds, while its RCL stays within 0.14–0.21 across all 9 runs. This is the estimation problem in high dimensions that motivated RTL and RCL.
Q. Why is GC-CBM's CTL below Hard/Sequential CBM, and why is it not zero?
- Lower than Sequential. GC-CBM's encoder is trained exactly as Sequential stage 1. Under matched hyperparameters the two therefore have identical CTL. The gap in the paper therefore reflects optimisation differences between separately trained runs, not the training scheme; on CUB, for example, our Sequential CBM used a different optimiser and learning rate. We will report the matched comparison.
- Not zero. CTL is a KSG estimate of task information in the soft concept probabilities beyond the ground-truth concepts. A concept-only encoder is not rewarded for carrying task information, but it is not prevented from carrying it either: its probabilities depend on the input, e.g. its confidence varies with the class. KSG also has a positive finite-sample bias. Hard and Sequential CBMs, which receive no task gradient at all, have the same non-zero floor (CTL 0.17 on concept-incomplete TabularToy, Table R6). CTL should therefore be read relative to this floor: joint CBMs reach 0.69–0.73 there.
Q. Why clamp RTL/RCL at zero? A negative held-out means the probe predicts a residual dimension worse than its mean, i.e. finite-sample noise rather than "negative leakage"; the conditional MI that RTL/RCL approximate is non-negative. We clamp so that noise on uninformative dimensions cannot cancel genuine leakage on others when averaging.
Minor. Interpreting RTL/RCL values. After removing the -conditional mean, is the residual variance of concept 's embedding that the task label explains, per embedding dimension and in units of the embedding's total variance: a variance-explained analogue of . is the analogue of . For Gaussian residuals, per dimension. On CUB (Fig. 6), RCL is 0.007–0.008 for CEM and 0.009–0.012 for GC-CEM: about 1% of the embedding variance, i.e. below 0.01 nats per dimension. As the reviewer suggests, this difference is practically negligible: it is an order of magnitude below CEM's RCL on TabularToy (0.09–0.12), and it comes with a 40–51% lower RTL and higher concept accuracy for GC-CEM on CUB. We will add this interpretation.
Clarity questions.
- Concept representation in GC-CBM: the sigmoid concept probabilities are passed to the task head. Fig. 1 shows logits, and we will correct it.
- Intervention values: interventions set the intervened concepts to their ground-truth binary values (0/1).
- Intervention type: all interventions are group interventions (App. J.1); we will state this in the main text.
- in line 295: yes, is the concept-embedding dimensionality.
- Epochs in App. J.2: the numbers differ because the models converge at different rates on CUB.
W4 and presentation. We thank the reviewer for the detailed suggestions and will implement all of them in the final version:
- larger, decluttered and higher-resolution figures (Fig. 1 enlarged, the wrapfigure in Fig. 2 removed);
- markers instead of lines for the discrete -axes of Figs. 4 and 7;
- the missing reference to Fig. 4, and the missing periods in the captions of Figs. 2, 3 and 9 and Tables 2, 5, 6 and 7.
Wording. We will revise the passages the reviewer points out:
- Line 108: we will state that Hard CBMs are trained independently in our experiments, noting that end-to-end hard CBMs exist [2, 5].
- Lines 343–346: we will state that GC-CBM matches Sequential CBM in concept accuracy, as Proposition 1 predicts, and outperforms joint CBMs.
- Lines 360–362: we will qualify the claim for TabularToy, where CTL drops more while task accuracy is unchanged.
- Lines 375–377: we will specify that the intervention gain is relative to the worst-performing Joint CBM.
We are grateful for the reviewer's offer to reconsider the score. We believe the main concerns are now addressed:
- GRL: we agree it reduces to a stop-gradient, and the contribution is the analysis, the metrics and the evidence that this gradient is the source of leakage.
- Completeness: we added concept-incomplete data, a Sequential CEM baseline, high-concept-weight joint models and a ablation on CUB, plus CIBM, MixCEM and RandInt.
- Clarity: we answered each question above.
I thank the authors for their detailed responses. Unfortunately, as the rebuttal was submitted somewhat late, I have only had time to skim through it. Nevertheless, it appears that most of my questions have been addressed and that the added experiments showed positive results. Here are my few remaining concerns:
GRL vs Detach and lambda values
As the authors themselves point out, when , joint training can achieve similar results. Furthermore, when , a simple stop-gradient operation in the code would suffice to achieve the same effect. Therefore, I believe the authors should consider removing the GRL entirely. I understand that this would constitute a major change to the proposed method.
I would also suggest the authors discuss why a practitioner might prefer to set , if there are any practical reasons for doing so.
Lastly, I believe an ablation study on different values of , in incomplete settings, would have been very informative.
Comments on presentation
I noticed that almost none of the presentation-related comments I raised were addressed. Unfortunately, I still believe that the current presentation of the work falls below the standard expected at ICLR.
Conclusion
Some of the experiments introduced in the ablation studies could have been incorporated into the main paper. ICLR allows an additional page during the rebuttal period, and I would strongly recommend that the authors use this opportunity to highlight the changes they have made in the revised PDF.
I believe this work has the potential to become a seminal contribution to the CBM literature. However, I remain deeply concerned about the presentation of the work and the use of GRL. I am confident that a subsequent version of this work could be significantly stronger.
Finally, I would encourage the authors to place greater emphasis on incomplete settings, where information leakage is an even more significant concern. It would also be valuable to provide a clearer explanation of why in a CBM is worse compared to GC-CBM.
We sincerely thank the reviewer for a thorough and constructive review, and for recognising the mathematical analysis, the new metrics and the single-stage training. We ran every suggested experiment; all new figures and tables are in App. R of the revised PDF.
Why a GRL rather than a detach? The reviewer is right. At λ=1, GC is exactly a stop-gradient on the bottleneck for the task loss, and a detach implementation produces identical gradients without any overhead. All our runs use λ=1 and a single shared task head. For λ<1, GC is equivalent to joint training with concept weight λc/(1−λ), so λ only re-parameterises the joint-training family. We will present GC as a stop-gradient in the final version and keep the GRL only as this interpretation.
We do not claim the operator as the contribution. The contribution, which the reviewer lists as strengths S1–S3, is:
The new experiments strengthen point 3:
Value of λ, and ablation of λ. λ=1 in all experiments; we will state this in the main text. We have added a λ sweep on CUB, λ∈{0,0.2,…,1} (Table R8, Fig. R7). For GC-CBM, intermediate λ raises concept accuracy (70.6% → 88.0%) and intervention AUC (55.2 → 78.0), like a larger concept weight, while CTL stays at the joint level (0.038–0.044); only λ=1 removes it (CTL 0.024, concept accuracy 94.7%, AUC 87.9). By the equivalence above, intermediate λ trades leakage for task accuracy exactly as a larger λc does, so the benefit of "allowing some leakage" is that of joint training with a re-weighted concept loss. What only λ=1 provides is the limit λc→∞, which joint training cannot reach.
Sequential CEM baseline. At λ=1 the GC-CEM encoder is trained exactly as Sequential stage 1, so on each frozen GC-CEM encoder (CUB, 5 folds) we fitted a Sequential CEM head with exactly GC's hyperparameters, and compared the two in a pre-specified test (Table R7, Fig. R6). The results:
GC-CEM thus attains Sequential-CEM leakage and accuracy in one end-to-end stage.