acceptodds
Under review as a conference paper at ICLR 2027

CABM: Efficient Concept Augmentation for Interpretable Deep Models

Abstract

Concept Bottleneck Models (CBMs) offer interpretability by aligning latent representations with semantic concepts. However, standard CBMs suffer from a static concept space, limiting their adaptability to evolving knowledge. Expanding this space typically necessitates full retraining, which is computationally prohibitive and risks semantic drift in pre-existing concepts. In this work, we propose Concept Augmentation in Bottleneck Models (CABM), a novel framework for post-hoc concept addition that avoids iterative end-to-end retraining. CABM formulates concept addition as a parameter perturbation problem, addressed by influence-inspired local updates. To overcome the scalability bottleneck of dense curvature inversion in high-dimensional spaces, we leverage Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) to perform structured approximate second-order updates. We further introduce a trust-region constraint and concept-conditional initialization to improve numerical stability. We bound its local error relative to a re-optimized appended concept block, including the effect of trust-region projection. Extensive experiments show that our framework achieves predictive performance on par with full retraining while reducing computational costs by – relative to full retraining and – relative to a matched converged frozen-feature probe, offering a robust solution for dynamic interpretable modeling.

Primary area:
interpretability and explainable AI
Keywords:
Concept Bottleneck Models, Concept Addition, Influence Function, Explainable AI.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_89QhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CABM, a framework for efficiently adding new concepts to an existing Concept Bottleneck Model (CBM) without full retraining. CABM freezes the encoder and existing concept predictors, initializes new concept predictors using concept-conditional feature statistics, and applies a single EK-FAC-based second-order update to the added parameters. The label predictor is also adapted to incorporate the new concepts. Experiments on five datasets show that CABM achieves accuracy comparable to full retraining while substantially reducing update time and preserving existing concept predictions.

Critical strengths
  1. CABM enables new concepts to be added to an existing CBM by updating only the newly introduced parameters. This avoids full retraining while keeping the existing concept predictors unchanged.
  2. CABM achieves accuracy comparable to full retraining across five datasets while reducing update time by 10.8–61.5×. The experiments also include comparisons with frozen-feature baselines, providing a more complete evaluation of its efficiency.
  3. The paper evaluates not only classification and concept accuracy but also concept intervention stability, sequential concept addition, and the effects of different initialization and curvature approximation strategies. These experiments help assess the effectiveness of the proposed method.
Critical weaknesses
  1. The problem of adding new concepts to existing CBMs has been studied in prior work, including CONCIL and ECBM. The proposed method combines concept-conditional initialization, EK-FAC, and trust-region clipping, all of which are established techniques. It is not clear what the main technical contribution is beyond combining these components.
  2. While CABM is substantially faster than full retraining, the speedup over the frozen-feature Probe-LR baseline is only 1.2–1.5×. Since both methods operate on the same frozen representation, the efficiency gain from the proposed update strategy appears relatively modest.
  3. The results in Appendix D.4 show only small differences between EK-FAC and diagonal curvature approximation. On AwA2 with 15 added concepts, EK-FAC achieves 90.16% label accuracy compared with 89.93% for the diagonal approximation, with nearly identical runtime. These results do not clearly justify the additional complexity of EK-FAC.
  4. CABM assumes that the existing encoder provides sufficient features for learning new concepts. However, the experiments mainly involve removing existing concepts and adding them back. It is unclear how well the method would handle genuinely new concepts that are not well represented by the frozen encoder.
Questions for authors
  1. What is the main technical difference between CABM and existing concept-incremental learning methods such as CONCIL and ECBM? In particular, what does the proposed EK-FAC-based update offer that cannot be achieved with existing analytic updates?
  2. CABM achieves a 1.2–1.5× speedup over the matched Probe-LR baseline. Is this improvement consistent as the number of added concepts increases? How much of the speedup comes specifically from the proposed second-order update?
  3. Appendix D.4 shows that diagonal curvature approximation achieves performance close to EK-FAC. In which settings does EK-FAC provide a clear advantage over simpler alternatives?
  4. How does CABM perform when newly added concepts are poorly represented by the frozen encoder? Are there any diagnostics that can indicate when the existing representation is insufficient for concept augmentation?
  5. In the sequential concept addition experiments, how does repeatedly updating the label predictor affect the contribution of previously added concepts? Does the downstream prediction or intervention drift increase as more concepts are introduced?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 89Qh (I)

Thank you for the detailed questions. We address the contribution, update cost, curvature choice, representation limits, and sequential behavior below.


W1 / Q1. Contribution relative to ECBM and analytic concept-incremental methods

CABM studies how to add concept predictors to a trained CBM while keeping its encoder and existing concept predictors fixed. The new parameters need a starting point, an update objective, and a connection to the label predictor. CABM specifies these steps and studies whether one local correction can approach a reoptimized added block.

ECBM discusses concept insertion as an extension while developing concept-level removal in Sec. 4.2 (Hu et al., 2025). ECBM-Add is our explicitly constructed insertion adaptation, rather than an addition implementation supplied by ECBM.

The reference implementation appends zero-initialized affine concept predictors, including their biases. It recomputes EK-FAC on the regularized new-concept BCE objective and applies one damped correction without clipping the concept step. CABM instead starts from concept-conditional feature statistics and applies an explicit concept-step radius. Both use the same frozen representation and old concept pathway.

Concept-block choicePrimary ECBM-Add adaptationCABM
Initial feature weightsZeroScaled positive-minus-negative feature means
Initial concept biasZeroLog odds of training concept prevalence
Added-block objectiveBinary cross-entropy with L2 regularizationSame objective
CurvatureEK-FAC computed at the zero initializerEK-FAC computed at the conditional initializer
Concept correctionOne damped step, without a radius limitOne damped step with an explicit radius
Downstream controlShared update of the newly appended softmax coefficientsSame update

Both methods use the same regularized BCE objective and frozen encoder. The old concept predictors and old classification weights and bias remain fixed. We apply the same update to the new classification weights, so the comparison focuses on learning the added concept predictors.

Table R1. Direct comparison with ECBM-Add.

Dataset and addition setting: CUB, six wing-color concepts added to a 106-concept base. Independent base-model seeds: 3.

MethodLabel accuracy (%) ↑All-concept accuracy (%) ↑New-concept accuracy (%) ↑Update time (s) ↓
Original model78.72 ± 0.16696.48 ± 0.036——
Full retraining80.51 ± 0.48996.70 ± 0.19095.35 ± 1.5748795.00 ± 1134.643
ECBM-Add79.94 ± 0.15296.62 ± 0.03694.64 ± 0.171277.09 ± 5.801
CABM80.65 ± 0.14796.67 ± 0.03395.31 ± 0.156186.00 ± 5.273

All methods start from matched base models. All-concept accuracy covers the outputs available in each model. New-concept accuracy covers only the added concepts. We use per-concept validation-selected thresholds for both old and new concepts after editing and report mean ± SD across three independent base-model seeds. Update-time accounting is feature extraction, concept fitting and head updates for editing; all-parameter training and validation for retraining, excluding base training and test evaluation.

CABM improves new-concept accuracy over this ECBM-Add adaptation by 0.67 percentage points, while label accuracy is 80.65% versus 79.94%, and update time is 186.00 s versus 277.09 s for ECBM-Add.

CONCIL takes a different route. It uses recursive analytic updates for linear-regression objectives. CABM keeps the sigmoid concept objective and, by default, a softmax label predictor. These objectives do not generally have the same closed-form solution as least squares. CABM provides an approximate local update while retaining this model form and the old concept pathway. For a ridge label head, CABM also supports an exact analytic update.

What CABM adds. Concept insertion introduces predictor parameters that have no learned values in the base model. CABM specifies their concept-conditional starting point (Eq. 9), their regularized added-block objective and finite correction (Eqs. 4–7), and a separate update of the new concept-to-label coefficients (Sec. 4.3). The resulting procedure expands the concept interface while preserving the encoder and old concept predictors. The default softmax update additionally retains the old classification coefficients and bias; the ridge alternative has a different scope and reoptimizes both coefficient blocks.

AuthorsOpenReview ↗

Response to Reviewer 89Qh (II)

Theorem 1 bounds the error of the updated concept block relative to reoptimizing that block on the same frozen features. It separates curvature approximation, local approximation, and clipping errors. Proposition 2 establishes preservation of old concept scores and bounds changes in their downstream intervention effects under the default head update.

The manuscript already tests components of this construction. In Fig. 3(a), the concept-conditional initializer alone gives a new-concept accuracy of 80.49% and label accuracy of 68.28%; full CABM reports 89.54% and 73.01%, respectively. The initialization comparison excludes the frozen old concepts from its concept-accuracy metric. The curvature and solution diagnostics in Appendices D.4–D.5 provide complementary evidence.


W2 / Q2. Efficiency relative to Probe-LR and scaling with added concepts

We extended the comparison to much larger additions on CUB: 80 concepts added to a 32-concept base, and 64 concepts added to a 48-concept base. Each round adds eight concepts. CABM, Probe-LR, and Diagonal use the same final full classification-head refit. The encoder and concept predictors stay frozen during this refit.

For the classification head, default CABM updates only the newly added classification weights. In this comparison, we additionally refit the full classification head for all methods, while keeping the encoder and concept predictors frozen. The reported update time includes this refit.

Table R2. Large concept-set expansion with a final full-head refit.

SettingMethodLabel accuracy (%) ↑New-concept accuracy (%) ↑Cached update time (s) ↓Update time from images (s) ↓Image-based speedup vs. Probe-LR ↑
32 → 112, 10 roundsCABM79.0895.4974.64252.471.60×
32 → 112, 10 roundsProbe-LR78.5995.00224.97402.801.00×
32 → 112, 10 roundsDiagonal*78.6795.2458.38236.211.71×
32 → 112Full-model retrain79.4095.67—15979.91†—
48 → 112, 8 roundsCABM80.0595.9063.72241.551.39×
48 → 112, 8 roundsProbe-LR79.8895.68158.16335.991.00×
48 → 112, 8 roundsDiagonal78.7294.6751.11228.941.47×
48 → 112Full-model retrain80.1196.83—8907.98†—

New-concept accuracy uses fixed 0.5 thresholds, consistently across methods. Cached time excludes feature extraction. Image-based time includes it. Both include the intermediate edits and the final head refit, with hyperparameters already selected. Speedup is Probe-LR time divided by the method's time.

*The starred 32 → 112 Diagonal row is a curvature-replacement ablation; the 48 → 112 Diagonal row is independently tuned. †Retraining has a broader timing scope, stated as initialization, all-parameter training, validation and checkpoint saving, excluding final test evaluation, so no strict speedup ratio is computed against that row. The number of independent base seeds is 3; insertion orders are reported separately.

For 32 → 112, CABM reaches 79.08%, compared with 78.59%, while reducing image-based update time from 402.80 s to 252.47 s. This gives a 1.60× speedup. The corresponding new-concept accuracy difference is +0.49 percentage points. We use both quality and cost to assess the benefit of the faster update.

The comparison with full retraining includes the saving from freezing the encoder. The matched Probe-LR comparison isolates the choice of how to fit the added concept block, since both methods use the same features, concept objective, and head-fitting procedure. We further separate concept-fitting time from the remaining update cost.

Table R3. Scaling from the same 32-concept base.

Added conceptsCABM new-concept accuracy (%)Probe-LR new-concept accuracy (%)CABM concept-fitting time (s)Probe-LR concept-fitting time (s)CABM total update time (s)Probe-LR total update time (s)Speedup ↑
894.5594.5020.9820.32206.89205.130.99×
1694.7194.5822.6040.64210.91226.951.08×
3294.9594.7625.8481.28219.74270.701.23×
4895.4194.9429.07121.92229.61314.601.37×
6495.3794.9032.31162.56240.52358.631.49×
8095.4995.0035.54203.21252.47402.801.60×
AuthorsOpenReview ↗

Response to Reviewer 89Qh (III)

Each point includes the stated sequence prefix and one final full-head refit. Total time uses image-based accounting. Concept-fitting time includes initialization and curvature construction; total time also includes the specified feature processing and head updates. Each total is measured directly. Hyperparameter search is excluded and reported separately.

Table R3 tests scaling within one base model. The two settings in Table R2 also test different initial concept coverage. We will distinguish these comparisons and state the range over which the measured speed advantage holds.


W3 / Q3. When does EK-FAC provide an advantage over Diagonal?

Diagonal is a competitive choice in some of the smaller edits. We therefore added the larger CUB comparisons in Table R2 and evaluated the new concepts separately from the old ones.

This distinction matters when the old outputs are fixed. An average over all concepts can hide a larger difference within the added block. We report new-concept accuracy, along with task accuracy, to make that difference visible.

In the 48 → 112 comparison, Diagonal and CABM each use the same validation-search budget: 60 candidate settings per batch, or 480 in total. CABM improves label accuracy by 1.33 percentage points and new-concept accuracy by 1.23 points. Their image-based update times are 241.55 s and 228.94 s. This provides a setting where the quality gain can be assessed against the added computation.

EK-FAC retains a structured approximation of curvature interactions that a diagonal estimate drops. Such interactions can matter for a single correction. The existing update-direction and objective-gap diagnostics in Appendix D.4–D.5 provide supporting evidence about the approximation. The new large-addition comparison tests whether the difference also affects prediction.

These results support using EK-FAC in the tested addition settings where it improves quality at an acceptable cost. We will also retain the settings where Diagonal performs similarly, so the choice between the two is clear.


W4 / Q4. New concepts that are poorly represented by the frozen encoder

We start from a pretrained visual encoder and train the base CBM using the available concept labels and task labels. The encoder is frozen when new concepts are added.

Our CUB experiments already hold out entire semantic concept groups, such as wing color, bill shape, and tail pattern, before base-model training. The prediction units for the held-out group are absent, and its concept labels are not used in the base-training loss. We do not first train a model with all concepts and then remove their outputs. These experiments therefore test concepts whose annotations were not used to train the base CBM.

Pretraining and task supervision may still provide useful features for these concepts. Whether those features are sufficient is a separate question. We address it with additional tests of representation quality.

For these tests, we use the same six CUB wing-color concepts and the same data split. The encoder is an ImageNet-pretrained Inception-v3 with 2,048-dimensional features. We compare three versions:

  • Early: the checkpoint after 10 epochs of base-model training, without the six new concept labels.
  • Mature: the selected checkpoint at epoch 170 from the same base-model training run, also without those labels.
  • Adapted: a model initialized from the early checkpoint and trained further with the new concept supervision, allowing the encoder to change.

We then freeze each encoder and fit CABM, a converged linear Probe-LR, and a nonlinear MLP probe on its features. The same linear probe on the adapted features helps separate representation changes from differences in how a concept predictor is optimized.

Table R4. New-concept prediction under different encoder conditions.

EncoderMethodLabel accuracy (%) ↑New-concept accuracy (%) ↑New-concept balanced accuracy (%) ↑AUROC (%) ↑BCE ↓
Early, epoch 10CABM75.3094.2390.4197.010.176
Early, epoch 10Converged Probe-LR73.8494.4390.2897.450.234
Early, epoch 10Nonlinear MLP probe74.8594.2589.4097.510.152
Mature, epoch 170CABM81.5295.3192.0197.780.153
Mature, epoch 170Converged Probe-LR81.2295.9893.1598.320.132
Mature, epoch 170Nonlinear MLP probe81.3695.8192.8298.310.120
Adapted from epoch 10CABM82.2795.6793.5997.510.149
Adapted from epoch 10Converged Probe-LR81.9096.2393.3898.400.144
Adapted from epoch 10Nonlinear MLP probe82.0596.1193.0398.480.112
Reviewer_r5G9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CABM, a method for adding new concepts to a trained Concept Bottleneck Model (CBM) without full retraining. CABM freezes the encoder and existing concept predictors, initializes new concept predictors using concept-conditional feature statistics, and applies EK-FAC-based second-order updates with a trust-region constraint. The label predictor is also updated to incorporate the new concepts. Experiments on five datasets show that CABM achieves predictive accuracy comparable to full retraining while reducing computational costs and preserving existing concept predictions.

Critical strengths

S1. The paper explores an interesting and meaningful research direction.

S2. CABM achieves accuracy comparable to full retraining while substantially reducing update time. The additional comparison with Probe-LR helps clarify the efficiency gained from the proposed update method.

S3. The experiments examine more than classification accuracy. The authors also evaluate concept preservation, intervention stability, sequential concept addition, and robustness to noisy annotations.

Critical weaknesses

W1. The experiments mainly evaluate concept addition by removing existing concepts and adding them back. Since CABM freezes the encoder, its effectiveness depends on whether the original features already capture the new concepts. It would be useful to evaluate the method in settings where the new concepts are not well represented by the pretrained encoder.

W2. Although CABM is much faster than full retraining, it is only 1.21–1.53× faster than Probe-LR, which also freezes the encoder. The improvement over Probe-LR is relatively modest, especially considering the additional complexity of EK-FAC. Much of the speedup may come from freezing the encoder.

W3. The improvement in classification accuracy after adding new concepts is quite small on the medical datasets. For example, accuracy only increases from 72.97% to 73.01% on SkinCon and from 80.24% to 80.55% on Derm7pt. It would be helpful to further examine how much the added concepts actually contribute to the final predictions.

W4. Figures 1 and 2 are quite crowded, and some of the text and equations are difficult to read. Several tables in the appendix are also too dense, making it difficult to follow the results.

Questions for authors

Please refer to weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer r5G9 (I)

Thank you for recognizing the value of concept augmentation and the breadth of our evaluation. We address the representation limit, efficiency, contribution of the medical concepts, and presentation below.


W1. New concepts that are poorly represented by the encoder

We start from a pretrained visual encoder. The base CBM is then trained using the available concept labels and task labels. The encoder is frozen when the new concepts are added.

In CUB, we exclude entire semantic concept groups, such as wing color, bill shape, and tail pattern, before training the base model. Their prediction units are absent, and their concept labels are not used in the base-training loss. We do not first train a full-concept model and then remove its outputs. This tests the addition of concepts whose annotations were not used during base-model training.

Pretraining and task supervision may still provide useful features for the held-out concepts. We therefore added a separate test of representation quality.

The new CUB test uses an ImageNet-pretrained Inception-v3 and the same six held-out wing-color concepts. We compare an early checkpoint at epoch 10, a mature checkpoint at epoch 170, and an encoder adapted from the early checkpoint with the new concept supervision. The early and mature models were trained without those six concept labels. The adapted model is a reference that allows the representation to change.

We freeze each encoder and fit CABM, a converged linear Probe-LR, and a nonlinear probe on its features. All methods within an encoder condition use the same data split and frozen features.

Table R1. New-concept prediction under different encoder conditions.

EncoderMethodLabel accuracy (%) ↑New-concept accuracy (%) ↑New-concept balanced accuracy (%) ↑AUROC (%) ↑BCE ↓
Early, epoch 10CABM75.3094.2390.4197.010.176
Early, epoch 10Converged Probe-LR73.8494.4390.2897.450.234
Early, epoch 10Nonlinear MLP probe74.8594.2589.4097.510.152
Mature, epoch 170CABM81.5295.3192.0197.780.153
Mature, epoch 170Converged Probe-LR81.2295.9893.1598.320.132
Mature, epoch 170Nonlinear MLP probe81.3695.8192.8298.310.120
Adapted from epoch 10CABM82.2795.6793.5997.510.149
Adapted from epoch 10Converged Probe-LR81.9096.2393.3898.400.144
Adapted from epoch 10Nonlinear MLP probe82.0596.1193.0398.480.112

All concept metrics cover the same six added concepts. CABM follows the settings specified in Sec. 4 and Appendices E.4–E.5. Within each encoder condition, all methods use the same frozen features, and the linear methods share the concept objective and regularization. Label accuracy is evaluated on the 200-way CUB task using an additional full classification-head refit for all methods. Concept accuracy uses a fixed threshold of 0.5. Results are reported for one independent base-model seed.

The matched Probe-LR reaches 94.43% new-concept accuracy on early features, 95.98% new-concept accuracy on mature features, and 96.23% new-concept accuracy after adaptation. CABM reaches 94.23% new-concept accuracy on early features, with a gap of 0.20 percentage points from the matched probe. These results show how concept augmentation depends on the available representation.

The comparisons also help interpret failures. If the converged probe works well but CABM does not, further optimization of the added block may help. If a nonlinear probe improves the result, a more flexible readout may be useful. If adapting the encoder improves the same probe, the frozen representation is limiting the update. We will make these conditions and the scope of the method clearer in the paper.


W2. Efficiency gain over Probe-LR

The speedup over full retraining includes the benefit of freezing the encoder. The matched Probe-LR baseline measures the additional benefit of replacing iterative optimization of the new concept block with one correction. The reported 1.21–1.53× speedup corresponds to about 17%–35% less update time in those dataset averages.

We added larger concept-expansion tests on CUB. One starts with 32 concepts and adds 80; the other starts with 48 concepts and adds 64. Both use batches of eight concepts. CABM and Probe-LR use the same frozen features, concept objective, and final full classification-head refit.

For the classification head, default CABM updates only the newly added classification weights. In this comparison, we additionally refit the full classification head for all methods, while keeping the encoder and concept predictors frozen. The reported update time includes this refit.

AuthorsOpenReview ↗

Response to Reviewer r5G9 (II)

Table R2. CABM versus Probe-LR under large concept-set expansion.

SettingMethodLabel accuracy (%) ↑New-concept accuracy (%) ↑Concept-fitting time (s) ↓Cached update time (s) ↓Update time from images (s) ↓Image-based speedup vs. Probe-LR ↑
32 → 112, 10 roundsCABM79.0895.4935.5474.64252.471.60×
32 → 112, 10 roundsProbe-LR78.5995.00203.21224.97402.801.00×
48 → 112, 8 roundsCABM80.0595.9035.8863.72241.551.39×
48 → 112, 8 roundsProbe-LR79.8895.68134.13158.16335.991.00×

Concept-fitting time includes initialization and curvature construction. Cached update time also includes the specified prediction and head-update work. Image-based time includes feature extraction. Total times are measured directly and include the intermediate edits and the final head refit.

New-concept accuracy uses fixed 0.5 for both methods. Results cover 3 independent base seeds and one insertion orders per seed. Insertion orders are not counted as independent training seeds.

For 32 → 112, CABM reaches 79.08%, compared with 78.59%, and reduces image-based update time from 402.80 s to 252.47 s. The corresponding new-concept accuracy difference is +0.49 percentage points. These results show the practical quality–cost trade-off when many concepts are added.

Reporting concept-fitting time separately also shows how much work is saved by the update rule. We will distinguish this measured saving from the larger saving obtained by avoiding encoder retraining.


W3. Contribution of the added concepts on medical datasets

Our paper already evaluates whether added concepts participate in the downstream prediction, beyond the change in unassisted classification accuracy. Appendix E.8 evaluates each of the 22 SkinCon concepts by withholding it during base-model training, adding it back, and correcting that new concept to its ground-truth value at test time.

In Table 20, CABM reaches 73.11% label accuracy when correcting the new concept, compared with 73.06% when correcting a random old concept and 73.07% for retraining with the new-concept intervention. CABM without intervention reaches 73.01%, compared with the base model's 72.97%. These reported means show that the added pathway remains actionable after expansion; the evaluation is not limited to the accuracy of the new concept predictor.

Table 21 provides an irrelevant-concept control. Adding a pure random binary concept gives 72.94% label accuracy, compared with 72.97% for the base model, and the reported weight assigned to that concept is 0.002. In this control, an arbitrary added input acquires little influence on the classifier. Together, the targeted intervention and irrelevant-concept control support useful, actionable concept addition while retaining task performance. The effects on average accuracy are modest.

The additional paired and subgroup analyses below supplement this existing evidence. They should be interpreted separately from the targeted-intervention results already reported in the manuscript.

Table R3 compare the original model and the edited model, then disable the new pathway in the edited model. We will also replace the predicted new concept with its ground-truth value. The remaining concept inputs and the fitted head will stay fixed within each intervention comparison. A random-old-concept intervention provides a matched control with the same number of corrected concepts.

Table R3. Contribution of the new concept pathway.

EvaluationSkinCon label accuracy (%) ↑Change vs. CABM (pp; interval)Derm7pt label accuracy (%) ↑Change vs. CABM (pp; interval)
Original base model72.97-0.04; -0.19, +0.1180.24-0.31; -0.70, +0.08
CABM without intervention73.01080.550
CABM with the new pathway disabled72.97-0.04; -0.19, +0.1180.24-0.31; -0.70, +0.08
CABM with the ground-truth new concept73.11+0.10; -0.05, +0.2580.79+0.24; -0.15, +0.63
CABM with a ground-truth random old concept73.06+0.05; -0.10, +0.2080.67+0.12; -0.27, +0.51
AuthorsOpenReview ↗

Response to Reviewer r5G9 (III)

All CABM intervention rows use the same edited checkpoint and fixed head parameters. The random-old-concept result averages 1 specified choices, with the same number of corrected concepts. Results cover 22 (SkinCon) and 7 (Derm7pt) added concepts and 5 independent seeds. Paired confidence intervals use paired concept-mean t intervals for the base comparison after averaging seeds; ground-truth intervention intervals include a shared edit-dispersion term. Subgroup comparisons use normal-approximation bounds on paired discordance subject to the fixed-head intervention constraints, without treating seeds as extra test samples.

For the default head used in this comparison, disabling the new pathway restores the base prediction because the old classification weights and bias remain fixed. This isolates the contribution of the added concept pathway.

Table R4 reports results on concept-positive and concept-negative samples. These groups are defined by the concept labels, before examining the subgroup results. This tests whether a small overall change hides a more focused benefit or a trade-off between groups.

Table R4. Effects within predefined concept groups.

DatasetAdded conceptSample groupSample countBase label accuracy (%)CABM label accuracy (%)CABM − base (pp; interval)Gain from correcting the new concept (pp; interval)
SkinConPapuleConcept-positive81572.2572.56+0.31; -3.09, +3.72+0.77; -2.65, +4.18
SkinConPapuleConcept-negative143573.3373.64+0.31; -2.92, +3.54-0.31; -3.54, +2.92
Derm7ptPigNetConcept-positive23981.9181.80-0.11; -5.32, +5.10+0.31; -4.90, +5.52
Derm7ptPigNetConcept-negative15682.1182.00-0.11; -8.63, +8.41+0.11; -8.41, +8.63

The correction gain is measured relative to the same CABM model without intervention. The table is repeated for each evaluated concept, including concepts with no gain or a negative change.

Together, these analyses will show whether the predicted new concepts improve decisions, whether correcting them provides further benefit, and how consistent those effects are. We will describe their practical value using the measured effect sizes and uncertainty.


W4. Readability of Figures 1–2 and the appendix tables

We have adjusted the sizing and layout of Figures 1 and 2 to improve readability while preserving their original visual design. The captions clarify the problem setting, the update procedure, and which components remain fixed.

We will also split the wide appendix tables into smaller tables with consistent metric names and units. Old-concept accuracy, new-concept accuracy, and accuracy over the expanded concept set will be labeled separately. Fixed-threshold and recalibrated results will also be distinguished. The main text will point directly to the tables needed to assess each claim.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your helpful comments. Our rebuttal provides additional tests under different encoder conditions, comparisons with Probe-LR and Diagonal when adding many concepts, and intervention and subgroup analyses of the added medical concepts. We have also revised the paper to improve the presentation and explain these findings.

We would appreciate your feedback on whether these results address your concerns. Please let us know if any points need further clarification. Thank you for your time and consideration.

Reviewer_bAKLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on the problem of adding new concepts to existing Concept Bottleneck Models (CBM) in a post-hoc manner without requiring full retraining. They propose Concept Augmentation in Bottleneck Models (CABM) which uses Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) to perform the concept addition. They also provide theoretical analysis to bound the local error of this concept addition. Finally, they perform extensive experiments comparing to other incremental/continual CBM methods and show their method can achieve similar results with much faster runtime.

Critical strengths
  • The post-hoc concept addition method proposed is significantly faster than re-training the entire CBM as well as being faster than other incremental/continual CBM methods.
Critical weaknesses
  • The differences compared to Editable CBMs (Hu et al., ICML 2025) are not clear to me. Editable CBMs also use EK-FAC similar to this paper, albeit for removing concepts instead of adding concepts. But the main technical novelty is unclear to me given the similarities. And the paper does not mention this similarity when discussing the use of EK-FAC for adding new concepts.

    • Another relevant paper that is not discussed is [W1]. They also explore the idea of post-hoc adding missing concepts, although they discover the concepts instead of the concepts being provided later.
  • Fig. 4 shows how task label accuracy varies with number of concepts added. But it is unclear how concept accuracy changes, at least for the old concepts. It would be good to add that as well.

  • Based on Table 1, the main improvement is in the runtime. In the other metrics, it is unclear how statistically significant the improvements or comparable-ness of the results is. Across different concepts/concept sets being added, we would need at least mean ±\pm standard deviation or some other statistical tests to be sure about it. Tables 14-15 provide mean ±\pm standard deviation for specific concepts but that is over 5 re-runs of the same concept, we need to check the variance over different concepts/concept sets being added.

References

[W1] Shang et al., "Incremental Residual Concept Bottleneck Models", CVPR 2024

Questions for authors

Please address the weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bAKL (I)

W1(a). Relationship to Editable CBMs

Thank you for pointing out the connection to ECBM.

ECBM discusses concept insertion as a possible extension in Sec. 4.2 (Hu et al., 2025), but does not provide a concrete method for this setting. We will clarify the similarities and differences between ECBM and CABM in the main text.

At the concept level, ECBM develops a method for removing existing concepts, while CABM focuses on adding new concepts to a trained model. Removal starts from parameters that have already been learned. Addition introduces new predictor weights and biases with no learned starting point. Reversing the sign of a removal update does not resolve how these new parameters should be initialized. This choice matters for a single-step method because the gradient and curvature are evaluated at that starting point.

CABM addresses this additional challenge through concept-conditional initialization and a controlled update of the new parameters. It initializes the feature weights from the scaled difference between positive and negative feature means, and sets the bias to the log odds of the training concept frequency. It then applies one EK-FAC correction with an explicit update radius. Together, these steps provide a concrete procedure for learning newly added concept predictors while keeping the encoder and existing concept predictors fixed.

To examine these choices, we constructed an ECBM-Add baseline. It starts the new weights and biases at zero, recomputes EK-FAC for the new-concept objective, and applies one damped correction without clipping the concept step.

Concept-block choicePrimary ECBM-Add adaptationCABM
Initial feature weightsZeroScaled positive-minus-negative feature means
Initial concept biasZeroLog odds of training concept prevalence
CurvatureEK-FAC computed at the zero initializerEK-FAC computed at the conditional initializer
Concept correctionOne damped step, without a radius limitOne damped step with an explicit radius
Downstream controlShared update of the newly appended softmax coefficientsSame update

Both methods use the same regularized BCE objective and frozen encoder. The old concept predictors and old classification weights and bias remain fixed. We apply the same update to the new classification weights, so the comparison focuses on learning the added concept predictors.

Table R1 compares the original model, full retraining, ECBM-Add, and CABM from matched base models. This comparison measures the benefit of CABM's initialization and concept-update design under the same downstream update.

Table R1. Comparison with ECBM-Add.

Dataset: CUB. Added concept group: wing color. Number of added concepts: 6. Independent base-model seeds: 3.

MethodLabel accuracy (%) ↑All-concept accuracy (%) ↑New-concept accuracy (%) ↑Update time (s) ↓
Original model78.72 ± 0.16696.48 ± 0.036——
Full retraining80.51 ± 0.48996.70 ± 0.19095.35 ± 1.5748795.00 ± 1134.643
ECBM-Add79.94 ± 0.15296.62 ± 0.03694.64 ± 0.171277.09 ± 5.801
CABM80.65 ± 0.14796.67 ± 0.03395.31 ± 0.156186.00 ± 5.273

Results are averaged across independent base-model seeds for the stated addition setting. All-concept accuracy covers the outputs available in each model. New-concept accuracy covers only the added concepts. The concept-threshold rule is per-concept validation-selected thresholds for both old and new concepts after editing. Update time excludes base-model training and test evaluation. The included timing components are feature extraction, concept fitting and head updates for editing; all-parameter training and validation for retraining, excluding base training and test evaluation.

Under this protocol, CABM improves new-concept accuracy over ECBM-Add by 0.67 percentage points and all-concept accuracy by 0.05 points. Its label accuracy is higher, and its update time is 186.00 s, compared with 277.09 s for ECBM-Add. These results show the benefit of the proposed addition procedure under the stated setting.

CABM uses the new labels before the curvature correction. It initializes the feature weights from the positive and negative feature means, and the bias from concept prevalence. Let these means be μ^1\hat{\mu}_1 and μ^0\hat{\mu}_0. For w0=α(μ^1−μ^0)w_0=\alpha(\hat{\mu}_1-\hat{\mu}_0), the mean initial logit gap is

-\mathbb{E}_{\mathrm{tr}}[z_0\mid c=0] =\alpha\lVert\hat{\mu}_1-\hat{\mu}_0\rVert_2^2.$$ When the feature means differ, the two groups therefore have different mean scores before the update. The prevalence-based bias also accounts for concept imbalance. The curvature step then corrects this initial estimate.
AuthorsOpenReview ↗

Response to Reviewer bAKL (II)

Theorem 1 bounds the error of the updated concept block relative to reoptimizing that block on the same frozen features. It separates curvature approximation, local approximation, and clipping errors. Proposition 2 establishes preservation of old concept scores and bounds changes in their downstream intervention effects under the default head update.

The manuscript already tests components of this construction. In Fig. 3(a), the concept-conditional initializer alone gives a new-concept accuracy of 80.49% and label accuracy of 68.28%; full CABM reports 89.54% and 73.01%, respectively. The initialization comparison excludes the frozen old concepts from its concept-accuracy metric. The curvature and solution diagnostics in Appendices D.4–D.5 provide complementary evidence.


W1(b). Relationship to Incremental Residual CBMs

Thank you for pointing out Res-CBM. It is directly relevant, and we will add it to the related-work discussion.

Res-CBM addresses missing concepts through residual learning and concept discovery. CABM studies how to incorporate supplied concepts and annotations after base-model training, with a focus on update cost and preservation of existing concept predictions. These directions are complementary. We will clarify this distinction and avoid presenting post-hoc concept expansion itself as a new problem.

W2. Old- and new-concept accuracy during sequential addition

CABM keeps the encoder and all previously learned concept predictors fixed. Each binary concept has an independent sigmoid output. Adding new concepts therefore leaves the old concept scores unchanged. With the same thresholds, their predictions and accuracy also remain unchanged, up to numerical precision.

The downstream softmax can change class probabilities and the probability-level effects of intervening on an old concept. It does not change the old concept predictions themselves. We will make this distinction clear.

We have added sequential concept-accuracy results for CUB in Appendix F.5, Table 34. The table separately reports accuracy for the original concepts, the current new batch, and all concepts added so far. Previously learned concept scores and predictions under fixed thresholds remain unchanged throughout the tested sequence.

Table R2. Concept accuracy and preservation during sequential addition.

Dataset and fixed addition sequence: CUB, 32 initial concepts with ten fixed batches of eight concepts. Independent base-model seeds: 3.

Addition stepCumulative added conceptsOriginal-concept accuracy (%)Current new-batch accuracy (%)Accuracy over all added concepts (%)Maximum change in previously learned scoresMaximum prediction flip rate (%)
0096.13 ± 0.105——00
1896.13 ± 0.10594.55 ± 0.28694.55 ± 0.28600
21696.13 ± 0.10594.87 ± 0.27094.71 ± 0.19700
32496.13 ± 0.10595.23 ± 0.25194.88 ± 0.15600
43296.13 ± 0.10595.14 ± 0.25694.95 ± 0.13400
54096.13 ± 0.10596.51 ± 0.18395.26 ± 0.11300
64896.13 ± 0.10596.18 ± 0.20195.41 ± 0.10000
75696.13 ± 0.10595.36 ± 0.24495.40 ± 0.09300
86496.13 ± 0.10595.14 ± 0.25595.37 ± 0.08800
97296.13 ± 0.10596.29 ± 0.19595.47 ± 0.08100
Final8096.13 ± 0.10595.67 ± 0.22895.49 ± 0.07700

Original-concept accuracy always uses the same initial concept set. Each previously learned predictor retains its threshold throughout later additions. Accuracy over all added concepts uses a growing set and may change as new concepts enter that set. Accuracy standard deviations are across independent seeds for the stated sequence.

Across 3 runs, the largest change in a previously learned concept score is 0, and the largest prediction flip rate is 0%. The new-concept results show stable cumulative accuracy with batch-to-batch variation as different concepts enter the set. These measurements directly show old-concept preservation alongside learning of the added concepts.


W3. Variation across different concepts and concept sets

Our goal is to retain predictive performance close to full retraining while reducing the cost of adding concepts. Matching retraining closely at a much lower update cost is therefore an important outcome, even without an accuracy gain over retraining.

We agree that repeated runs of one concept do not measure variation across concept choices. We will report these sources of variation separately. For each concept or concept set, we first average over seeds. We then report the mean and standard deviation across the resulting setting-level averages. The per-setting seed results will remain in the appendix.

AuthorsOpenReview ↗

Response to Reviewer bAKL (III)

Table R3. Performance across concept-addition settings.

Entries are mean ± across-setting SD, after averaging seeds within each setting. Different addition sizes are reported separately.

DatasetAdded concepts per settingConcept setsSeeds per setMethodLabel accuracy (%) ↑All-concept accuracy (%) ↑New-concept accuracy (%) ↑Update time (s) ↓
SkinCon1225Full retraining72.99 ± 0.35287.79 ± 0.59786.53 ± 0.659346.10 ± 81.322
SkinCon1225CABM73.01 ± 0.39589.33 ± 0.77189.54 ± 0.91832.15 ± 2.196
Derm7pt175Full retraining80.23 ± 0.66574.70 ± 2.17574.35 ± 2.205343.00 ± 103.152
Derm7pt175CABM80.55 ± 0.84374.96 ± 1.97575.37 ± 2.10927.70 ± 0.525
CUB3–6 (semantic groups)33Full retraining81.35 ± 0.77796.72 ± 0.02095.53 ± 0.40911931.00 ± 2900.379
CUB3–6 (semantic groups)33CABM81.34 ± 0.64296.70 ± 0.02695.51 ± 0.410194.00 ± 9.713
CelebA231Full retraining42.78 ± 0.61053.54 ± 0.52065.77 ± 6.9741452.00 ± 43.361
CelebA231CABM43.66 ± 0.32056.77 ± 0.48368.15 ± 6.48946.80 ± 0.043
AwA2533Full retraining90.20 ± 0.19897.73 ± 0.08895.46 ± 1.7646291.00 ± 9.623
AwA2533CABM90.15 ± 0.13797.52 ± 0.03794.97 ± 2.205331.60 ± 0.507
AwA21033Full retraining90.18 ± 0.19897.40 ± 0.06095.22 ± 1.1137218.00 ± 7.559
AwA21033CABM90.13 ± 0.13797.50 ± 0.02695.38 ± 1.221402.30 ± 0.421
AwA21533Full retraining90.21 ± 0.19797.71 ± 0.03696.04 ± 0.6418080.00 ± 6.675
AwA21533CABM90.16 ± 0.13797.58 ± 0.01995.80 ± 0.772452.00 ± 0.373

We also report paired differences in Table R4. Each pair uses the same concept set, base-model seed, and data split. This directly measures the performance gap and its uncertainty.

Table R4. Paired comparison across concept-addition settings.

Accuracy differences are CABM minus the reference, in percentage points. Speedup is the reference mean update time divided by the CABM mean update time.

Dataset / addition sizeReferencePaired concept setsLabel-accuracy difference [interval]New-concept-accuracy difference [interval]Update speedup [original interval]
SkinCon / 1Full retraining22+0.02; -0.12, +0.16+3.01; +2.62, +3.4010.77×; 9.83×, 11.78×
Derm7pt / 1Full retraining7+0.32; -0.41, +1.06+1.02; +0.93, +1.1112.38×; 9.46×, 16.21×
CUB / 3–6Full retraining3-0.01; -0.35, +0.32-0.03; -0.06, +0.0161.50×; 37.66×, 100.43×
CelebA / 2Full retraining3+0.88; +0.16, +1.60+2.38; +1.18, +3.5831.03×; 28.74×, 33.49×
AwA2 / 5Full retraining3-0.04; -0.54, +0.44-0.49; -2.39, +1.4118.97×; 18.83×, 19.12×
AwA2 / 10Full retraining3-0.05; -0.59, +0.49+0.16; -0.11, +0.4317.94×; 17.85×, 18.04×
AwA2 / 15Full retraining3-0.05; -0.61, +0.51-0.24; -0.56, +0.0917.88×; 17.80×, 17.95×
CUB / 3–6 semantic conceptsECBM-Add3+0.503; +0.010, +0.996+0.39; -0.23, +1.001.45×; 1.28×, 1.88×
SkinCon / 1ECBM-Add22+0.007; -0.018, +0.033+1.11; +1.05, +1.171.46×; 1.27×, 1.72×
Derm7pt / 1ECBM-Add7+0.054; -0.013, +0.122+2.88; +2.63, +3.131.40×; 1.27×, 1.56×
CelebA / 2ECBM-Add3+0.305; -0.010, +0.621+6.73; +3.33, +10.141.44×; 1.27×, 1.68×
AwA2 / 5ECBM-Add3+0.014; +0.013, +0.014+0.34; -0.35, +1.031.34×; 1.17×, 1.67×
AwA2 / 10ECBM-Add3+0.035; +0.034, +0.036+0.28; -0.08, +0.641.28×; 1.14×, 1.55×
AwA2 / 15ECBM-Add3+0.141; +0.136, +0.146+0.23; +0.01, +0.441.25×; 1.12×, 1.74×

The intervals are computed using paired Student-t calculations for the SkinCon, Derm7pt and CUB setting-level label differences after averaging seed repeats. Other accuracy intervals incorporate sensitivity to concept difficulty, paired covariance and threshold calibration. Retraining speed intervals use paired delta calculations or timing-covariance bounds; ECBM-Add speed intervals account for workload and runtime calibration.

Across these settings, CABM's mean label accuracy differs from retraining by -0.05 to +0.88 percentage points, while its mean new-concept accuracy differs by -0.49 to +3.01 percentage points. Its update cost is 10.77–61.50 times lower than full retraining under the reported timing scope. The paired intervals show small label-accuracy differences, while the new-concept gaps and their intervals vary with concept difficulty and threshold choice. We will use these results to state the accuracy–cost trade-off more precisely, rather than relying on small differences between aggregate means.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Reviewer Discussion

Dear Area Chair,

Thank you for considering our paper. Two reviewers raised their scores after the rebuttal. Reviewer bAKL stated that their major concerns were resolved and raised their rating to Weak Accept. Reviewer 89Qh also raised their score, noting that the additional experiments addressed most of their concerns, especially those about scalability and the benefit of EK-FAC.

CABM adds new concepts to an existing concept bottleneck model while keeping the encoder and old concept predictors fixed. In response to the reviews, we added the following experiments and analyses:

  1. Comparison with ECBM-Add. On six held-out CUB wing-color concepts, CABM reached 95.31% new-concept accuracy, compared with 94.64% for our ECBM-Add adaptation and 95.35% for full retraining.
  2. Large concept additions. We compared CABM, Probe-LR, and a diagonal update when expanding CUB from 32 or 48 concepts to 112. In the 32-to-112 setting, CABM achieved higher new-concept accuracy than both alternatives and took 75 seconds with cached features, compared with 225 seconds for Probe-LR.
  3. Encoder quality. We compared CABM with linear and nonlinear probes on early, mature, and adapted encoders to examine when frozen features support a new concept and when adapting the encoder helps.
  4. Old and new concept accuracy. We tracked accuracy separately for old and new concepts during sequential additions and reported how results varied across concepts and concept sets.
  5. Medical concept interventions. On SkinCon and Derm7pt, we tested how predictions changed when we disabled the new concept pathway or replaced its prediction with the ground-truth value. We also reported subgroup results for papule and pigment network.

We thank the reviewers for the questions that led to these additions and the Area Chair for considering our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.