- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CABM, a framework for efficiently adding new concepts to an existing Concept Bottleneck Model (CBM) without full retraining. CABM freezes the encoder and existing concept predictors, initializes new concept predictors using concept-conditional feature statistics, and applies a single EK-FAC-based second-order update to the added parameters. The label predictor is also adapted to incorporate the new concepts. Experiments on five datasets show that CABM achieves accuracy comparable to full retraining while substantially reducing update time and preserving existing concept predictions.
Critical strengths
- CABM enables new concepts to be added to an existing CBM by updating only the newly introduced parameters. This avoids full retraining while keeping the existing concept predictors unchanged.
- CABM achieves accuracy comparable to full retraining across five datasets while reducing update time by 10.8–61.5×. The experiments also include comparisons with frozen-feature baselines, providing a more complete evaluation of its efficiency.
- The paper evaluates not only classification and concept accuracy but also concept intervention stability, sequential concept addition, and the effects of different initialization and curvature approximation strategies. These experiments help assess the effectiveness of the proposed method.
Critical weaknesses
- The problem of adding new concepts to existing CBMs has been studied in prior work, including CONCIL and ECBM. The proposed method combines concept-conditional initialization, EK-FAC, and trust-region clipping, all of which are established techniques. It is not clear what the main technical contribution is beyond combining these components.
- While CABM is substantially faster than full retraining, the speedup over the frozen-feature Probe-LR baseline is only 1.2–1.5×. Since both methods operate on the same frozen representation, the efficiency gain from the proposed update strategy appears relatively modest.
- The results in Appendix D.4 show only small differences between EK-FAC and diagonal curvature approximation. On AwA2 with 15 added concepts, EK-FAC achieves 90.16% label accuracy compared with 89.93% for the diagonal approximation, with nearly identical runtime. These results do not clearly justify the additional complexity of EK-FAC.
- CABM assumes that the existing encoder provides sufficient features for learning new concepts. However, the experiments mainly involve removing existing concepts and adding them back. It is unclear how well the method would handle genuinely new concepts that are not well represented by the frozen encoder.
Questions for authors
- What is the main technical difference between CABM and existing concept-incremental learning methods such as CONCIL and ECBM? In particular, what does the proposed EK-FAC-based update offer that cannot be achieved with existing analytic updates?
- CABM achieves a 1.2–1.5× speedup over the matched Probe-LR baseline. Is this improvement consistent as the number of added concepts increases? How much of the speedup comes specifically from the proposed second-order update?
- Appendix D.4 shows that diagonal curvature approximation achieves performance close to EK-FAC. In which settings does EK-FAC provide a clear advantage over simpler alternatives?
- How does CABM perform when newly added concepts are poorly represented by the frozen encoder? Are there any diagnostics that can indicate when the existing representation is insufficient for concept augmentation?
- In the sequential concept addition experiments, how does repeatedly updating the label predictor affect the contribution of previously added concepts? Does the downstream prediction or intervention drift increase as more concepts are introduced?
7 comments
Response to Reviewer 89Qh (II)
Theorem 1 bounds the error of the updated concept block relative to reoptimizing that block on the same frozen features. It separates curvature approximation, local approximation, and clipping errors. Proposition 2 establishes preservation of old concept scores and bounds changes in their downstream intervention effects under the default head update.
The manuscript already tests components of this construction. In Fig. 3(a), the concept-conditional initializer alone gives a new-concept accuracy of 80.49% and label accuracy of 68.28%; full CABM reports 89.54% and 73.01%, respectively. The initialization comparison excludes the frozen old concepts from its concept-accuracy metric. The curvature and solution diagnostics in Appendices D.4–D.5 provide complementary evidence.
W2 / Q2. Efficiency relative to Probe-LR and scaling with added concepts
We extended the comparison to much larger additions on CUB: 80 concepts added to a 32-concept base, and 64 concepts added to a 48-concept base. Each round adds eight concepts. CABM, Probe-LR, and Diagonal use the same final full classification-head refit. The encoder and concept predictors stay frozen during this refit.
For the classification head, default CABM updates only the newly added classification weights. In this comparison, we additionally refit the full classification head for all methods, while keeping the encoder and concept predictors frozen. The reported update time includes this refit.
Table R2. Large concept-set expansion with a final full-head refit.
| Setting | Method | Label accuracy (%) ↑ | New-concept accuracy (%) ↑ | Cached update time (s) ↓ | Update time from images (s) ↓ | Image-based speedup vs. Probe-LR ↑ |
|---|---|---|---|---|---|---|
| 32 → 112, 10 rounds | CABM | 79.08 | 95.49 | 74.64 | 252.47 | 1.60× |
| 32 → 112, 10 rounds | Probe-LR | 78.59 | 95.00 | 224.97 | 402.80 | 1.00× |
| 32 → 112, 10 rounds | Diagonal* | 78.67 | 95.24 | 58.38 | 236.21 | 1.71× |
| 32 → 112 | Full-model retrain | 79.40 | 95.67 | — | 15979.91† | — |
| 48 → 112, 8 rounds | CABM | 80.05 | 95.90 | 63.72 | 241.55 | 1.39× |
| 48 → 112, 8 rounds | Probe-LR | 79.88 | 95.68 | 158.16 | 335.99 | 1.00× |
| 48 → 112, 8 rounds | Diagonal | 78.72 | 94.67 | 51.11 | 228.94 | 1.47× |
| 48 → 112 | Full-model retrain | 80.11 | 96.83 | — | 8907.98† | — |
New-concept accuracy uses fixed 0.5 thresholds, consistently across methods. Cached time excludes feature extraction. Image-based time includes it. Both include the intermediate edits and the final head refit, with hyperparameters already selected. Speedup is Probe-LR time divided by the method's time.
*The starred 32 → 112 Diagonal row is a curvature-replacement ablation; the 48 → 112 Diagonal row is independently tuned. †Retraining has a broader timing scope, stated as initialization, all-parameter training, validation and checkpoint saving, excluding final test evaluation, so no strict speedup ratio is computed against that row. The number of independent base seeds is 3; insertion orders are reported separately.
For 32 → 112, CABM reaches 79.08%, compared with 78.59%, while reducing image-based update time from 402.80 s to 252.47 s. This gives a 1.60× speedup. The corresponding new-concept accuracy difference is +0.49 percentage points. We use both quality and cost to assess the benefit of the faster update.
The comparison with full retraining includes the saving from freezing the encoder. The matched Probe-LR comparison isolates the choice of how to fit the added concept block, since both methods use the same features, concept objective, and head-fitting procedure. We further separate concept-fitting time from the remaining update cost.
Table R3. Scaling from the same 32-concept base.
| Added concepts | CABM new-concept accuracy (%) | Probe-LR new-concept accuracy (%) | CABM concept-fitting time (s) | Probe-LR concept-fitting time (s) | CABM total update time (s) | Probe-LR total update time (s) | Speedup ↑ |
|---|---|---|---|---|---|---|---|
| 8 | 94.55 | 94.50 | 20.98 | 20.32 | 206.89 | 205.13 | 0.99× |
| 16 | 94.71 | 94.58 | 22.60 | 40.64 | 210.91 | 226.95 | 1.08× |
| 32 | 94.95 | 94.76 | 25.84 | 81.28 | 219.74 | 270.70 | 1.23× |
| 48 | 95.41 | 94.94 | 29.07 | 121.92 | 229.61 | 314.60 | 1.37× |
| 64 | 95.37 | 94.90 | 32.31 | 162.56 | 240.52 | 358.63 | 1.49× |
| 80 | 95.49 | 95.00 | 35.54 | 203.21 | 252.47 | 402.80 | 1.60× |
Response to Reviewer 89Qh (III)
Each point includes the stated sequence prefix and one final full-head refit. Total time uses image-based accounting. Concept-fitting time includes initialization and curvature construction; total time also includes the specified feature processing and head updates. Each total is measured directly. Hyperparameter search is excluded and reported separately.
Table R3 tests scaling within one base model. The two settings in Table R2 also test different initial concept coverage. We will distinguish these comparisons and state the range over which the measured speed advantage holds.
W3 / Q3. When does EK-FAC provide an advantage over Diagonal?
Diagonal is a competitive choice in some of the smaller edits. We therefore added the larger CUB comparisons in Table R2 and evaluated the new concepts separately from the old ones.
This distinction matters when the old outputs are fixed. An average over all concepts can hide a larger difference within the added block. We report new-concept accuracy, along with task accuracy, to make that difference visible.
In the 48 → 112 comparison, Diagonal and CABM each use the same validation-search budget: 60 candidate settings per batch, or 480 in total. CABM improves label accuracy by 1.33 percentage points and new-concept accuracy by 1.23 points. Their image-based update times are 241.55 s and 228.94 s. This provides a setting where the quality gain can be assessed against the added computation.
EK-FAC retains a structured approximation of curvature interactions that a diagonal estimate drops. Such interactions can matter for a single correction. The existing update-direction and objective-gap diagnostics in Appendix D.4–D.5 provide supporting evidence about the approximation. The new large-addition comparison tests whether the difference also affects prediction.
These results support using EK-FAC in the tested addition settings where it improves quality at an acceptable cost. We will also retain the settings where Diagonal performs similarly, so the choice between the two is clear.
W4 / Q4. New concepts that are poorly represented by the frozen encoder
We start from a pretrained visual encoder and train the base CBM using the available concept labels and task labels. The encoder is frozen when new concepts are added.
Our CUB experiments already hold out entire semantic concept groups, such as wing color, bill shape, and tail pattern, before base-model training. The prediction units for the held-out group are absent, and its concept labels are not used in the base-training loss. We do not first train a model with all concepts and then remove their outputs. These experiments therefore test concepts whose annotations were not used to train the base CBM.
Pretraining and task supervision may still provide useful features for these concepts. Whether those features are sufficient is a separate question. We address it with additional tests of representation quality.
For these tests, we use the same six CUB wing-color concepts and the same data split. The encoder is an ImageNet-pretrained Inception-v3 with 2,048-dimensional features. We compare three versions:
- Early: the checkpoint after 10 epochs of base-model training, without the six new concept labels.
- Mature: the selected checkpoint at epoch 170 from the same base-model training run, also without those labels.
- Adapted: a model initialized from the early checkpoint and trained further with the new concept supervision, allowing the encoder to change.
We then freeze each encoder and fit CABM, a converged linear Probe-LR, and a nonlinear MLP probe on its features. The same linear probe on the adapted features helps separate representation changes from differences in how a concept predictor is optimized.
Table R4. New-concept prediction under different encoder conditions.
| Encoder | Method | Label accuracy (%) ↑ | New-concept accuracy (%) ↑ | New-concept balanced accuracy (%) ↑ | AUROC (%) ↑ | BCE ↓ |
|---|---|---|---|---|---|---|
| Early, epoch 10 | CABM | 75.30 | 94.23 | 90.41 | 97.01 | 0.176 |
| Early, epoch 10 | Converged Probe-LR | 73.84 | 94.43 | 90.28 | 97.45 | 0.234 |
| Early, epoch 10 | Nonlinear MLP probe | 74.85 | 94.25 | 89.40 | 97.51 | 0.152 |
| Mature, epoch 170 | CABM | 81.52 | 95.31 | 92.01 | 97.78 | 0.153 |
| Mature, epoch 170 | Converged Probe-LR | 81.22 | 95.98 | 93.15 | 98.32 | 0.132 |
| Mature, epoch 170 | Nonlinear MLP probe | 81.36 | 95.81 | 92.82 | 98.31 | 0.120 |
| Adapted from epoch 10 | CABM | 82.27 | 95.67 | 93.59 | 97.51 | 0.149 |
| Adapted from epoch 10 | Converged Probe-LR | 81.90 | 96.23 | 93.38 | 98.40 | 0.144 |
| Adapted from epoch 10 | Nonlinear MLP probe | 82.05 | 96.11 | 93.03 | 98.48 | 0.112 |
Response to Reviewer 89Qh (I)
Thank you for the detailed questions. We address the contribution, update cost, curvature choice, representation limits, and sequential behavior below.
W1 / Q1. Contribution relative to ECBM and analytic concept-incremental methods
CABM studies how to add concept predictors to a trained CBM while keeping its encoder and existing concept predictors fixed. The new parameters need a starting point, an update objective, and a connection to the label predictor. CABM specifies these steps and studies whether one local correction can approach a reoptimized added block.
ECBM discusses concept insertion as an extension while developing concept-level removal in Sec. 4.2 (Hu et al., 2025). ECBM-Add is our explicitly constructed insertion adaptation, rather than an addition implementation supplied by ECBM.
The reference implementation appends zero-initialized affine concept predictors, including their biases. It recomputes EK-FAC on the regularized new-concept BCE objective and applies one damped correction without clipping the concept step. CABM instead starts from concept-conditional feature statistics and applies an explicit concept-step radius. Both use the same frozen representation and old concept pathway.
Both methods use the same regularized BCE objective and frozen encoder. The old concept predictors and old classification weights and bias remain fixed. We apply the same update to the new classification weights, so the comparison focuses on learning the added concept predictors.
Table R1. Direct comparison with ECBM-Add.
Dataset and addition setting: CUB, six wing-color concepts added to a 106-concept base. Independent base-model seeds: 3.
All methods start from matched base models. All-concept accuracy covers the outputs available in each model. New-concept accuracy covers only the added concepts. We use per-concept validation-selected thresholds for both old and new concepts after editing and report mean ± SD across three independent base-model seeds. Update-time accounting is feature extraction, concept fitting and head updates for editing; all-parameter training and validation for retraining, excluding base training and test evaluation.
CABM improves new-concept accuracy over this ECBM-Add adaptation by 0.67 percentage points, while label accuracy is 80.65% versus 79.94%, and update time is 186.00 s versus 277.09 s for ECBM-Add.
CONCIL takes a different route. It uses recursive analytic updates for linear-regression objectives. CABM keeps the sigmoid concept objective and, by default, a softmax label predictor. These objectives do not generally have the same closed-form solution as least squares. CABM provides an approximate local update while retaining this model form and the old concept pathway. For a ridge label head, CABM also supports an exact analytic update.
What CABM adds. Concept insertion introduces predictor parameters that have no learned values in the base model. CABM specifies their concept-conditional starting point (Eq. 9), their regularized added-block objective and finite correction (Eqs. 4–7), and a separate update of the new concept-to-label coefficients (Sec. 4.3). The resulting procedure expands the concept interface while preserving the encoder and old concept predictors. The default softmax update additionally retains the old classification coefficients and bias; the ridge alternative has a different scope and reoptimizes both coefficient blocks.