- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on the problem of linear concept-bank expansion in exemplar-free Continual Concept Bottleneck Models (CBMs) under the class-incremental learning (CIL) setting. The authors propose LACE (Lightweight Attribution-guided Concept Evolution). Specifically, the framework stores the Concept Bank, historical CBL weights, and class prototype vectors (mean CLIP image features per class) to synthesize pseudo-features for mitigating catastrophic forgetting. To control the growing size of the concept bank, the authors formulate a gradient-path-based Concept Attribution (CA) score with a finite-step loss-decomposition bound to rank newly introduced concepts, paired with a fixed retention budget or an optional ridge-hat-matrix-based Concept Verification (CV) heuristic. However, as documented in Section 4 and Appendix C, the experimental tables report "legacy" pipeline results that suffer from implementation discrepancies.
Critical strengths
-
Managing the continuously growing Concept Bank in continual language-guided CBMs is a practical and relevant problem. The modular separation of prototype-based feature replay (storing class mean vectors to generate pseudo-features), CBL alignment, and post-hoc coordinate pruning is intuitive.
-
The paper provides a remarkably transparent theoretical audit of its own heuristics—explicitly deriving the finite-step Taylor remainder bound for CA (Proposition 3.1), exposing the "shared-shift blind spot" where opposite sample gradients cancel out, and clarifying the mathematical gap between scalar trace corrections and exact block-level leave-one-out deletion in CV (Appendix B).
Critical weaknesses
- In Section 3.3, the theoretical formulation of Concept Attribution (CA) defines the exploration update by first computing the average gradient across all training samples and then taking the sign:
which leads to the exact nonnegative simplification
in Eq. (11) and Proposition 3.1. However, as revealed in Section 4 and Appendix C (prune_model), the actual code used to generate all experimental results iterates over batches and updates each sample by its own individual gradient sign for 10 steps of size , accumulating the sample-wise gradient times displacement before averaging across samples. Mathematically, the sign of an average is not equivalent to the average of signs:
This is not a minor implementation detail; it is a distinct algorithm (measuring mean absolute sample sensitivity rather than shared-shift aggregate residual sensitivity) that alters the exploration trajectory and can substantially change the resulting concept importance rankings. Consequently, the empirical results do not validate the mathematical method proposed in Section 3.3.
- As noted in Appendix C, the historical codebase (
incremental_train) constructs the training-selection loader usingsource='test'during both training and post-pruning fine-tuning, which invalidates the results as a clean, untouched test-set evaluation. Moreover, looking at the component ablations (Table 3 and Table 6), almost the entire performance gain of LACE comes from prototype augmentation (without which collapses from to ). Under the same 900-concept budget, the proposed CA ranking () shows virtually no advantage over simple weight-magnitude pruning () and performs slightly worse than no pruning at all (). Combined with the fact thatmain_pruninguses a hardcoded fixed ratio while leaving the CV candidate-ratio array unused, the core contributions (CA and CV) lack valid empirical support.
Questions for authors
-
Can the authors re-run the core benchmarks (e.g., ImageNet-100 and CUB-200) using an implementation that strictly follows the dataset-averaged gradient sign rule in Eq. (8), while fixing the data loader issue so that checkpoint selection does not touch the test set? How much do the concept rankings and final accuracies differ between the sample-wise sign update and the dataset-mean sign update?
-
Given that simple weight-magnitude pruning achieves vs. for CA on ImageNet-100 (Table 3), and random/activation-mean pruning achieve (Table 6), what is the practical justification for the multi-step gradient exploration in CA and the hat-matrix trace approximation in CV?
3 comments
I will keep my rating.
Thank you for your reply. It did not fundamentally dispel my doubts. I will keep my rating.
Thank you for the reply. To make the status of each point you raised easy to check:
| Your concern | What we did | Result |
|---|---|---|
| W1: the code used sample-wise signs, not Eq. (8) | Reimplemented Eq. (8) exactly and reran with three seeds | 81.32% average and 79.16% final accuracy, on par with the sample-wise rule |
| W2: checkpoints were selected on the test loader | Fixed last epoch; validation split taken from the training set | No test-based selection anywhere |
| W2: the CV array was never used | Executed Eq. (13) with a Woodbury solve | Runs without forming the nC × nC matrix; comparisons rely on the fixed budget |
| Q1: how much do the two rules differ | Scored all 33 checkpoints with both rules | Rank correlation about 0.56; different deletion sets at every checkpoint |
| Q2: why CA rather than simple baselines | Compared seven rules at equal budget | CA has the highest final accuracy, and no simple baseline is ahead of CA on both measures |
In total, the response adds 48 clean-protocol training sequences and 528 task stages, covering seven ranking rules, a prototype ablation, two pruning rates, and three task orders. Each defect you documented in the archived code has been corrected and re-evaluated. With CA the bank is 10% smaller and final accuracy rises from 78.86% to 79.16%. If a specific doubt remains, we would be glad to address it, and we would be sincerely grateful if you could reconsider your rating before the discussion closes.
We reran the method on CUB-200 B100/Inc10 with an implementation that follows Eq. (8) exactly and involves no test-based selection, using three seeds per configuration.
W1 / Q1. We ran both rules under the same protocol, with no test-based selection. Their accuracies are close, as the first two rows show, but their rankings differ substantially: the rank correlation between the two scorers is about 0.56, and they delete different coordinate sets at every one of the 33 checkpoints.
W2 / Q2, on CA. At the same budget, CA reaches the highest final accuracy among the compared rules and matches magnitude in average incremental accuracy, and it exceeds the unpruned bank in final accuracy with 10% fewer coordinates. Because rules with similar accuracy keep different coordinates, the retained set should come from a rule with a known meaning, which Eq. (8) provides. Selection takes about 0.2 s per task on CPU, and inference does not recompute attribution. Prototype augmentation supplies old-class support and is needed for accuracy.
W2 / Q2, on CV. The nC × nC matrix is never formed: the trace is computed with an FP64 Woodbury solve on the local ridge reference of Appendix B. We executed Eq. (13) on CUB. Its selected budget varies strongly with the regularization setting, so the paper's comparisons rely on the fixed budget.