acceptodds
Under review as a conference paper at ICLR 2027

Dynamic-V2C: Editable and Continual Vision-to-Concept Bottleneck Models via Influence Functions

Abstract

Concept Bottleneck Models (CBMs) offer intrinsic interpretability by grounding predictions in human-understandable concepts. However, contemporary CBMs face three key limitations: reliance on ungrounded textdriven concepts, vulnerability to spurious correlations that require costly retraining to rectify, and severe catastrophic forgetting during continual learning. We introduce Dynamic-V2C, a unified framework that constructs a Vision-to-Concept (V2C) tokenizer and incorporates an Influence Function (IF) engine for rapid, retraining-free model editing. By isolating approximate inverse-curvature updates to the low-dimensional bottleneck and leveraging damping, our method performs rapid head-level data removal and spurious concept debugging in seconds. Furthermore, Dynamic-V2C extends to continual learning through influence-triggered vocabulary expansion and damped protection of historical directions. Experiments across six visual benchmarks demonstrate competitive predictive performance while reducing editing time by over two orders of magnitude (over 131× on CUB) relative to full retraining. Dynamic-V2C corrects worst-group spurious correlations in under two seconds, limits catastrophic forgetting to −3.1% Backward Transfer, and reduces the output-level privacy signal of removed data while leaving frozen-backbone leakage explicitly out of scope.

Primary area:
interpretability and explainable AI
Keywords:
Concept Bottleneck Model, Editable Model, Influence Function

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_99dNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Dynamic-V2C, a concept bottleneck framework that combines clustered visual prototypes, influence-based model editing, and continual vocabulary expansion. A frozen CLIP backbone provides patch features, while a trainable projection predicts prototype activations. Data removal updates both the projection and the linear classifier, including a correction for the resulting change in concept activations. Concept removal masks selected classifier inputs and updates the remaining weights. For continual learning, an influence-based signal triggers vocabulary expansion, while historical activation covariance attenuates updates along previously active directions. The paper evaluates classification, editing efficiency, output-level membership leakage, spurious-correlation debugging, and task-aware continual learning.

Critical strengths

The framework addresses a practical maintenance problem. Updating a concept projection can change the inputs seen by the classifier, so explicitly accounting for that representation shift is sensible. Connecting editing to vocabulary expansion also gives the framework a coherent use case beyond a static classification model.

The paper makes several useful scope distinctions and includes relevant checks. It explicitly describes its privacy result as output/head-level editing, acknowledges that visual clusters are not guaranteed to be semantically pure concepts, and identifies the continual-learning setting as task-aware. The appendix includes component ablations, removal-ratio experiments, a larger-backbone test, and a feature-space privacy probe. These are appropriate checks, although the concerns below limit how confidently I can interpret the results.

Critical weaknesses

The source of Table 2’s results needs clarification. Four central F1 scores exactly match Hu et al.’s published Editable CBM results, and the runtimes closely match after conversion from minutes to seconds. Yet this submission describes three-seed reimplementations on an A100, while the earlier paper reports an RTX 3090 setup and distinguishes concept-label editing from data removal. This correspondence does not establish what happened, but it needs a concrete explanation because these results support the headline 131-fold speedup.

Similar aggregate performance does not establish fidelity to retraining. Nearly identical F1 or accuracy can hide substantial differences in individual predictions, and the unedited model already has similar test utility. The RMIA experiment is useful, but needs a retrained-model reference under the same attack. A direct ablation of the representation-shift correction and prediction-level comparisons on removed, retained, and test examples would make the editing claim more convincing.

Questions for authors

Can you explain the numerical correspondence with Hu et al.’s concept-label editing results? Please provide per-seed results, timing logs, and the exact configuration for Table 2. Please distinguish any inherited reference values from newly measured results, and clarify why these values appear under a data-removal protocol here.

How closely does the edited model match the retrained model on individual predictions? Similar overall F1 scores do not necessarily mean that the models behave similarly. Please compare their predictions and confidence scores on removed, retained, and test examples, include retraining in the RMIA comparison, and test the editing procedure with the representation-shift correction disabled.

Could you clarify the exactness claim following Theorem 1? The update uses curvature from the original training objective, but removing examples generally changes that curvature. Even for strictly convex quadratic losses, a single update using the original curvature need not recover the exact retrained solution. Please distinguish the valid local derivative from the approximation used for complete removal.

What exactly is done in the CelebA debugging experiment, and what does the reported runtime include? Section 5.5 describes ranking examples and penalizing routes, while Section 4.2.2 describes masking and zeroing concept columns. Please explain which procedure produces Table 3, how the routes are selected, and whether timing includes route selection, curvature refresh, and classifier correction. For the DFR comparison, please separate feature extraction and initial training from the debugging update itself.

Reviewer_CtofOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Most concept bottleneck models (CBMs) in recent years rely on LLM-generated concepts, which are often verbose, abstract, and not visually grounded. To improve on this, this work proposes Dynamic-V2C. It clusters the patch features of CLIP into visual prototypes, forming a visually grounded vocabulary for building the concept bottleneck, and it can also work in continual learning scenarios. To mitigate forgetting in continual learning, it triggers vocabulary expansion through influence scores to add new visual prototypes, while using damped orthogonal gradient protection to restrict gradient updates along historical directions. When poisoned training data or spurious correlations are found, it performs fast data removal and concept correction through concept-shift-aware influence function (IF) edits and EK-FAC local curvature updates. The proposed method is evaluated on six benchmarks and compared with multiple baselines. It substantially shortens the editing time while maintaining competitive predictive performance, and also shows good continual learning performance.

Critical strengths
  1. The visually grounded prototype vocabulary built by clustering CLIP patch features, instead of arbitrary LLM-generated training concepts.

  2. Compared with existing methods, the proposed method greatly reduces the model editing time while maintaining competitive predictive performance.

Critical weaknesses
  1. I noticed that the font and other formatting of the paper have been modified, and I am not sure whether this is allowed.

  2. Some paragraphs are inconsistent with the figures. For example, around line 859 (Appendix D.3), the text describes Figure 5 using a "Wood Duck" example, but the CUB panel of Figure 5 actually shows a "Black-footed Albatross".

  3. The visualizations of the localization of the visual prototypes are not rich enough.

  4. The continual learning experiments are limited. More fine-grained results should be added, for example, on the Split-CUB benchmark.

Questions for authors
  1. Why do you extract patch features from the penultimate layer of CLIP-ViT? Besides, do the patch features of CLIP-ViT carry semantic information comparable to the CLS feature?

  2. After extracting the patch features and running k-means to obtain the K cluster centroids, how are the patch-to-centroid assignments pooled into the prototype targets used in Eq. 1?

  3. Why did you conduct the mixed-granularity CUB/CIFAR stream experiment?

  4. How is the adaptive threshold based on running stream statistics concretely implemented?

Reviewer_zxLMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic-V2C for fast editing and continual learning in concept bottleneck models. It builds visual prototypes from image patches, uses influence functions to edit the model, and expands the concept vocabulary while protecting previous knowledge. Experiments show faster editing with accuracy close to retraining, improved robustness to spurious correlations, and reduced forgetting.

Critical strengths

1.The paper addresses a useful problem by allowing concept bottleneck models to correct unwanted dependencies and learn new tasks over time. The motivation for combining model editing with continual learning is clear.

2.The editing and retraining comparisons use the same frozen backbone, which helps isolate the benefit of the proposed update method. Reporting prediction quality alongside runtime makes the efficiency results easier to assess.

Critical weaknesses

1.It is unclear whether the visual clusters match the names assigned to them. Appendix D.5 shows that removing important clusters hurts prediction, but this does not tell us whether their names are accurate. Please check this using the CUB and CelebA attributes, and test whether editing one concept also affects others.

2.The paper tests editing and continual learning separately. Please also test them together. For example, edit an old concept after learning several tasks, then continue training to see whether the correction holds and earlier tasks still perform well.

3.Figure 9 varies the amount of data removed, but does not show what happens when the same model is edited repeatedly. Please add this experiment to check whether errors build up and whether the results drift further from retraining after each edit.

Questions for authors

Please address the concerns above with additional qualitative and quantitative evidence in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.