Dynamic-V2C: Editable and Continual Vision-to-Concept Bottleneck Models via Influence Functions

Concept Bottleneck Models (CBMs) offer intrinsic interpretability by grounding predictions in human-understandable concepts. However, contemporary CBMs face three key limitations: reliance on ungrounded textdriven concepts, vulnerability to spurious correlations that require costly retraining to rectify, and severe catastrophic forgetting during continual learning. We introduce Dynamic-V2C, a unified framework that constructs a Vision-to-Concept (V2C) tokenizer and incorporates an Influence Function (IF) engine for rapid, retraining-free model editing. By isolating approximate inverse-curvature updates to the low-dimensional bottleneck and leveraging damping, our method performs rapid head-level data removal and spurious concept debugging in seconds. Furthermore, Dynamic-V2C extends to continual learning through influence-triggered vocabulary expansion and damped protection of historical directions. Experiments across six visual benchmarks demonstrate competitive predictive performance while reducing editing time by over two orders of magnitude (over 131× on CUB) relative to full retraining. Dynamic-V2C corrects worst-group spurious correlations in under two seconds, limits catastrophic forgetting to −3.1% Backward Transfer, and reduces the output-level privacy signal of removed data while leaving frozen-backbone leakage explicitly out of scope.