What Continual Learning Preserves: An Interventional Study of Vision Models

Continual learning requires retaining prior knowledge while acquiring new information. Beyond accuracy, prior work examines what continual learners preserve through representations, explanations, learned features, and important components. We extend this analysis to causal organization, using activation patching to measure component influences on final representations (causal profiles) and one another (effective networks). In controlled visual learning and continual self-supervised learning, preservation of this organization tracks retained performance across methods, preservation strengths and preservation targets. Two simple regularizers that preserve causal profiles or effective networks improve retention over fine-tuning. Preserving shuffled or reversed profiles can instead impair performance even when representations remain similar to their learned state. Controls further show that larger representation changes need not accompany larger accuracy losses. Decomposing performance changes across current, past, and future tasks reveals different learning–retention trade-offs. Strong weight regularization retains knowledge mainly by limiting further learning, whereas representation and causal-profile preservation allow continued acquisition. On Split-CIFAR-100, continued learning also improves previously learned tasks, allowing causal-profile preservation to outperform freezing and transferred weight-regularization baselines. On Split-ImageNet-R, causal-profile and effective-network preservation reduce forgetting relative to fine-tuning. These findings establish an intervention-based analysis framework and identify causal organization as a useful preservation target for retaining knowledge without unnecessarily restricting new learning.