Stability Where It Matters: Decision-Sensitive Representation in Continual Learning

Continual learning with pre-trained models commonly mitigates forgetting by constraining parameters, gradients, or entire feature representations. While effective, these strategies do not explicitly distinguish representation changes that alter historical decisions from changes that are functionally irrelevant to them. We introduce Decision-Sensitive Drift Regularization (DSDR), a training-time principle that uses the decision geometry of previously learned classifiers to separate temporal representation drift into decision-sensitive and decision-null components. DSDR selectively suppresses only the former, directly linking representation stability to preservation of historical predictions while leaving complementary directions available for plasticity. The method requires no additional inference-time module and can be integrated into parameter-efficient continual adaptation without altering the inference procedure. Experiments under strict class-incremental learning on ImageNet-R across 5, 10, and 20-task settings, together with CIFAR-100 and DomainNet, demonstrate robust performance across task horizons with reduced forgetting and competitive final accuracy. These results support decision-sensitive drift as a simple and general principle for balancing stability and plasticity in continual learning. Code will be made available upon final acceptance.