Direction-Aware Negative Repulsion for Replay-Free CLIP Continual Learning

CLIP-based continual learning can benefit from strong pre-trained representations and often learns sequential tasks without storing old samples. A common choice is task-local cross-entropy, where the softmax contains only classes from the current task. This objective pulls an image toward its ground-truth class text and pushes it away from other current-task class texts, while old classes receive no direct negative gradients. However, we observe that the distance between old image features and their corresponding text features can still increase after learning new tasks. This indicates that current-task optimization is still associated with changes in old cross-modal representations. We find that this effect is closely related to updates of the shared image encoder, and that different negative updates have very different effects on old tasks. In particular, negatives with stronger overlap with old-task-sensitive directions exhibit higher interference risk. To capture this directional sensitivity, we construct a sample-normalized Average Gradient Outer Product (AGOP) for each old task and use it to measure the interference risk of current-task negatives. Based on this idea, we propose AGOP-Guided Repulsion Attenuation (AGRA), which preserves low-risk negative signals while selectively attenuating high-risk ones. Extensive experiments show that AGRA outperforms a wide range of CLIP-based continual learning methods and better preserves zero-shot generalization.