acceptodds
Under review as a conference paper at ICLR 2027

Direction-Aware Negative Repulsion for Replay-Free CLIP Continual Learning

Abstract

CLIP-based continual learning can benefit from strong pre-trained representations and often learns sequential tasks without storing old samples. A common choice is task-local cross-entropy, where the softmax contains only classes from the current task. This objective pulls an image toward its ground-truth class text and pushes it away from other current-task class texts, while old classes receive no direct negative gradients. However, we observe that the distance between old image features and their corresponding text features can still increase after learning new tasks. This indicates that current-task optimization is still associated with changes in old cross-modal representations. We find that this effect is closely related to updates of the shared image encoder, and that different negative updates have very different effects on old tasks. In particular, negatives with stronger overlap with old-task-sensitive directions exhibit higher interference risk. To capture this directional sensitivity, we construct a sample-normalized Average Gradient Outer Product (AGOP) for each old task and use it to measure the interference risk of current-task negatives. Based on this idea, we propose AGOP-Guided Repulsion Attenuation (AGRA), which preserves low-risk negative signals while selectively attenuating high-risk ones. Extensive experiments show that AGRA outperforms a wide range of CLIP-based continual learning methods and better preserves zero-shot generalization.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Kyc4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies catastrophic forgetting in CLIP-based incremental learning without replay. The authors propose AGRA, which identifies potential gradient interference from new tasks and reduces the impact of negative categories that may harm previously learned representations. Experiments on five datasets show that AGRA outperforms several existing methods, with relatively low additional computational and memory costs.

Critical strengths
  1. The paper identifies a relatively novel issue where negative gradients from new tasks can still affect old-task representations through the shared visual encoder, even when old categories are excluded from the current loss.

  2. AGRA does not require storing old images and shows consistent improvements across multiple datasets with relatively low computational overhead.

Critical weaknesses
  1. It is unclear whether AGOP can reliably reflect actual forgetting caused by parameter updates. Further gradient analysis and controlled experiments are needed to clarify its effectiveness and the impact of reweighting on the Softmax gradients.

  2. The paper does not show whether AGRA's improvements come from AGOP-based risk selection or simply reducing negative weights.

Questions for authors

See the Weaknesses.

2 comments

We sincerely thank you for your insightful and constructive comments on our work. We have carefully considered your concerns and would like to address them in the following responses.

W1. Further Validation of AGOP Interference Risk and Gradient Regulation

1. Relationship Between AGOP Risk and Forgetting.

Figure 3(a) shows a strong correlation between AGOP risk and old-task representation drift. To further validate this metric, we examined its relationship with old-task accuracy degradation on ImageNet-100. The additional experiment reveals a strong positive correlation, suggesting that AGOP risk also reflects actual forgetting. The experimental results are available at the following link: AGOP Risk vs. Old-task Forgetting.

Furthermore, the gradient-strength-matched interventions in Figure 3(b) show that suppressing high-risk negatives reduces immediate forgetting more effectively than suppressing low-risk or randomly selected negatives. Together, these findings provide further evidence for AGOP risk as a useful indicator of potential interference.

2. Gradient Effects of Negative Reweighting.

We agree that negative reweighting in Eq. (13) affects other class gradients through softmax normalization.

Let ZZ and Z~\widetilde Z denote the original and modified softmax denominators, respectively. Since αj≤1\alpha_j\leq1, we have Z~≤Z\widetilde Z\leq Z. With stop-gradient applied to the attenuation weights, the gradients are

=\frac{e^{s_y}}{\widetilde Z}-1,\qquad \frac{\partial\mathcal L}{\partial s_j} =\frac{\alpha_j e^{s_j}}{\widetilde Z}.$$ The smaller denominator increases the positive-class probability, thereby reducing its gradient magnitude $1-e^{s_y}/\widetilde Z$. For low-risk negatives $(\alpha_j=1)$, the smaller denominator increases their gradient magnitudes. Meanwhile, attenuation reduces the relative contributions of high-risk negatives compared with low-risk negatives. This redistribution allows AGRA to place greater emphasis on lower-risk discriminative signals while mitigating potential interference with old tasks. We further compared the average absolute logit gradients of Task-local CE and AGRA on ImageNet-100 using identical model states and input batches. | Method| Positive Grad | High-risk Grad | Low-risk Grad | |-|-|-|-| | Task-local CE | 0.1426 | 0.0322 | 0.0112 | | AGRA | 0.1258 | 0.0223 | 0.0116 | | Change | −11.8% | −30.7% | +3.6% | The results show that AGRA reduces the average positive-class and high-risk negative gradients while slightly increasing low-risk negative gradients. These observations are consistent with our analysis and further clarify AGRA's gradient regulation mechanism. We will incorporate these additional analyses and results into the revised manuscript. ## W2. Additional Comparisons and Experimental Clarifications **1. Comparison with Alternative Negative Attenuation Strategies.** We compared AGRA with uniform decay, random decay, probability-based selection, and gradient magnitude-based selection on ImageNet-100 under the same training protocol. Each method was independently run three times, and we report the mean and standard deviation. | Method | Avg (%) | Last (%) | | ---------------------------------- | ------------ | ------------ | | Task-local CE | 87.31 ± 0.54 | 78.10 ± 0.48 | | Uniform Decay | 85.14 ± 0.69 | 74.75 ± 0.56 | | Random Decay | 84.19 ± 2.31 | 74.20 ± 2.15 | | Probability-based Selection | 87.28 ± 0.59 | 78.44 ± 0.47 | | Gradient Magnitude-based Selection | 86.74 ± 0.30 | 79.11 ± 0.41 | | AGRA (Ours) | 88.42 ± 0.36 | 81.05 ± 0.32 | These comparisons further evaluate the effectiveness of directional risk-aware attenuation against alternative negative attenuation strategies. **2. VTAB Protocol and Baseline Results.** For VTAB, we follow the protocol adopted by MGCLIP, consisting of five sequential tasks: RESISC45, DTD, Oxford-IIIT Pets, EuroSAT, and Oxford Flowers, with 10 classes per task. Training uses only current-task data, while evaluation covers all previously learned classes without task identity. For other baseline methods, we mainly use the results reported in the MGCLIP paper. For LfI, we directly adopt the results available in its original paper and reproduce the results on datasets not covered in that paper using its publicly available code. We also independently reproduced MGCLIP over multiple runs and report its mean performance. We will clarify the sources of baseline results and experimental protocols in the revised manuscript. We also plan to release our code and experimental configurations upon publication to facilitate reproducibility.

Thank you to the authors for their response. I have read and considered the authors’ response and have decided to maintain my original score.

Reviewer_5NqZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AGOP-Guided Repulsion Attenuation (AGRA), a CLIP-based continual learning method. The proposed method attempts to construct a sample-normalized Average Gradient Outer Product (AGOP) for each old task and use it to measure the interference risk of the current task. This paper discusses an interesting topic and formulates a unique algorithm to handle modality similarity drift. In general, this paper is well presented and easy to follow. However, it has several issues that need clarification. Please see weaknesses and questions.

Critical strengths
  1. It proposes a unique idea based on Average Gradient Outer Product (AGOP).

  2. The paper is well presented and easy to follow.

  3. The proposed method is evaluated with respective CL datasets, and compared to the existing CLIP based methods.

Critical weaknesses
  1. From my perspective, the novelty of the proposed method is not strong enough for a top-tier venues such as ICLR. The main idea of the proposed mechanism (section 4.1 and 4.2) is similar to LGCL, where LGCL utilize previous task-wise and class-wise prototypes as loss anchors, while AGRA utilize the historical matrices carrying feature gradients as its elements.

  2. Performance Issue: Table 1 shows main results on 5 CL datasets, where AGRA achieve a very small margin in compare to the existing SOTAs, i.e., less than or around 1 %, except from last average accuracy on ImageNet100. This indicates that the proposed unique ideas does not have a promising impact for CLIP based CL methods.

  3. Metrics Issue: Continual learning is the study to address catastrophic forgetting (CF). While the paper mentions and analyzes the forgetting in section 4 (immediate forgetting), It does not provide a forgetting analysis in the result and analysis section. It raise a question if the proposed method can truly handle CF?

  4. Baselines: While comparing to more than 10 existing methods, the only up-to-date baselines are MGCLIP and LfI only. Please kindly add more baselines from the newest study (2026). As the authors mention the other approach as baseline such as prompt, please include the latest prompt-based methods. Please also include LoRA based methods.

Questions for authors
  1. The proposed method is claimed as a rehearsal-free method. However, it requires to compute positive and negative similarity (as well as their drift) of the previous (old) tasks as shown in eq. 5-6. How the similarity and drift are computed in a rehearsal-free manner? In addition, matrix M_{old} containing features gradient of the previous tasks are saved as utilize to compute interference risk. Isn’t that considered as a rehearsal manner?

  2. It is still unclear to me why the model suffers from similarity drift. Which parameters are trained/tuned continuously that caused change of task representation?

  3. Could you please formulate the task/representation interference? how the similarity drift is strongly aligned with it? And how task interference lead to catastrophic forgetting?

  4. As shown in Figure 2, where the positive similarity has a stronger resilient to drift, why don’t rely on the positive similarity to minimize the interference? Or at least assign a bigger weight for the positive similarity.

  5. Could you please elaborate the essence of figure 3b? How “suppressing high-risk negatives provides the strongest protection” is shown by the figure?

LGCL: Introducing Language Guidance in Prompt-based Continual Learning [ICCV 2023]

4 comments

Thank you for your detailed, thoughtful, and constructive review.

First of all, we sincerely hope the reviewer could reconsider our paper, as some concerns appear to stem from a misunderstanding of our methodology. For example, Eqs. (5)–(6) seem to have been interpreted as part of AGRA's training procedure. However, these equations are used exclusively for the diagnostic analysis in Section 4.1 and are never involved in actual training. We are concerned that this misunderstanding may affect the interpretation of our core mechanism and, consequently, the assessment of our novelty. To facilitate a clearer understanding, we provide a brief overview of our key contributions:

(1) Observation: Even when old classes are entirely excluded from the current-task loss, negative repulsion within the current task can still cause old-task representation drift through shared visual encoder updates.

(2) Mechanism: We further identify the direction-dependent nature of this interference. The overlap between current-task negative gradients and old-task-sensitive directions is an important indicator of potential interference risk.

(3) Method: Based on this finding, we propose AGRA, which uses historical gradient-direction statistics to identify and selectively attenuate high-risk negatives. This mitigates implicit cross-task interference without replaying old samples.

We hope these clarifications help convey our research motivation and core contributions more accurately. We greatly appreciate your valuable time and willingness to consider our explanations and additional evidence. In the following, we will address your concerns and questions one by one.

W1. Novelty of AGRA and Its Distinction from LGCL.

We understand the reviewer's perspective in comparing AGRA with LGCL and would like to further clarify their differences. First, leveraging historical information is a common practice in continual learning to mitigate catastrophic forgetting. Therefore, we believe that the use of historical information alone may not be sufficient to establish similarity between the core mechanisms of the two methods. Regarding motivation, LGCL aims to guide representation learning using historical language semantics, while AGRA investigates implicit interference under task-local optimization. Specifically, we find that even when old classes are excluded from the current-task loss, negative repulsion among current-task classes can still cause old-task representation drift through shared visual encoder updates. This interference is direction-dependent. Regarding methodology, LGCL uses task-wise and class-wise language representations as semantic anchors and introduces additional language-guided losses. In contrast, AGRA uses gradient-sensitivity statistics to estimate the potential interference risk of current-task negatives and selectively attenuates high-risk negative repulsion within the existing task-local CE. Therefore, we believe the two methods differ substantially in their research motivations, the roles of historical information, and optimization mechanisms. We hope these clarifications help highlight AGRA's distinct contribution. To make these differences clearer, we will also expand the discussion comparing AGRA with LGCL in the revised manuscript.

W2. Effectiveness of Performance Improvements.

In CLIP-based continual learning, achieving further gains over strong baselines such as MGCLIP and LfI is challenging. Importantly, we consider the more challenging setting of learning without replaying old samples. Under this constraint, AGRA achieves consistent improvements across five benchmarks and further improves zero-shot generalization. We hope you will kindly take these results, together with our analysis of direction-dependent interference, into consideration when evaluating AGRA's contributions.

W3. Catastrophic Forgetting Analysis.

We agree that evaluating catastrophic forgetting is essential for continual learning. While Section 4 analyzes Immediate Forgetting to investigate the interference mechanism, we have additionally evaluated Average Forgetting to assess long-term knowledge retention. We report comparisons with representative baselines across five benchmarks in the table below. These results provide further evidence of AGRA's effectiveness in mitigating catastrophic forgetting.

MethodCIFAR-100 ↓ImageNet-R ↓ImageNet-100 ↓ImageNet-1K ↓
MGCLIP5.575.5210.539.59
LfI4.785.3210.279.03
AGRA (Ours)4.064.909.358.71

The results show that AGRA achieves lower Average Forgetting than existing methods, further demonstrating its effectiveness in mitigating catastrophic forgetting and preserving previously learned knowledge.

W4. Comparison with Recent Methods.

We further considered three recent methods: AREA (ICML 2026)[1], RLAP-CLIP (ICLR 2026)[2], and LoDA (ICML 2026)[3]. AREA and LoDA use LAION-400M-pretrained CLIP, whereas AGRA uses OpenAI CLIP. RLAP-CLIP relies on 20 stored exemplars per class, which differs from our replay-free setting. We therefore selected AREA and LoDA for additional comparison and reproduced both on ImageNet-100 using the same OpenAI CLIP ViT-B/16 backbone, 10-task split, and class orders, while retaining their original optimization strategies. The results are shown below. We will extend these comparisons to the remaining benchmarks in the revised manuscript.

MethodAvg (%)Last (%)
MGCLIP87.3178.38
LfI87.2678.10
AREA86.9277.89
LoDA86.4777.24
AGRA (Ours)88.4281.05

The results demonstrate that AGRA achieves the best performance in both Avg and Last Accuracy, further validating the effectiveness of our proposed direction-aware negative repulsion mechanism.

[1] AREA: Attribute Extraction and Aggregation for CLIP-Based Class-Incremental Learning (ICML 2026)

[2] RLAP-CLIP: Continual Multimodal Learning with Prototype Adaptation and Difficulty-Aware Routing (ICLR 2026)

[3] Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning (ICML 2026)

Q1. Clarification of Historical Task Analysis and Replay-Free Setting.

We would like to clarify the distinction between mechanism analysis and training. First, the old-task similarities and their drift in Eqs. (5)–(6) are used solely for the diagnostic analysis in Section 4.1 and are not involved in AGRA's training. Second, the sensitivity matrix MkM_k is computed using current-task data at the end of task kk. Subsequent tasks only use the aggregated matrix MoldM_{\mathrm{old}} to estimate interference risk. This does not constitute rehearsal, as AGRA retains only aggregated gradient statistics without storing or replaying old samples or features. This follows the same principle as regularization-based methods that preserve historical importance statistics. Moreover, the matrix requires only approximately 1 MB of FP32 memory for 512-dimensional CLIP features and supports recursive updates, keeping storage overhead constant across tasks. Therefore, AGRA satisfies the replay-free continual learning setting. We will clarify this in the revised manuscript.

Q2. Cause of Similarity Drift.

Similarity drift mainly arises from updates to the shared visual encoder during continual learning. As described in Section 5.1, we insert rank-8 LoRA adapters into the Key and Value projections of the visual Transformer's attention layers and continuously update these parameters across tasks, while keeping the pretrained backbone weights and text encoder frozen. Consequently, although old-class text features remain unchanged, the updated LoRA parameters alter old-task image representations, leading to changes in image-text similarities. Importantly, even when old classes are excluded from the current-task loss, current-task negative gradients can still indirectly affect old-task representations through shared visual encoder updates. This mechanism is further explained by the gradient analysis in Appendix B.1.

Q3. Task Interference and Catastrophic Forgetting.

From an optimization perspective, task interference can be characterized by the change in old-task loss caused by new-task updates: ΔLold≈−η∇θLold⊤∇θLnew.\Delta \mathcal{L}_{old} \approx -\eta \nabla_{\theta} \mathcal{L}_{old}^{\top} \nabla_{\theta} \mathcal{L}_{new}. Conflicting gradients can increase the old-task loss. Specifically, current-task negative gradients update the shared visual encoder, causing old-task representation and similarity drift. Such drift can lead to catastrophic forgetting when it disrupts old-class discrimination. AGRA uses AGOP to measure the overlap between current-task negative gradients and old-task-sensitive directions. Figure 3(a) shows a strong correlation between the estimated risk and old-task feature drift (Spearman ρ=0.802ρ = 0.802). Figure 3(b) further confirms that suppressing high-risk negatives reduces feature drift and immediate forgetting. These findings support our proposed direction-dependent interference mechanism.

Q4. Why Focus on Negative Repulsion?

Figure 2 shows that positive similarity stabilizes early, while negative similarity continues to decrease and is associated with old-task representation drift. This motivates our focus on negative repulsion. Since task-local CE already encourages positive alignment, increasing the positive weight does not directly address harmful negative gradients. AGRA instead selectively attenuates high-risk negatives using historical directional sensitivity while preserving the positive term and low-risk negative signals. Figure 3(b) further supports this choice, showing that suppressing high-risk negatives more effectively reduces old-task drift and immediate forgetting.

Q5. Interpretation of Figure 3(b).

Figure 3(b) aims to validate the direction-dependent nature of negative interference while controlling for gradient magnitude. Specifically, we suppress High-risk, Low-risk, and Random negatives under matched gradient strength and compare them against the unsuppressed baseline (None). Suppressing High-risk negatives achieves the strongest protection across all three metrics: immediate forgetting decreases by 0.93 percentage points, old-task feature drift decreases by 0.018, and the old-task negative similarity change improves by 0.015. In contrast, suppressing Low-risk or Random negatives yields substantially smaller benefits. These results demonstrate that negatives with comparable gradient magnitudes can have different interference effects, highlighting the importance of directional sensitivity in identifying high-risk negatives.

Reviewer_5NqZOpenReview ↗

Official Comment by Reviewer 5NqZ

Thank you for your response,

While the authors address some of my concerns, several issues, mainly Q2, Q3, and W2, remain unclear. Empirical observations/plots will be helpful to support your arguments. Thus, I keep my score.

Reviewer_WJhwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the catastrophic forgetting problem in CLIP-based continual learning. It argues that new task gradients update the shared image encoder, causing old image features to drift away from their corresponding text representations. The paper propose AGRA which reduces high risk negative gradient contributions while leaving the low-risk ones. It first uses AGOP matrix to describe sensitive feature directions, then measures how strongly each current-task negative gradient overlaps with historical sensitive directions, finally reduces high-risk negative terms in cross-entropy while leaving their low-risk weights unchanged. The method is evaluated on CIFAR-100, ImageNet-R, ImageNet-100, ImageNet-1K, and VTAB under replay-free continual learning. AGRA consistently improves average accuracy over its MGCLIP baseline.

Critical strengths
  1. The research question is solid, the paper addresses a meaningful limitation of existing CLIP continual learning methods.
  2. The paper offers a more fine-grained mechanism rather than globally restricting optimization. Both magnitude and directional sensitivity have identifiable roles. Its selectivity differs from MG-CLIP.
  3. The performance gains are consistent across multiple benchmarks.
Critical weaknesses
  1. In ablation experiments 1, although the combined magnitude-and-direction formulation achieves the best performance, the direction-only variant performs worse than magnitude-only, and both individual variants underperform the MGCLIP baseline in Average Accuracy. These results weaken the core claim that directional sensitivity is useful. The paper does not sufficiently explain these observations.
  2. In eq (13), AGRA modifies the task-local loss, it indeed attenuates high-risk negative repulsion while preserving low-risk negative signals, but the effects on current-task class and non-target class gradients are different. For current-task class, the gradient magnitude decreases; for low-risk non-target class, the gradient magnitude increases. The authors should provide further analysis of these effects.
Questions for authors

See weakness.

1 comment

We sincerely thank you for your careful review, constructive feedback, and positive assessment of our work. Your comments provide valuable insights for further improving our paper. We have carefully considered your concerns and address them below.

W1. Ablation Analysis of Interference Risk

We understand your concern regarding the ablation results. The key idea of AGRA is to estimate interference risk by combining gradient magnitude and directional sensitivity. Using direction alone may overestimate the risk of weak gradients, while magnitude alone may suppress strong negatives that have limited impact on old tasks. Both can weaken useful learning signals. In contrast, the combined risk considers both gradient strength and its overlap with old-task-sensitive directions. Figure 4 shows that the combined strategy improves Avg and Last Accuracy by 2.05 and 1.95 percentage points over magnitude-only attenuation, respectively. Furthermore, Figure 3(b) shows that suppressing high-risk negatives reduces forgetting more effectively under matched gradient strength. Together, these results support the complementary role of directional information and motivate our joint risk formulation. We have revised the discussion in Section 5.4 of the main manuscript to clarify these findings.

W2. Gradient Effects of Negative Reweighting

We agree with your observation regarding the gradient changes in Eq. (13) and would like to clarify their effects. Let Z~\widetilde Z denote the modified softmax denominator. Since the attenuation weights are treated as constants during backpropagation, the gradients are

=\frac{e^{s_y}}{\widetilde Z}-1,\qquad \frac{\partial\mathcal L}{\partial s_j} =\frac{\alpha_j e^{s_j}}{\widetilde Z}.$$ Since $\alpha_j\leq1$, we have $\widetilde Z\leq Z$. This increases the positive-class probability and consequently reduces its gradient magnitude. Meanwhile, the gradients of low-risk negatives $(\alpha_j=1)$ increase. For high-risk negatives, the absolute gradient magnitude depends on both attenuation and softmax normalization, but their contributions relative to low-risk negatives are reduced. This gradient redistribution is consistent with AGRA's design motivation: reducing the relative influence of high-risk negative repulsion while preserving useful low-risk discriminative signals. To further validate this analysis, we compared the average absolute logit gradients of Task-local CE and AGRA on ImageNet-100 using identical model states and input batches. | Method | Positive Grad | High-risk Grad | Low-risk Grad | |---|---:|---:|---:| | Task-local CE | 0.1426 | 0.0322 | 0.0112 | | AGRA | 0.1258 | 0.0223 | 0.0116 | | Change | −11.8% | −30.7% | +3.6% | The results show that AGRA reduces the average positive-class and high-risk negative gradient magnitudes while slightly increasing low-risk negative gradients. These observations are consistent with our analysis and further clarify how AGRA redistributes optimization signals across classes. We have added the detailed gradient derivation and experimental analysis to Appendix B.6 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.