- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies catastrophic forgetting in CLIP-based incremental learning without replay. The authors propose AGRA, which identifies potential gradient interference from new tasks and reduces the impact of negative categories that may harm previously learned representations. Experiments on five datasets show that AGRA outperforms several existing methods, with relatively low additional computational and memory costs.
Critical strengths
-
The paper identifies a relatively novel issue where negative gradients from new tasks can still affect old-task representations through the shared visual encoder, even when old categories are excluded from the current loss.
-
AGRA does not require storing old images and shows consistent improvements across multiple datasets with relatively low computational overhead.
Critical weaknesses
-
It is unclear whether AGOP can reliably reflect actual forgetting caused by parameter updates. Further gradient analysis and controlled experiments are needed to clarify its effectiveness and the impact of reweighting on the Softmax gradients.
-
The paper does not show whether AGRA's improvements come from AGOP-based risk selection or simply reducing negative weights.
Questions for authors
See the Weaknesses.
2 comments
Thank you to the authors for their response. I have read and considered the authors’ response and have decided to maintain my original score.
We sincerely thank you for your insightful and constructive comments on our work. We have carefully considered your concerns and would like to address them in the following responses.
W1. Further Validation of AGOP Interference Risk and Gradient Regulation
1. Relationship Between AGOP Risk and Forgetting.
Figure 3(a) shows a strong correlation between AGOP risk and old-task representation drift. To further validate this metric, we examined its relationship with old-task accuracy degradation on ImageNet-100. The additional experiment reveals a strong positive correlation, suggesting that AGOP risk also reflects actual forgetting. The experimental results are available at the following link: AGOP Risk vs. Old-task Forgetting.
Furthermore, the gradient-strength-matched interventions in Figure 3(b) show that suppressing high-risk negatives reduces immediate forgetting more effectively than suppressing low-risk or randomly selected negatives. Together, these findings provide further evidence for AGOP risk as a useful indicator of potential interference.
2. Gradient Effects of Negative Reweighting.
We agree that negative reweighting in Eq. (13) affects other class gradients through softmax normalization.
Let Z and Z denote the original and modified softmax denominators, respectively. Since αj≤1, we have Z≤Z. With stop-gradient applied to the attenuation weights, the gradients are
=\frac{e^{s_y}}{\widetilde Z}-1,\qquad \frac{\partial\mathcal L}{\partial s_j} =\frac{\alpha_j e^{s_j}}{\widetilde Z}.$$ The smaller denominator increases the positive-class probability, thereby reducing its gradient magnitude $1-e^{s_y}/\widetilde Z$. For low-risk negatives $(\alpha_j=1)$, the smaller denominator increases their gradient magnitudes. Meanwhile, attenuation reduces the relative contributions of high-risk negatives compared with low-risk negatives. This redistribution allows AGRA to place greater emphasis on lower-risk discriminative signals while mitigating potential interference with old tasks. We further compared the average absolute logit gradients of Task-local CE and AGRA on ImageNet-100 using identical model states and input batches. | Method| Positive Grad | High-risk Grad | Low-risk Grad | |-|-|-|-| | Task-local CE | 0.1426 | 0.0322 | 0.0112 | | AGRA | 0.1258 | 0.0223 | 0.0116 | | Change | −11.8% | −30.7% | +3.6% | The results show that AGRA reduces the average positive-class and high-risk negative gradients while slightly increasing low-risk negative gradients. These observations are consistent with our analysis and further clarify AGRA's gradient regulation mechanism. We will incorporate these additional analyses and results into the revised manuscript. ## W2. Additional Comparisons and Experimental Clarifications **1. Comparison with Alternative Negative Attenuation Strategies.** We compared AGRA with uniform decay, random decay, probability-based selection, and gradient magnitude-based selection on ImageNet-100 under the same training protocol. Each method was independently run three times, and we report the mean and standard deviation. | Method | Avg (%) | Last (%) | | ---------------------------------- | ------------ | ------------ | | Task-local CE | 87.31 ± 0.54 | 78.10 ± 0.48 | | Uniform Decay | 85.14 ± 0.69 | 74.75 ± 0.56 | | Random Decay | 84.19 ± 2.31 | 74.20 ± 2.15 | | Probability-based Selection | 87.28 ± 0.59 | 78.44 ± 0.47 | | Gradient Magnitude-based Selection | 86.74 ± 0.30 | 79.11 ± 0.41 | | AGRA (Ours) | 88.42 ± 0.36 | 81.05 ± 0.32 | These comparisons further evaluate the effectiveness of directional risk-aware attenuation against alternative negative attenuation strategies. **2. VTAB Protocol and Baseline Results.** For VTAB, we follow the protocol adopted by MGCLIP, consisting of five sequential tasks: RESISC45, DTD, Oxford-IIIT Pets, EuroSAT, and Oxford Flowers, with 10 classes per task. Training uses only current-task data, while evaluation covers all previously learned classes without task identity. For other baseline methods, we mainly use the results reported in the MGCLIP paper. For LfI, we directly adopt the results available in its original paper and reproduce the results on datasets not covered in that paper using its publicly available code. We also independently reproduced MGCLIP over multiple runs and report its mean performance. We will clarify the sources of baseline results and experimental protocols in the revised manuscript. We also plan to release our code and experimental configurations upon publication to facilitate reproducibility.