- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates knowledge preservation in continual learning through intervention-defined causal organization. Using activation patching, the authors introduce causal profiles and effective networks to measure component influences and propose corresponding preservation objectives. Target-perturbation experiments and task-wise accuracy decomposition investigate the importance of component correspondence and the trade-off between retention and new knowledge acquisition.
Critical strengths
An interesting intervention-based perspective on knowledge preservation. The paper goes beyond conventional parameter and representation analyses by investigating how internal component influences evolve during continual learning. The controlled target-perturbation experiments provide an interesting starting point for understanding the relationship between internal organization and forgetting.
Critical weaknesses
-
The empirical improvements are limited and do not convincingly justify the proposed complexity. Causal-profile preservation achieves almost identical accuracy to standard representation preservation on both Split-CIFAR-100 from baseline 89.45% to Causal profile 89.46% and Split-ImageNet-R from 70.40% to 70.42%, while effective-network preservation performs worse. On Split-ImageNet-R, even simple L2 regularization and the frozen pretrained encoder outperform the proposed methods. Given that intervention-based regularization requires substantially more complicated component-level computations, the absence of clear performance benefits or quantitative computational cost comparisons makes its practical value questionable.
-
The experimental evaluation is insufficient to establish the generality of the proposed approach. The main continual-learning experiments rely on a single pretrained DINOv2 backbone and two class-split benchmarks, with only 200 training updates per task. Moreover, evaluation relies on separately trained all-class linear probes rather than a classifier that must itself learn incrementally, while the pretrained encoder already provides very strong performance. It remains unclear whether the proposed preservation mechanisms would remain effective in more challenging settings involving substantial knowledge acquisition, longer task sequences, or significant domain shifts, such as multi-domain task incremental learning (MTIL)[1].
-
The novelty and empirical competitiveness of the proposed approach are insufficiently established. PNR [2] also addresses the stability–plasticity trade-off in continual self-supervised learning through regularization, using overlapping evaluation settings such as CIFAR-100. However, the authors neither clarify the novelty of their approach relative to PNR nor provide performance comparisons against it.
[1] Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models. ICCV 2023.
[2] Regularizing with Pseudo-Negatives for Continual Self-Supervised Learning. ICML 2024
Questions for authors
Please refer to the critical weaknesses above.
Final decision
The authors did not present any rebuttal, therefore I maintain my initial rating.