- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether CLIP’s text branch is actually necessary for class-incremental learning and finds that textual classifier weights can be suboptimal because of the modality gap between visual and textual representations. Motivated by this observation, the authors propose VIS (Visual Incremental SVM), which removes the text branch and performs incremental classification entirely in the visual space. VIS further improves the visual representation by combining informative features from multiple CLIP visual layers and learning a task-adaptive representation using only base-session data. Finally, it employs a kernelized incremental least-squares SVM whose classifier can be updated using additive sufficient statistics, allowing new classes to be incorporated while preserving knowledge of previously learned classes and reducing catastrophic forgetting. Experiments show that this simple visual-only approach achieves state-of-the-art class-incremental learning performance without relying on textual classifiers.
Critical strengths
The paper makes three main contributions. First, it introduces a visual-only approach to CLIP-based class-incremental learning, eliminating the textual branch and constructing the classifier entirely in the visual space. Second, it develops a task-adaptive visual representation by integrating features from multiple visual layers and adapting them using only the base-session data, providing stronger representations for incremental learning. Third, it employs an incremental least-squares SVM (LS-SVM) with additive sufficient statistics, enabling efficient classifier updates across new classes while preserving previously learned knowledge and mitigating catastrophic forgetting.
Critical weaknesses
major
-
It is unclear how the proposed method specifically addresses catastrophic forgetting, which is a central challenge in class-incremental learning. The paper does not sufficiently analyze the method from this perspective. If the incremental LS-SVM update is the primary mechanism for mitigating forgetting, the methodological novelty may be limited, while much of the performance improvement appears to come from feature adaptation and empirical observations.
-
Since visual adaptation is performed only on base-session data, the learned representation may not optimally capture new classes or distributions introduced in later sessions.
-
While the paper shows that visual classifiers outperform text-based ones in its setting, completely removing the text branch may discard useful semantic knowledge. Improving visual-text alignment or adapting textual prototypes could provide a better alternative.
minor
- Explain more what are the major differentiators from the prior works in detail in terms of how they maintain the prior learning.
Questions for authors
What is the major contribution of the proposed method for maintaining the previous learning? Is that because of you are using LS-SVM, instead of SVM or incremental weight update? If it is, is there any supporting experiment for this? The resolution to prevent catastrophic forgetting is core for solving incremental learning problem but this part is not focused in the paper.
Paper Summary
This paper proposes VIS, a visual-only framework for CLIP-based class-incremental learning. Motivated by the image-text modality gap, VIS removes the textual branch at deployment and combines base-session visual representation adaptation with a kernelized incremental LS-SVM based on accumulated sufficient statistics.
Strengths
1.The paper is well-written and easy to follow, with clear and informative figures and tables.
2.The proposed method is effective and achieves strong performance across multiple CIL benchmarks.
Weaknesses
Review Summary
Overall, this is a solid paper with clear presentation and strong empirical results. However, my main concern lies in its central motivation. While the modality gap is a valid issue in CLIP-based CIL, completely removing the textual branch seems more like circumventing the problem than solving it. The paper does not sufficiently establish why a CLIP-based framework is preferable to a conventional vision-only backbone in this setting. Together with the relatively incremental methodological contributions, these concerns weaken the overall significance of the work. From the perspective of methodological novelty, I lean toward rejection. However, considering the overall quality of the paper, my recommendation is weak rejection.