acceptodds
Under review as a conference paper at ICLR 2027

Visual Branch is What You Need for CLIP-based Class-Incremental Learning

Abstract

Class-Incremental Learning (CIL) requires models to recognize new classes over time without forgetting previously learned ones. With the rise of vision-language pre-training, CLIP has become a strong foundation for CIL. A common design in CLIP-based CIL is to construct textual classifier weights by encoding class-name templates with the CLIP text encoder, and then classify visual features by image-text cosine similarity. This design is appealing: since CLIP aligns images and text in a shared embedding space, textual weights appear to provide an off-the-shelf classifier for incremental classes. However, we show that this seemingly natural design is not always beneficial, as a modality gap can still separate the two modalities and make textual classifier weights deviate from visual class distributions. Empirically, under identical task-wise CIL training, initializing the cosine classifier with visual class centers yields lower loss and better incremental accuracy than using CLIP textual features. Motivated by these observations, we propose VIS (Visual Incremental SVM), a visual-only method for CLIP-based CIL that removes the deployed textual branch and constructs the incremental classifier entirely in the visual space. To obtain stronger task-adaptive visual representations, VIS uses only base-session data to enhance CLIP's final visual representation with informative visual-layer features. Built on the enhanced visual representation, VIS employs a simple kernelized incremental least-squares SVM, whose classifier weights are solved in closed form from additive sufficient statistics. When new classes arrive, VIS accumulates their sufficient statistics and recomputes the classifier weights for all seen classes, enabling efficient incremental updates while preserving historical class knowledge. Extensive experiments show that VIS achieves state-of-the-art performance without a textual branch.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual leaning, clip, class incremental learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_BvtkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether CLIP’s text branch is actually necessary for class-incremental learning and finds that textual classifier weights can be suboptimal because of the modality gap between visual and textual representations. Motivated by this observation, the authors propose VIS (Visual Incremental SVM), which removes the text branch and performs incremental classification entirely in the visual space. VIS further improves the visual representation by combining informative features from multiple CLIP visual layers and learning a task-adaptive representation using only base-session data. Finally, it employs a kernelized incremental least-squares SVM whose classifier can be updated using additive sufficient statistics, allowing new classes to be incorporated while preserving knowledge of previously learned classes and reducing catastrophic forgetting. Experiments show that this simple visual-only approach achieves state-of-the-art class-incremental learning performance without relying on textual classifiers.

Critical strengths

The paper makes three main contributions. First, it introduces a visual-only approach to CLIP-based class-incremental learning, eliminating the textual branch and constructing the classifier entirely in the visual space. Second, it develops a task-adaptive visual representation by integrating features from multiple visual layers and adapting them using only the base-session data, providing stronger representations for incremental learning. Third, it employs an incremental least-squares SVM (LS-SVM) with additive sufficient statistics, enabling efficient classifier updates across new classes while preserving previously learned knowledge and mitigating catastrophic forgetting.

Critical weaknesses

major

  • It is unclear how the proposed method specifically addresses catastrophic forgetting, which is a central challenge in class-incremental learning. The paper does not sufficiently analyze the method from this perspective. If the incremental LS-SVM update is the primary mechanism for mitigating forgetting, the methodological novelty may be limited, while much of the performance improvement appears to come from feature adaptation and empirical observations.

  • Since visual adaptation is performed only on base-session data, the learned representation may not optimally capture new classes or distributions introduced in later sessions.

  • While the paper shows that visual classifiers outperform text-based ones in its setting, completely removing the text branch may discard useful semantic knowledge. Improving visual-text alignment or adapting textual prototypes could provide a better alternative.

minor

  • Explain more what are the major differentiators from the prior works in detail in terms of how they maintain the prior learning.
Questions for authors

What is the major contribution of the proposed method for maintaining the previous learning? Is that because of you are using LS-SVM, instead of SVM or incremental weight update? If it is, is there any supporting experiment for this? The resolution to prevent catastrophic forgetting is core for solving incremental learning problem but this part is not focused in the paper.

Reviewer_tXP1OpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes VIS, a visual-only CLIP-based class-incremental learning method combining base-session multi-layer feature fusion, a random nonlinear feature map, and an incrementally updated LS-SVM.

Critical strengths

The empirical evaluation is extensive: nine datasets, two incremental protocols, strong baselines, ablations, robustness tests, and efficiency analysis.

VIS achieves consistent accuracy gains while remaining exemplar-free and computationally efficient.

Critical weaknesses

The technical novelty over RanPAC/ACIL appears limited. Random feature expansion and closed-form least-squares updates are established; the main additions are multi-layer residual fusion and ±1 LS-SVM coding. The reported Ridge–LS-SVM differences are also relatively small.

The conclusion about textual information is broader than the evidence. “VIS + Text” is evaluated on only two datasets, and the construction and weighting of textual prototypes are insufficiently detailed. This does not establish that stronger text-guided designs are generally unnecessary.

Questions for authors

See above

Comments on the paper

1 comment

Paper Summary

This paper proposes VIS, a visual-only framework for CLIP-based class-incremental learning. Motivated by the image-text modality gap, VIS removes the textual branch at deployment and combines base-session visual representation adaptation with a kernelized incremental LS-SVM based on accumulated sufficient statistics.

Strengths

1.The paper is well-written and easy to follow, with clear and informative figures and tables.

2.The proposed method is effective and achieves strong performance across multiple CIL benchmarks.

Weaknesses

  1. The paper argues that the modality gap may negatively affect CLIP-based CIL and therefore proposes to discard the textual branch. While this motivation is reasonable, I am not fully convinced by the proposed solution. If the method ultimately relies entirely on visual representations, why not directly use a conventional pre-trained ViT instead of CLIP? The proposed approach seems to bypass the modality-gap problem rather than address it. Moreover, showing that visual classifier weights outperform textual ones does not necessarily imply that textual information should be discarded.
  2. The proposed base-session adaptation and sufficient-statistics-based classifier updates are reasonable but closely related to existing analytic continual learning methods, such as ACIL and RanPAC. Although the kernelized incremental LS-SVM achieves strong performance, its fundamental advantages over existing closed-form classifiers remain unclear.

Review Summary

Overall, this is a solid paper with clear presentation and strong empirical results. However, my main concern lies in its central motivation. While the modality gap is a valid issue in CLIP-based CIL, completely removing the textual branch seems more like circumventing the problem than solving it. The paper does not sufficiently establish why a CLIP-based framework is preferable to a conventional vision-only backbone in this setting. Together with the relatively incremental methodological contributions, these concerns weaken the overall significance of the work. From the perspective of methodological novelty, I lean toward rejection. However, considering the overall quality of the paper, my recommendation is weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.