acceptodds
Withdrawn from ICLR 2027

Gated Modality Fusion for CLIP-based Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Class-Incremental Learning (CIL) enables models to continuously learn new classes without forgetting previously acquired knowledge. Due to the rich and transferable visual–text representations learned by Contrastive Language–Image Pre-training (CLIP) from large-scale image–text data, CIL methods employing CLIP as a pretrained backbone have recently achieved impressive performance. However, visual and textual representations provide complementary discriminative cues, whose relative effectiveness varies across class pairs. Most existing methods typically combine them using a fixed global ratio, implicitly assuming that their relative reliability is invariant across inputs and classes. We empirically show that this assumption is suboptimal: the optimal fusion ratio varies across class pairs and may change as the representation space evolves during incremental learning. In this paper, we propose GMF-CLIP, a Gated Modality Fusion framework that adaptively integrates CLIP’s visual prototypes and textual classifiers through a learnable gating mechanism. For each input, GMF-CLIP dynamically estimates the modality contribution for each candidate class, enabling sample- and class-conditioned fusion. To sustain reliable gated fusion as the visual representation space evolves, we use the Visual Prototype Refresh (VPR) mechanism tailored to the gating network, which leverages changes in the learned fusion ratios to estimate representation shifts and calibrate old-class prototypes. Together, the gated fusion and prototype refresh mechanisms improve classifier reliability and reduce interference between old and new classes. Experimental results on four commonly used datasets demonstrate that GMF-CLIP achieves state-of-the-art performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, incremental learning, class incremental learning, catastrophic forgetting, contrastive language–image pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QCLdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In Class-Incremental Learning (CIL) using Vision-Language Models like CLIP, existing methods rely on a fixed global ratio to combine visual prototypes and text classifiers. The authors show this is suboptimal because the best fusion ratio varies across inputs and classes.To fix this, the paper introduces GMF-CLIP, a framework featuring: Gating Network (Dynamically adjusts modality contributions for each input and candidate class, supported by retention and augmentation-consistency losses.) and Visual Prototype Refresh (Automatically recalibrates old-class prototypes based on shifts in the gating ratios). Extensive experiments show that GMF-CLIP achieves state-of-the-art performance across four standard datasets.

Critical strengths

The paper introduces GMF-CLIP, The major contributions are 1) a Gated Modality Fusion framework for class-incremental learning that replaces suboptimal fixed global ratios with a lightweight gating network to dynamically adapt visual prototype and text classifier contributions per input and class and 2) Visual Prototype Refresh (VPR) mechanism to effectively prevent catastrophic forgetting. The proposed method presents state-of-the-art performance across four standard benchmarks with lower auxiliary memory and computational overhead than task-expanding architectures.

Critical weaknesses

Major

The contributions are minor and lean more toward engineering, although the method shows better performance than prior works. The proposed method essentially uses a CLIP-based CIL framework, and its primary contribution is learning per-class modality fusion weights, which can be seen as an optimization of existing systems rather than a fundamental change.

The discrepancy between visual and textual embedding distributions was already well-studied in the computer vision community in the late 2010s or early 2020s for image classification. Consequently, the paper's motivation lacks novelty. And, the reviewer is unsure if this motivation is relevant for CIL. The major goal for CIL is maintain the learning experience while accepting new experience. Reviewer is unsure the connection between the discrepancy and CIL.

The differences from prior work are not stated sufficiently.

Minor

Some figures are too small to read the text inside them.

Questions for authors

It would be better to explain what is the meaning of the equations, instead of presenting equation and relying on readers to understand the equations.

Reviewer_8kvpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an approach for class-incremental learning using CLIP. The authors identify a limit of existing methods in that they often combine the visual and textual representations using a fixed weighting. They hypothesise that different classes and inputs may benefit from different combinations of visual and textual info. Their proposed approach consists of 2 main components: gated modality fusion and visual prototype refresh (VPR). The gating mechanism learns class- and input-dependent weights to adaptively combine visual prototypes and text embeddings. VPR addresses the issue of old-class visual prototypes becoming outdated as the model learns new classes. It refreshes the prototypes using replay samples, with the degree of refresh controlled by changes in the average gating weights, which they interpret as evidence of a shift in modality reliability.
The method is evaluated on four class-incremental learning benchmarks (CIFAR-100, ImageNet-R, CUB-200 and SUN397), with results showing improvements over the compared baselines, particularly over longer task sequences. The authors also provide ablation studies examining the contributions of the different components, as well as experiments on few-shot class-incremental learning and computational efficiency.

Critical strengths
  • Well-motivated approach: the paper identifies a limitation of fixed visual-textual fusion weighting in class-incremental learning. The approach is conceptually simple and computationally efficient, with compelling motivation.
  • Reasonably strong and consistent experimental results with useful ablations: The method achieves good and reasonably consistent results across four benchmarks and multiple incremental learning settings
Critical weaknesses
  • Visual prototype refresh would benefit from further justification: I think the idea of using gating-weight changes to control old-class prototype refresh is more distinctive than simply learning adaptive fusion weights, but I think the experiments do not yet convincingly support that a change in the fusion weight is a reliable measure of visual prototype staleness. The current ablation compares VPR with a naive prototype update, but it doesn't convincingly establish that the gating-shift-guided aspect of VPR is important. It would be good to see experiments that establish that the proposed modality-shift measure correlates with actual prototype degradation.

Alternatively, comparisons with other simpler refresh strategies would help establish the value - for example:

  • Refreshing all prototypes at a fixed rate.
  • Periodically recomputing prototypes from replay samples.
  • Determining refresh intensity using embedding-space drift rather than gating changes.

Since VPR contributes substantially to the reported improvements, I think these comparisons are important for understanding whether the gating-shift-guided aspect of the approach is responsible for the gains, or whether similar improvements could be achieved through simpler prototype maintenance.

  • Limited analysis of the benefits of adaptive gating:. The experiments don’t totally establish that learning a class- and input-dependent weighting is essential to achieving the attained improvements compared to simpler forms of adaptive fusion. It would be good to extend the gating ablation with some simpler alternatives, like:
    *class-specific weights estimated from validation-set performance or class-level visual/textual similarity statistics.
    *a global input-dependent fusion weight shared across classes.
    *learned class-specific fusion weights that are independent of the input.

Additionally, the t-SNE visualisation and image-blur experiment are useful, but I believe there's room for a bit more analysis of what the gating network is actually learning. For example, the blur experiment shows that the gating weights change as image quality decreases, but it's not clear whether this leads to better classification performance. It would be good to compare accuracy with and without adaptive gating at different levels of image degradation, to see whether the gating mechanism is actually improving robustness.

Questions for authors

-How does GMF-CLIP compare with simpler learned class-specific fusion weights that do not depend on the input? This would help establish whether the proposed gating mechanism is necessary.

-Would a fixed or periodic prototype-refresh strategy achieve comparable performance to the proposed gating-shift-guided VPR? This would help establish whether the central idea behind VPR contributes meaningfully beyond refreshing old prototypes.

Reviewer_U3cZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GMF-CLIP, a CLIP-based CIL framework that addresses the limitations of fixed visual-textual fusion ratios. The method introduces a gating network to adaptively combine visual prototypes and textual classifiers based on input samples and candidate classes. It further employs a Visual Prototype Refresh (VPR) mechanism to update old-class prototypes according to changes in gating weights. Experimental results on benchmarks demonstrate that GMF-CLIP outperforms several existing baselines.

Critical strengths
  1. This paper presents a well-motivated approach that combines adaptive modality fusion with visual prototype refresh for CLIP-based CIL.
  2. The proposed GMF-CLIP achieves strong and consistent results across CIL benchmarks.
Critical weaknesses
  1. The gating network relies solely on visual features without explicitly modeling textual information or cross-modal relationships. However, given its goal of estimating the relative reliability of both modalities, the effectiveness of this visual-only design remains insufficiently justified. A comparison with a gating mechanism incorporating both visual and textual features is recommended.

  2. VPR updates visual prototypes based on gating-weight changes, while the updated prototypes subsequently influence the learning of the gating network. Such mutual dependence may cause unstable updates and amplify estimation errors during incremental learning, which is not adequately discussed.

  3. The paper lacks sensitivity analyses for several key hyperparameters, including the prototype refresh intensity (\alpha_0), refresh interval (k), and loss weights (\alpha_1,\alpha_2,\alpha_3). These parameters may interact, but their effects are not investigated. This raises concerns about the robustness of the reported results.

  4. The experiments cover at most 300 classes, while the gating network's output dimension grows with the number of classes. An additional large-scale benchmark such as ImageNet-1K would further strengthen the evaluation of the method's scalability. This is, however, a relatively minor concern given the existing evaluation.

  5. RAPF and CLG-CBM are exemplar-free methods, whereas GMF-CLIP uses replay samples. This makes the comparison less balanced, as the performance gains may partly come from access to historical data. Evaluating these baselines with the same replay budget would provide a fairer assessment of the proposed method.

Questions for authors

see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.