acceptodds
Under review as a conference paper at ICLR 2027

Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning

Abstract

Class-Incremental Learning (CIL) enables models to continuously integrate new knowledge while mitigating catastrophic forgetting. Driven by the remarkable generalization of CLIP, leveraging pre-trained vision-language models has become a dominant paradigm in CIL. However, current work primarily focuses on aligning global image embeddings (i.e., [CLS] token) with their corresponding text prompts (i.e., [EOS] token). Despite their good performance, we find that they discard the rich patch-level semantic information inherent in CLIP's encoders. For instance, when recognizing a rabbit, local patches may encode its distinctive cues, such as long ears and a fluffy tail, which can provide complementary evidence for recognition. Based on the above observation, we propose SPA (Semantic-guided Patch-level Alignment) for CLIP-based CIL, which aims to awaken long-neglected local representations within CLIP. Specifically, for each class, we first construct representative and diverse visual samples and feed them to GPT-5 as visual guidance to generate class-wise semantic descriptions. These descriptions are used to guide the selection of discriminative patch-level visual features. Building upon these selected patches, we further employ optimal transport to align selected patch tokens with semantic tokens from class-wise descriptions, yielding a structured cross-modal alignment that improves recognition. Furthermore, we introduce task-specific projectors for effective adaptation to downstream incremental tasks, and sample pseudo-features from stored class-wise Gaussian statistics to calibrate old-class representations, thereby mitigating catastrophic forgetting. Extensive experiments demonstrate that SPA achieves state-of-the-art performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, CLIP, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Lba3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPA addresses exemplar-free class-incremental learning with CLIP by combining global classification with attribute-guided local alignment. GPT-generated class attributes guide top-K patch selection, followed by optimal transport between selected patches and attribute embeddings. Task-specific projectors and global Gaussian feature replay support incremental adaptation. Experiments across nine datasets show consistent improvements, with additional analyses of shared semantic priors, class-order robustness, forgetting, and computational cost.

Critical strengths
  1. Broad empirical validation. SPA consistently outperforms the evaluated baselines across nine datasets and two initial-class settings using the same CLIP backbone, supporting its effectiveness across diverse recognition tasks.

  2. Useful controls and practical evaluation. Comparisons using identical GPT-generated attributes across five seeds help distinguish SPA’s gains from stronger semantic priors. Inference latency, memory, and scalability analyses further support its practical relevance.

Critical weaknesses

The contribution beyond closely related methods remains insufficiently established. Although PLOT and GalLoP are cited, their methodological differences from SPA are not examined in depth, and their local mechanisms are not compared under matched semantics, projectors, and calibration. Figure 4a provides only a stepwise ablation on Aircraft, without mathematically defining the “simple matching” baseline. Table 4 controls semantic priors but does not resolve this mechanism-level comparison. ENGINE, a directly relevant method using GPT-generated descriptions for CLIP-based incremental learning, is also cited but not evaluated. The results establish SPA as an effective combination, but provide insufficient evidence of its advance over the closest approaches.

Questions for authors
  1. Under identical GPT attributes, global branches, projectors, and Gaussian calibration, how does SPA compare with alternatives using PLOT-style set-based OT or GalLoP-style semantic top-K matching? Please define the “simple matching” baseline in Figure 4a and explain the omission of ENGINE.

  2. Does the local branch use temperature scaling or CLIP’s logit scale? Does Equation (15) combine pre-softmax scores or probabilities? Is (R_c(x)) constructed separately for each candidate class during both training and inference? Please provide implementation-consistent equations and pseudocode, clarifying why Algorithm 1 specifies only (T_y).

  3. New projectors in Equation (8) change the aggregated representations of old-class inputs, while Gaussian replay is applied only to the global branch. How well does the local branch retain old-class performance? Please report its standalone old-class accuracy and separately ablate projector expansion and Gaussian calibration. Table 5 evaluates the fused model, leaving their individual contributions unclear.

1 comment

Dear Reviewer,

Thank you again for your time and constructive review. In our previous responses, we have clarified SPA's differences from existing methods, its forgetting mitigation mechanisms, and key implementation details. We have also provided additional ablation studies and experimental comparisons to further address your concerns.

As the discussion phase is drawing to a close, we would greatly appreciate it if you could kindly revisit our detailed rebuttals and let us know whether these clarifications and additional experimental results have addressed your concerns. We remain fully available for any further clarification.

Thank you very much for your time and consideration.

Reviewer_gghSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets exemplar-free CIL on a frozen CLIP backbone. It argues that existing CLIP-based methods match only global [CLS]/[EOS] tokens and ignore informative patch tokens, and that using patches naively suffers from background noise and a mismatch with a global prompt. SPA prompts GPT-5.4 with representative and diverse class images to obtain attribute descriptions. It then selects the patches most similar to each class’s attributes and aligns them with the attributes via balanced optimal transport, using the transport-weighted similarity as a local logit. Task-specific projectors, summed with frozen projectors from earlier tasks, adapt both branches, and Gaussian pseudo-features calibrate the global branch. Experiments on nine datasets claim state-of-the-art results.

Critical strengths
  1. SPA ranks first across nine datasets and two split protocols (Table 6, Figs. 10 and 11). It includes useful controls: baselines given the same GPT semantics with five-seed intervals (Table 4), and attribute-guided vs random and class-name patch selection (Fig. 7b).
  2. The backbone stays frozen, the inference overhead is modest (Tables 2 and 3), and forgetting is reported even where SPA is not best (Table 5).
Critical weaknesses
  1. On Cars, SPA beats BOFA 94.43 vs 94.21, but in Fig. 4c many of which fall below 94.21, and the grid appears to be test-selected.
  2. Missing cheaper baselines: Global-only SPA across the main table. A global branch whose class embedding averages the same GPT descriptors.
  3. The image-conditioned sample construction (Eqs. (3)-(5)) is never ablated against text-only prompting.
  4. Novelty: Each component has a precedent: patch-text OT (PLOT), global plus selected-local alignment (GalLoP), cumulative projectors as in PROOF (Eq. (8)), and Gaussian calibration (PASS).
  5. The local branch has no old-class calibration.
  6. Eq. (12) applies softmax without a temperature to bounded scores, so the local term likely affects only near-ties at inference.
  7. the model is called GPT-5 in places and GPT-5.4 in others; N, M and n are overloaded; several figures report curves without values.
Questions for authors
  1. What are the Fig. 4a values, and the global-only SPA results on the main datasets?
  2. Does Eq. (12) use a temperature? How often does the local term change the final prediction?
  3. Were K and β selected on test data? How were the baselines tuned?
  4. How were the GPT semantics injected into the baselines in Table 4, and what are their results without them?
Reviewer_ssfVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on class-incremental learning and proposes SPA, a semantic-guided patch-level alignment framework based on CLIP. It first generates class-specific attribute descriptions from selected training images, then uses these descriptions to select relevant image patches, and finally applies optimal transport (OT) to align the selected patches with attribute representations. In addition, to alleviate the catastrophic forgetting, it uses task-specific projectors and Gaussian pseudo-feature replay. Experiments across nine datasets show that SPA can achieve consistent improvements over the evaluated baselines.

Critical strengths
  1. The method addresses two plausible limitations of global CLIP alignment in CIL: irrelevant background patches and the semantic mismatch between local regions and class-name prompts. Using attributes and local patches for image recognition sounds reasonable.

  2. This paper conducts extensive experiments on 9 benchmarks.

Critical weaknesses
  1. The main idea of matching local image patches with fine-grained textual descriptions seems well established in vision-language learning. For example, PLOT[1] uses local visual–textual alignment through optimal transport.

  2. The mechanism to attack the catastrophic forgetting - expanding task-specific projectors, replaying pseudo-features sampled from class-wise Gaussian statistics - largely follow common strategies. It seems that the manuscript does not clearly identify a new mechanism or insight into catastrophic forgetting. In particular, freezing historical projectors does not guarantee stable old-class representations, since newly added projectors also contribute to the aggregated output in Equation (8).

  3. The justification for constructing a representative and diverse visual sample set is insufficient. According to Eq.(4)–(5), SPA first selects the image closest to the class prototype, then selects additional images that are farthest from this representative image. However, distance from a single reference does not ensure diversity among the selected samples. Several distant images may belong to the same compact intra-class subcluster and be highly similar to one another.

Questions for authors
  1. Does the implementation use a local temperature or learnable scale? Does Equation (15) combine pre-softmax logits or post-softmax probabilities?

  2. Are patches selected independently for every candidate class when computing training and inference scores? How should the use of TyT_y in Algorithm 1 be interpreted?

  3. Under matched conditions, how much does OT improve accuracy and forgetting separately? Could the authors provide paired results across existing seeds and a comparison with ENGINE?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.