acceptodds
Withdrawn from ICLR 2027

SP-CLIP: Stable-Plastic LoRA Consolidation and Contextual Bandit Prompt Routing for Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning requires a model to adapt to new classes while retaining previously acquired knowledge. Although LoRA-based CLIP methods, exemplified by MG-CLIP, have shown promising performance, they remain limited in text prototype adaptation and cross-task parameter management. Fixed text templates cannot adequately accommodate instance variations and evolving visual representations, while sharing LoRA parameters across tasks may cause parameter interference and catastrophic forgetting. To address these limitations, we propose SP-CLIP, a two-stage framework for replay-free class-incremental learning that combines Stable-Plastic LoRA consolidation with LinUCB-based prompt routing. At task boundaries, visual and textual LoRA updates are consolidated into the base weights, and the LoRA modules are reinitialized to accumulate task knowledge while reducing cross-task interference. The consolidated CLIP model is then frozen, and LinUCB dynamically routes learnable prompts to construct instance-adaptive text prototypes. This design coordinates stability and plasticity through task-level parameter management and instance-level semantic adaptation.Experiments show that the two components provide complementary benefits, while SP-CLIP achieves consistent improvements over MG-CLIP on most benchmark datasets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, CLIP, LoRA, Prompt Learning, Contextual Bandits

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KPctOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SP-CLIP, a replay-free class-incremental learning framework built on CLIP. The method has two stages. First, visual and textual LoRA updates are learned for each task, merged into the backbone at the task boundary. Second, the consolidated CLIP model is frozen and a pool of learnable prompts is routed on a per-image basis using LinUCB. The router models the expected improvement of a prompt over a fixed text template using a linear reward model and augments this estimate with uncertainty and frequency exploration terms. The method is evaluated on CIFAR-100, ImageNet-100, ImageNet-A, ImageNet-R, and generally outperforms other baselines.

Critical strengths
  1. The paper identifies two key challenges in CLIP-based continual learning: cross-task parameter interference and sample-dependent text adaptation. It addresses these challenges through a two-stage framework that combines LoRA consolidation with instance-specific prompt routing, while avoiding replay and the need for task identity at inference.

  2. I found the formulation of prompt routing as a contextual bandit interesting. In particular, using the improvement in classification margin over a fixed template as the reward gives the router a task-relevant signal rather than relying only on feature similarity.

Critical weaknesses
  1. A concern with the LinUCB formulation is that the regression statistics are collected in a non-stationary representation space. The context is defined as ci,j=Norm(vi+γej),c_{i,j} = \mathrm{Norm}(v_i + \gamma e_j), where eje_j is computed from the learnable prompt PjP_j. Since PjP_j is continuously updated by gradient descent, historical terms stored in HjH_j and bjb_j were produced using older versions of the prompt. Moreover, the reward itself changes as the prompt improves, so the linear reward model is effectively chasing a moving target. The paper does not address the prompt drift while learning the ridge regression reward estimation.

  2. I am not fully convinced that Merge + Reset is the right mechanism for stability. Resetting the low-rank factors changes the optimization parameterization, but after merging, subsequent task updates can still interfere with previously learned knowledge, so stability is not guaranteed. The paper mainly compares against shared LoRA, whereas there are more targeted LoRA-based continual learning approaches that explicitly control cross-task interference. For example, O-LoRA learns successive tasks in approximately orthogonal low-rank subspaces [1], while InfLoRA constructs task-update subspaces to eliminate interference with previously learned tasks [2]. Comparing against these alternatives would make the motivation and claimed stability benefits considerably stronger.

  3. The most novel part of the paper, in my view, is the UCB-based prompt routing, but the gain attributable to this component appears quite modest. In Table 3, the Stage-I model with a fixed template already achieves 92.08% average accuracy on CIFAR-100, outperforming all baselines reported in Table 1. This suggests that the CLIP model is already very strong without the proposed prompt-routing component. Adding the full prompt-routing stage improves the average accuracy only to 92.27%, a gain of 0.19 percentage points. Given the small magnitude of this improvement, it is unclear from the reported results whether the gain is statistically significant, especially considering the additional complexity introduced by prompt learning, reward computation, and LinUCB routing.

[1] Wang et al., Orthogonal Subspace Learning for Language Model Continual Learning, Findings of EMNLP 2023.

[2] Liang, et al., InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. Proceedings of CVPR 2024.

Questions for authors
  1. The paper states that during evaluation, prompt selection uses the complete LinUCB score, including both the uncertainty and frequency-based exploration terms (line 271). Since the LinUCB is frozen at inference time, what is the motivation for retaining these exploration bonuses? Would it be more appropriate at test time to select prompts using only the estimated reward term, j∗=arg⁡max⁡jci,j⊤θ^j,j^* = \arg\max_j c_{i,j}^{\top}\hat{\theta}_j, while using the full UCB score only during training?

  2. There seems to be an inconsistency between Tables 3 and 4 for the CIFAR-100 Stage-I baseline. Table 3 reports the Stage-I model with fixed template at 92.08% Avg accuracy, while Table 4 reports the "Fixed Template (Stage-I)" result at 91.63%. Both appear to correspond to the same model configuration, could the authors clarify the source of this difference?

Reviewer_cjPHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage method for rehearsal-free class-incremental learning. First, visual and textual LoRA updates are merged into the CLIP weights, and the LoRA branches are reset. Then, LinUCB is used to select learnable text prompts for individual samples. Experiments cover four image benchmarks, and the paper reports component ablations and parameter overhead.

Critical strengths
  1. By consolidating LoRA parameters at task boundaries and employing dynamic text prompts during inference, the authors propose a two-stage framework with clearly defined roles for its components.This design has certain practical value and does not require task identity during inference.
  2. The combination of methods is clearly presented. The paper evaluates the proposed method on four different image classification benchmarks and compares it with multiple CLIP-based continual learning methods.
Critical weaknesses
  1. In Table 4, the full LinUCB improves both the average and final accuracy on CIFAR-100 by only 0.05 percentage points compared with the version using only reward estimates. In Table 3, the prompt stage provides average improvements of 0.11–0.58 percentage points across the four datasets. The current results make it difficult to establish that the exploration term is an important contribution.
  2. On ImageNet-R, the full method achieves 97.22 in Table 1 but 96.54 in Table 3; on ImageNet-A, the corresponding results are 88.31 and 89.32, respectively. If these numbers correspond to different random seeds or experimental settings, the paper should clearly explain the differences and provide matched comparisons.
  3. The paper lacks an AI Use Disclosure.
Questions for authors
  1. Could the authors explain the differences in the relevant performance results between Table 1 and Table 3, and further clarify whether these experiments use the same data splits, training configurations, and random seeds?
  2. Could the authors further compare the full LinUCB, exploitation-only, and other simple learned prompt-routing methods under the same prompt pool size, training budget, and random seeds?
Reviewer_ayNiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author proposed a method i.e., SP-CLIP for Class-Incremental Learning task.A LinUCB router selects one prompt for each image to construct an instance-adaptive text prototype, using a reward defined by a tanh-scaled relative margin between the prompted and fixed-template predictions. The method is evaluated on four benchmarks against nine baselines.

Critical strengths

The paper clearly identifies two complementary challenges: managing parameter updates across tasks and adapting text prototypes to individual samples. The two-stage design provides a coherent way to address these challenges separately. The adapter storage overhead remains constant with respect to the number of tasks. This is a meaningful practical advantage, and the accompanying derivation is straightforward to verify.

Critical weaknesses

1 The experiments are week to the statements with limited dataset (number of case and experiments setting) 2. Several entries appear to describe the same configuration but report different results:

  • On CIFAR-100, “Merge + Reset + fixed template” achieves 91.63 in Tables 2 and 4, but 92.08 in Table 3, whose caption explicitly defines Stage I as Merge + Reset with a fixed text template.
  • Full SP-CLIP achieves an Avg accuracy of 97.22 on ImageNet-R in Table 1, compared with 96.54 in Table 3.
  • Corresponding discrepancies also appear on ImageNet-A (88.31 vs. 89.32) and ImageNet-100 (95.41 vs. 95.35).
  • The authors’ implementation of “Shared LoRA + fixed template” achieves 91.13, below the cited MG-CLIP result of 91.78. Consequently, the gain within the ablation (+1.14) and the gain over the reported baseline (+0.49) use different reference points and should be clearly distinguished. The ImageNet-R discrepancy is especially consequential: Table 3 attributes only a +0.11 gain to Stage II, whereas the result in Table 1 would imply a +0.77 gain over the same Stage I configuration. This substantially changes the apparent contribution of the proposed routing mechanism. The authors should reconcile these discrepancies, explain any differences in experimental settings, and identify the corrected results.
  1. The evidence for the central novelty is not yet convincing.
Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.