- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SP-CLIP, a replay-free class-incremental learning framework built on CLIP. The method has two stages. First, visual and textual LoRA updates are learned for each task, merged into the backbone at the task boundary. Second, the consolidated CLIP model is frozen and a pool of learnable prompts is routed on a per-image basis using LinUCB. The router models the expected improvement of a prompt over a fixed text template using a linear reward model and augments this estimate with uncertainty and frequency exploration terms. The method is evaluated on CIFAR-100, ImageNet-100, ImageNet-A, ImageNet-R, and generally outperforms other baselines.
Critical strengths
-
The paper identifies two key challenges in CLIP-based continual learning: cross-task parameter interference and sample-dependent text adaptation. It addresses these challenges through a two-stage framework that combines LoRA consolidation with instance-specific prompt routing, while avoiding replay and the need for task identity at inference.
-
I found the formulation of prompt routing as a contextual bandit interesting. In particular, using the improvement in classification margin over a fixed template as the reward gives the router a task-relevant signal rather than relying only on feature similarity.
Critical weaknesses
-
A concern with the LinUCB formulation is that the regression statistics are collected in a non-stationary representation space. The context is defined as where is computed from the learnable prompt . Since is continuously updated by gradient descent, historical terms stored in and were produced using older versions of the prompt. Moreover, the reward itself changes as the prompt improves, so the linear reward model is effectively chasing a moving target. The paper does not address the prompt drift while learning the ridge regression reward estimation.
-
I am not fully convinced that Merge + Reset is the right mechanism for stability. Resetting the low-rank factors changes the optimization parameterization, but after merging, subsequent task updates can still interfere with previously learned knowledge, so stability is not guaranteed. The paper mainly compares against shared LoRA, whereas there are more targeted LoRA-based continual learning approaches that explicitly control cross-task interference. For example, O-LoRA learns successive tasks in approximately orthogonal low-rank subspaces [1], while InfLoRA constructs task-update subspaces to eliminate interference with previously learned tasks [2]. Comparing against these alternatives would make the motivation and claimed stability benefits considerably stronger.
-
The most novel part of the paper, in my view, is the UCB-based prompt routing, but the gain attributable to this component appears quite modest. In Table 3, the Stage-I model with a fixed template already achieves 92.08% average accuracy on CIFAR-100, outperforming all baselines reported in Table 1. This suggests that the CLIP model is already very strong without the proposed prompt-routing component. Adding the full prompt-routing stage improves the average accuracy only to 92.27%, a gain of 0.19 percentage points. Given the small magnitude of this improvement, it is unclear from the reported results whether the gain is statistically significant, especially considering the additional complexity introduced by prompt learning, reward computation, and LinUCB routing.
[1] Wang et al., Orthogonal Subspace Learning for Language Model Continual Learning, Findings of EMNLP 2023.
[2] Liang, et al., InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. Proceedings of CVPR 2024.
Questions for authors
-
The paper states that during evaluation, prompt selection uses the complete LinUCB score, including both the uncertainty and frequency-based exploration terms (line 271). Since the LinUCB is frozen at inference time, what is the motivation for retaining these exploration bonuses? Would it be more appropriate at test time to select prompts using only the estimated reward term, while using the full UCB score only during training?
-
There seems to be an inconsistency between Tables 3 and 4 for the CIFAR-100 Stage-I baseline. Table 3 reports the Stage-I model with fixed template at 92.08% Avg accuracy, while Table 4 reports the "Fixed Template (Stage-I)" result at 91.63%. Both appear to correspond to the same model configuration, could the authors clarify the source of this difference?