SP-CLIP: Stable-Plastic LoRA Consolidation and Contextual Bandit Prompt Routing for Class-Incremental Learning
Abstract
Continual learning requires a model to adapt to new classes while retaining previously acquired knowledge. Although LoRA-based CLIP methods, exemplified by MG-CLIP, have shown promising performance, they remain limited in text prototype adaptation and cross-task parameter management. Fixed text templates cannot adequately accommodate instance variations and evolving visual representations, while sharing LoRA parameters across tasks may cause parameter interference and catastrophic forgetting. To address these limitations, we propose SP-CLIP, a two-stage framework for replay-free class-incremental learning that combines Stable-Plastic LoRA consolidation with LinUCB-based prompt routing. At task boundaries, visual and textual LoRA updates are consolidated into the base weights, and the LoRA modules are reinitialized to accumulate task knowledge while reducing cross-task interference. The consolidated CLIP model is then frozen, and LinUCB dynamically routes learnable prompts to construct instance-adaptive text prototypes. This design coordinates stability and plasticity through task-level parameter management and instance-level semantic adaptation.Experiments show that the two components provide complementary benefits, while SP-CLIP achieves consistent improvements over MG-CLIP on most benchmark datasets.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.