Distributive Similarity Composition for CLIP-based Class-Incremental Learning

Parameter-efficient fine-tuning (PEFT) provides an effective way to adapt vision-language models to sequential tasks while preserving their pre-trained backbones. However, when separate task-specific modules are maintained, task-agnostic inference in class-incremental learning typically relies on routing or retrieval to activate the appropriate module, making final predictions vulnerable to selection errors. In this paper, we exploit a distinctive property of CLIP to replace selection-based inference with task-representation composition. Since CLIP performs classification through inner-product similarities, and the inner product distributes over vector addition, the similarity of a composed visual representation can be decomposed into the additive contributions of individual task-wise representations. Based on this observation, we propose Distributive Similarity Composition for Incremental Learning (DiSCIL), a three-stage rehearsal-free framework. Specifically, DiSCIL freezes the CLIP textual space and trains a visual LoRA for each incoming task. Motivated by the observation that visual features are more crowded than their corresponding text anchors, each LoRA is optimized to produce high responses to relevant textual anchors while suppressing unrelated responses. To improve inference efficiency and stability, we further introduce an incremental two-teacher consolidation strategy that distills the previous merged LoRA and the current task LoRA into a new merged LoRA using mean squared error and cosine objectives. Consequently, the final merged LoRA requires only one visual forward pass, and its output is classified using all seen-class text anchors together with an auxiliary cumulative cosine classifier. Extensive experiments on multiple benchmarks demonstrate that DiSCIL achieves strong continual recognition performance with favorable parameter and inference efficiency.