Towards Efficient Continual Learning: Task Discrimination via Incremental LDA for VLMs

Continual learning (CL), which enables vision-language models (VLMs) to incrementally acquire new knowledge from non-stationary data streams while avoiding catastrophic forgetting (CF), is commonly addressed by parameter-efficient fine-tuning (PEFT) with task-specific low-rank adapters. However, in the more realistic task-agnostic setting, such methods either rely on explicit task identities or resort to auxiliary trainable task-discrimination modules, that are computationally expensive and unreliable in performance. In this work, we identify a previously overlooked phenomenon, supported by a theoretical analysis: owing to large-scale pre-training, the intermediate-layer representations of VLMs naturally exhibit strong between-task separability. Building on this insight, we propose ILDA-CL, a lightweight framework that decouples CL into two independent pathways: (i) a training-free distribution discriminator based on Incremental Linear Discriminant Analysis (ILDA) that incrementally refines task prototypes into a discriminative subspace, thereby enabling accurate task-identity prediction; and (ii) a set of parameter-isolated task-specific adapters that, guided by the predicted task identity, acquire new knowledge without interfering with previously learned tasks. Extensive experiments demonstrate that ILDA-CL achieves state-of-the-art accuracy while incurring substantially lower computational cost compared with existing approaches.