Similarity-Aware Prompt Routing for Data-Efficient Continual Learning

Continual Learning (CL) enables models to adapt to new data after deployment. Recent approaches increasingly use representations from pretrained or foundation models to reduce catastrophic forgetting as new tasks arrive. Yet, recent approaches often assume abundant per-task data or strictly non-overlapping tasks. In this paper, we address a more realistic setting where tasks have limited data and may contain overlapping concepts. In this setting, knowledge transfer must be carefully controlled to overcome low-sample noise while mitigating interference between tasks. To address this, we propose Similarity-Aware Prompt Routing (SPR), an adaptive routing framework for prompt-based continual learning that progressively builds task-similarity awareness over a pool of prompts. SPR relies on two key components: incremental global pooling, which mitigates prompt association noise through gradual prompt introduction, and instance-wise prompt masking, which separates incoming samples into those aligning with current prompts and those requiring new ones. Experiments across varying data volumes and inter-task similarities demonstrate that our method actively prevents negative transfer, enhances sample efficiency, and is broadly applicable.