CoLoRA-Med: Continual Medical Learning with Low-Rank Adaptation

Vision-language models must continually adapt to new tasks without catastrophic forgetting of earlier ones, a requirement that is especially critical in medical applications, where reliability depends on retaining previously acquired capabilities. Existing low-rank adaptation methods for continual learning either constrain parameter updates or isolate them per task, and thus struggle to simultaneously share knowledge across related tasks and prevent interference between conflicting ones. We introduce CoLoRA-Med, a continual learning method that decomposes a low-rank adapter into shared and task-specific components. CoLoRA-Med freezes the components learned for previous tasks and guides new updates according to their relationship with prior ones: updates aligned with related tasks are encouraged to reuse shared directions, conflicting updates are steered into separate subspaces, and changes to parameters important for earlier tasks are regularized. We evaluate CoLoRA-Med on 11 medical visual question answering datasets across seven imaging modalities and five tasks: classification, multi-label classification, detection, cell counting, and report generation. Compared with the average of five low-rank continual learning baselines, CoLoRA-Med improves final average normalized performance by 3% and reduces mean normalized forgetting from 10.2% to 6.9%, a 32% relative reduction. These results show that combining shared adaptation with selective task specialization effectively balances plasticity and stability. Hence, CoLoRA-Med moves medical AI toward systems that keep learning new clinical skills without losing the ones clinicians already rely on. Code and checkpoints will be released.