RE-LoRA: Residual Evolving LoRA for Continual Learning

Rehearsal-free class-incremental learning (CIL) with frozen pre-trained models is commonly realized through Low-Rank Adaptation (LoRA), yet existing methods still face a fundamental stability-plasticity-efficiency dilemma: they either allocate task-specific low-rank components whose parameter footprint grows with the number of tasks, or keep a compact shared adapter and rely on a regularization loss term to reduce inter-task interference. We propose Residual Evolving LoRA (RE-LoRA), a rehearsal-free framework built upon a single evolving LoRA that attains strong stability and sustained plasticity under a constant parameter budget. RE-LoRA comprises three components. Residual Adapter Learning (RAL) trains each task's low-rank update as a residual that is additively composed with the frozen consolidated adapter in weight space, so that a new task refines rather than overwrites historical knowledge without any auxiliary regularization. Historical Knowledge Consolidation (HKC) then merges the residual update into the historical adapter and re-compresses the result via truncated SVD, which yields the optimal low-rank approximation under a fixed rank budget, so that the single adapter evolves across tasks while its parameter and memory footprints remain constant. Finally, Prototype-Guided Task Routing (PGTR) uses stable prototypes from the frozen backbone to identify the task and masks the logits to its classes, eliminating prediction bias in inference. We prove that HKC achieves the optimal approximation of the RAL combined adaptation in the Frobenius norm under a given rank constraint, and that its truncation error grows at most linearly without compounding over the task stream. Extensive experiments on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A across ViT-B/16, DINO, and iBOT backbones show that RE-LoRA consistently outperforms state-of-the-art LoRA-based CIL baselines on every evaluated dataset.