COLORS: Consolidated Orthogonal LoRA Subspaces For Continually Learning Tasks

We introduce Consolidated Orthogonal LoRA Subspaces (COLORS), a continual learning framework that enables foundational models to learn an expanding set of tasks while consolidating shared skills, with minimal parameter growth and without forgetting previously learned tasks. COLORS makes no assumptions about task ordering and provably preserves previously learned tasks exactly. Under a cheaply verifiable incoherence condition between tasks, we further show that each new task can be learned to near-oracle quality. Our approach addresses a growing challenge created by the widespread use of low-rank adapters (LoRAs): while LoRAs make it easy to adapt foundation models to individual tasks, they provide no principled mechanism for organizing or consolidating knowledge across tasks. Existing continual learning methods either repeatedly update a shared subspace, causing task drift, or add new parameters for each new task, which increases model size and training costs as more tasks are learned. COLORS instead organizes LoRA updates into orthogonal subspaces that preserve previously learned task solutions while consolidating reusable structures within a compact parameter budget, $3.5\times$ smaller than separate adapters. Across challenging image-to-image transformation task sets, including image restoration, we demonstrate that COLORS continually adapts multiple foundational-model backbones while often outperforming state-of-the-art continual learning methods in reconstruction quality.