Reducing Pretrained Representation Degradation in Parameter-Efficient Continual Learning

Continual learning (CL) aims to enable models to acquire new knowledge while retaining previously learned information, yet suffers from catastrophic forgetting. With the rise of pretrained models (PTMs), recent continual learning approaches increasingly rely on parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA). However, existing approaches primarily focus on learning new tasks and mitigating inter-task interference, while paying limited attention to preserving the pretrained representations that encode previously acquired knowledge. In this work, we show that sequential low-rank updates can progressively alter these pretrained representations, resulting in degraded performance on previously learned tasks and unstable adaptation. Motivated by this observation, we propose Anchored Orthogonal Low-Rank Adaptation (AO-LoRA), which anchors task-specific orthogonal transformations to the pretrained model and accumulates them in a common reference space. This design preserves a consistent reference for adaptation across tasks while retaining the parameter efficiency of low-rank fine-tuning. Extensive experiments on four vision benchmarks demonstrate that AO-LoRA consistently achieves superior or competitive performance compared to existing LoRA-based CL methods, while requiring fewer trainable parameters. Furthermore, AO-LoRA exhibits improved training stability, particularly under large learning rates, suggesting that maintaining representation structure helps mitigate instability in continual learning. Code is provided in the supplement and will be publicly released.