RG-LoRA: Residual Geometry-Guided Continual Low-Rank Adaptation

Continual learning requires a model to acquire new tasks while retaining earlier knowledge, yet sequential adaptation can cause catastrophic forgetting. This challenge persists in low-rank adaptation, where task updates may interfere with previously learned representations. We ask whether past updates can themselves provide a bounded history for guiding future adaptation. We introduce RG-LoRA, a residual geometry-guided approach to continual low-rank adaptation. RG-LoRA summarizes each learned residual into a rank-bounded history that captures the directions and strengths of past model changes. This history attenuates new updates along strongly represented directions without preventing their reuse. After each task, the learned residual is merged into the adapted model, while its geometry updates the bounded history. Across continual classification and instruction-generation benchmarks, RG-LoRA is competitive with reported baselines on T5-Large and achieves substantially higher final performance on Llama-2-7B-Chat and Llama-3.2-3B, with larger gains on longer task sequences. These results suggest that residual geometry provides a compact and effective signal for continual low-rank adaptation without requiring a separate post-task data collector.