Bilateral spectral Low-Rank Adaptation for Class-Incremental Learning

Merging sequential low-rank updates into a shared residual enables LoRA-based class-incremental learning (CIL) without retaining task-specific adapters, but introduces severe cross-task interference. Prior methods mitigate forgetting through subspace or gradient constraints, while recent spectral approaches mainly regulate task-local adaptation spectra, leaving the joint input–output spectral structure of the accumulated residual underexplored. In this paper, we conduct a spectral analysis of the accumulated residual to diagnose this interference. We reveal that (i) incoming updates increasingly overlap with historical left and right singular subspaces; and (ii) bilateral-overlap components are substantially more interference-prone than one-sided or complementary components. Based on these observations, we introduce SF-LoRA, a bilateral spectral filtering framework that achieves a better stability-plasticity trade-off. SF-LoRA constructs a compact spectral memory through energy-density-guided truncation, preserving informative modes without unnecessarily sacrificing plastic capacity. At each optimization step, SF-LoRA applies a closed-form proximal correction that selectively attenuates interference-prone bilateral components while granting new tasks full access to complementary orthogonal spaces. Across ImageNet-R, ImageNet-A, CIFAR-100, and CUB-200, SF-LoRA establishes a new state of the art, lifting last accuracy by up to 1.60 percentage points while reducing forgetting without retaining task-specific adapters.