acceptodds
Desk rejected at ICLR 2027

Sequential Orthogonal Low-Rank Adaptationwith Feature Distribution Drift Compensation for Continual Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Orthogonal low-rank adaptation offers a parameter-efficient approach to class-incremental learning, but data-dependent subspace construction can introduce additional computational overhead.Moreover, orthogonality between parameter updates does not ensure stable feature representations, leaving stored historical class statistics misaligned with the evolving model and limiting the effectiveness of classifier alignment. To address these challenges, we propose Sequential Orthogonal Low-Rank Adaptation with Feature Distribution Drift Compensation. Our method sequentially allocates disjoint blocks of a shared, fixed orthonormal basis to the input-side LoRA factors, constructing mutually orthogonal adaptation subspaces without data-dependent subspace estimation. To account for the remaining feature drift, we fit a regularized residual transition using paired features extracted from current-task data before and after adaptation. This transition updates historical class means and covariances before classifier alignment, without retaining historical exemplars. By combining explicit parameter-space separation with feature-distribution correction, our framework addresses both subspace construction overhead and stale statistical representations. Experiments on standard and fine-grained class-incremental learning benchmarks demonstrate competitive performance while simplifying orthogonal subspace construction

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, Low-Rank Adaptation, Sequential Orthogonal Basis, Classifier Alignment, Feature Distribution Drift

Related papers

Discussion (0)

Sign in to comment.