Beyond Interference Control: Retaining and Structuring Knowledge for Parameter-Efficient Continual Learning

Parameter-Efficient Continual Learning (PECL) enables pretrained models to continually acquire new knowledge through lightweight adaptation, yet existing methods primarily focus on mitigating cross-task interference rather than explicitly controlling how knowledge is retained and organized throughout continual adaptation. In particular, the retention of shared knowledge is often governed by local update rules whose cumulative effect depends on the task optimization horizon, while architectural separation between shared and task-specific adaptations does not necessarily establish differentiated structural roles. To address these limitations, we propose Retaining and Structuring Knowledge (RSK), a general PECL framework that complements interference control through explicit knowledge retention and structural organization. RSK introduces Retention-Controlled Knowledge Consolidation with EMA (RKC-EMA), which parameterizes consolidation by a task-level cumulative retention target and derives the corresponding local EMA coefficient according to the optimization horizon. We theoretically establish that this formulation provides horizon-invariant cumulative retention and further characterize its effect on consolidated shared-state drift. RSK further introduces Structured Shared–Specific Knowledge Decomposition (SKD), which imposes asymmetric structural priors to encourage compact and historically stable shared adaptation while representing task-dependent variations through selective task-specific innovations. Extensive experiments on four continual-learning benchmarks demonstrate that RSK improves parameter-efficient continual adaptation and complements multiple representative PECL methods with marginal computational overhead.