acceptodds
Under review as a conference paper at ICLR 2027

Beyond Interference Control: Retaining and Structuring Knowledge for Parameter-Efficient Continual Learning

Abstract

Parameter-Efficient Continual Learning (PECL) enables pretrained models to continually acquire new knowledge through lightweight adaptation, yet existing methods primarily focus on mitigating cross-task interference rather than explicitly controlling how knowledge is retained and organized throughout continual adaptation. In particular, the retention of shared knowledge is often governed by local update rules whose cumulative effect depends on the task optimization horizon, while architectural separation between shared and task-specific adaptations does not necessarily establish differentiated structural roles. To address these limitations, we propose Retaining and Structuring Knowledge (RSK), a general PECL framework that complements interference control through explicit knowledge retention and structural organization. RSK introduces Retention-Controlled Knowledge Consolidation with EMA (RKC-EMA), which parameterizes consolidation by a task-level cumulative retention target and derives the corresponding local EMA coefficient according to the optimization horizon. We theoretically establish that this formulation provides horizon-invariant cumulative retention and further characterize its effect on consolidated shared-state drift. RSK further introduces Structured Shared–Specific Knowledge Decomposition (SKD), which imposes asymmetric structural priors to encourage compact and historically stable shared adaptation while representing task-dependent variations through selective task-specific innovations. Extensive experiments on four continual-learning benchmarks demonstrate that RSK improves parameter-efficient continual adaptation and complements multiple representative PECL methods with marginal computational overhead.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, parameter-efficient learning, catastrophic forgetting, representation learning, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B7BSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the limitation of parameter-efficient continual learning (PECL), this paper proposes the RSK framework, comprising two components:

  1. RKC-EMA: Rather than directly setting a local EMA coefficient α\alpha, RKC-EMA takes a task-level cumulative retention target RR and derives the local coefficient as αt=R1/Kt\alpha_t = R^{1/K_t} from the task optimization horizon KtK_t. It is claimed to achieve horizon-invariant cumulative retention, supported by Theorem 1.
  2. SKD: SKD imposes an asymmetric structural prior on the shared and task-specific LoRA paths. The shared branch uses a Frobenius-norm compactness term plus a fidelity term to the consolidated state θˉ\bar{\theta}; the task-specific branch applies ℓ1\ell_1 sparsity only to BB, leaving AA unconstrained.

Experiments on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A apply RSK as a plug-in on top of multiple LoRA CL methods. Results report final/average accuracy, forgetting, and stability, together with ablations, sensitivity analyses on RR and EMA decay, and computational overhead comparisons.

Critical strengths
  1. Sound problem framing and a general framework: treating “retention” and “structural organization” as dimensions distinct from interference control and designing RSK as a plug-and-play mechanism yields consistent positive gains over baselines on four benchmarks and across four backbone architectures.

  2. A comprehensive analysis of overhead and robustness. Fig. 3(c) shows that training time and memory are comparable to those of Vanilla LoRA. Tab. 2 reports performance under varying task lengths, and Tab. 3 provides ablation results across four benchmarks.

Critical weaknesses
  1. The core mechanism is essentially inactive in several experimental settings, so the claimed horizon-invariance advantage lacks effective evidence. On balanced benchmarks such as CIFAR-100, since Kt=∣Bt∣⋅Et=20∗10=200K_t = |B_t|·E_t=20*10=200 (batch * epoch), all tasks share the same KtK_t, so αt=R1/Kt\alpha_t = R^{1/K_t} is constant within a run and RKC-EMA is mathematically equivalent to standard EMA with a fixed α=R1/K\alpha = R^{1/K}. Thus the distinctive design feature may never be activated.

  2. The sensitivity comparison (Fig. 3a vs. 3b, Fig. 5) is confounded because the RR sweep 0.4, 0.9 and the decay sweep 0.9, 0.99999 cover different effective-retention ranges, the latter's low end (α=0.9,0.99\alpha = 0.9, 0.99) maps to αK≈0\alpha^K ≈ 0, below the swept RR range. As α\alpha and RR are a monotone bijection for fixed KK, the claimed advantage of RR is an artifact of the mismatched intervals, and the sweeps must be compared over the same effective-retention range.

Questions for authors
  1. The consolidated-drift bound is circular and does not support its stated implication. Theorem 1 bounds the consolidated drift only conditionally: assuming that every online shared state stays within distance δt\delta_t of the pre-task consolidated state. The premise thus already assumes the bounded deviation that the bound is meant to establish, so the result is close to being a restatement of its own hypothesis. Moreover, δt\delta_t is neither measured nor explicitly defined, and it is likely to grow with RR, since retaining more of the pre-task state may require the online states to move farther to fit new tasks. Consequently, the claimed monotonicity that a larger RR yields a tighter bound is not established.

  2. The most relevant baselines are missing. EMA-LoRA (Gao et al., 2024) and Lou et al. (2026) are prior EMA-based LoRA-merging methods that the paper itself cites as motivation. RKC-EMA is essentially a reparameterization of EMA, yet it is not compared with existing EMA methods.

  3. RSK introduces an additional shared branch on top of all base methods, whereas Algorithm 3 explicitly retains and returns all task-specific factors. Yet Section 4.2 claims: “...despite requiring neither historical task-specific parameters nor accumulated subspaces or Fisher statistics,” and the Conclusion reiterates: “...without explicitly maintaining historical task-specific parameters.”

4 comments

We sincerely thank the reviewer for the thorough evaluation and constructive feedback. We have carefully considered all the concerns and questions and provide our point-by-point responses below.

Critical Weakness 1 (Limited activation of the horizon-aware mechanism):

Regarding the limited activation of the horizon-aware mechanism: For balanced datasets such as CIFAR-100, all tasks have the same number of training samples and epochs, resulting in a constant optimization horizon KtK_t. Consequently, αt=R1/Kt\alpha_t=R^{1/K_t} also remains constant across tasks, making RKC-EMA mathematically equivalent to standard EMA with a fixed decay coefficient. In contrast, \textcolor{red}{datasets such as ImageNet-R, ImageNet-A, and DomainNet exhibit different task-size distributions, which may lead to different optimization horizons across datasets and potentially across tasks.} RKC-EMA \textcolor{red}{accommodates such variations by determining} αt\alpha_t according to the actual optimization horizon Kt\textcolor{red}{K_t}, \textcolor{red}{thereby} maintaining a consistent cumulative retention target RR. Furthermore, Fig. 3(a) and Tables 8--10 in Appendix A.7 report the sensitivity of RKC-EMA to RR across different datasets, while Fig. 3(b) shows that the optimal fixed EMA decay coefficient varies across datasets (0.999 for CIFAR-100 and 0.993 for ImageNet-R). These results highlight the dataset dependency of fixed decay coefficients and support parameterizing EMA decay through a unified cumulative retention target RR.

Critical Weakness 2 (Unmatched effective-retention ranges in sensitivity analyses):

Regarding the mismatched ranges in the sensitivity analyses: For a given optimization horizon KtK_t, the cumulative retention target RR and the EMA decay coefficient αt\alpha_t satisfy αt=R1/Kt\alpha_t=R^{1/K_t}, establishing a one-to-one correspondence between them. However, the original experiments used different sweep ranges for RR (0.4--0.9) and α\alpha (0.9--0.99999), which do not cover the same effective-retention range. Since the optimization horizons vary across datasets, we provide the corresponding values of RR and α\alpha for all four datasets in Table 9 of Appendix A.7.

Question 1 (Conditional nature and interpretation of the consolidated-drift bound):

(1) Regarding the circularity of the theoretical argument: We revised Theorem 1 and its proof by defining δt\delta_t as the maximum deviation of the online shared parameters from the initial consolidated state along the finite optimization trajectory of the current task. By unrolling the EMA recurrence and applying the triangle inequality, we directly derive Dt≤(1−R)δtD_t\leq(1-R)\delta_t, without requiring any additional a priori boundedness assumption.

(2) Regarding the empirical measurement of δt\delta_t: We added per-task drift measurements under different values of RR on ImageNet-A (Appendix A.3, Table 5). All nine tasks employing RKC-EMA satisfy the theoretical bound. For example, on Task 1, as RR increases from 0.01 to 0.99, DtD_t decreases from 3.15 to 0.03, while δt\delta_t remains approximately 3.27, demonstrating that increasing RR effectively reduces the actual EMA drift.

(3) Regarding the monotonicity claim: We clarify that the upper bound decreases with increasing RR only when the optimization trajectory is fixed (i.e., δt\delta_t remains unchanged). Across different training trajectories, neither the actual drift nor its theoretical upper bound is guaranteed to decrease monotonically.

Question 2 (Missing comparisons with existing EMA-based LoRA methods):

Existing EMA-based LoRA methods all employ standard EMA updates, but differ in their overall method designs, making it difficult to isolate the effect of the EMA strategy through direct method-level comparisons. Therefore, we have added multidimensional comparisons with standard EMA in Fig. 3(a)(b), Fig. 6, and Tables 8 and 10 in the Appendix, analyzing parameter sensitivity and update stability.

Question 3 (Inconsistent statements regarding historical task-specific parameter storage):

The original wording was imprecise and may have caused confusion. Our intention was to emphasize that the shared branch introduced by RSK does not itself require explicitly maintaining historical task-specific parameters, accumulated subspaces, or Fisher statistics. The task-specific factors retained and returned in Algorithm 3 are solely intended to preserve the original parameter structure of the integrated baseline methods, rather than indicating that the RSK shared branch depends on these historical parameters. We have clarified this distinction in the revised manuscript to avoid ambiguity.

We sincerely thank the reviewer again for the constructive comments and suggestions, which have helped us improve the experimental evaluation, theoretical analysis, and presentation of our work. We look forward to further discussion.

Thank you for the detailed response. The revised definition of δₜ and the qualification of the monotonicity claim address my concerns about the theorem’s formal statement. However, my main experimental concerns remain unresolved.

  • The benefit of horizon adaptation is still unclear. Different optimal decay values across datasets do not show that adapting the decay across tasks explains the gains. Tab.9 reports average coefficients, while Tab.2 evaluates the full framework, neither isolates the benefit of horizon adaptation.
  • The sensitivity comparison is still confounded. Tabs. 8 and 10 retain the unmatched retention ranges. Tab. 9 documents the mapping but does not correct the comparison. Normalizing accuracy does not fix this either, so the claim that RKC-EMA is less sensitive remains unsupported.
  • The relevant EMA-based baselines are still missing. Comparing against standard EMA within the proposed framework does not replace comparisons with EMA-LoRA (Gao et al., 2024) and Lou et al. (2026) . Differences in their architectures do not resolve the question of how RSK compares with these prior methods.
  • The storage claim remains inconsistent. The conclusion now refers specifically to the shared branch, but Section 4.2 (line 430) still makes the broader claim about RSK, whereas Algorithm 3 retains and returns all task-specific factors.

As these issues do not alter the main grounds for my original assessment, I regret that I must maintain the original score.

We sincerely appreciate the reviewer’s valuable time and thoughtful effort in reviewing our manuscript and providing constructive feedback. Our revisions are detailed below.

  1. To clarify the benefit of horizon adaptation, we have conducted an additional controlled ablation study to directly compare RKC-EMA with fixed EMA. Specifically, RKC-EMA dynamically adjusts the EMA decay factor according to the optimization steps of each task, using a predefined global retention target (R). In contrast, fixed EMA applies a uniform decay factor across all tasks, calculated as the arithmetic mean of the task-wise decay factors corresponding to the same retention target (R). This design enables a comparison under a relatively consistent range of EMA decay factors while isolating the effect of task-level horizon adaptation. The corresponding results have been added to Tables 11 and 12 in Appendix A.7, as detailed below.

(1). Results with RKC-EMA (Horizon Adaptation)

RĀ₁₀ (↑)Avg. (↑)Forgetting (↓)Plasticity (↑)Stability (↑)P+S
0.4060.4770.936.3257.1051.28108.38
0.4559.6770.386.3856.5050.09106.59
0.5060.8770.815.9857.6452.00109.64
0.5560.0070.457.7657.9850.36108.34
0.6059.8770.056.1856.2050.07106.27
0.6560.0070.296.2156.5250.40106.92
0.7060.4070.856.4257.0250.63107.65
0.7558.9369.666.4655.4749.22104.69
0.8059.8769.755.0954.9749.81104.78
0.8559.4769.535.6155.2249.45104.67
0.9058.9369.386.0855.3049.41104.71

(2). Results with Fixed EMA (Without Horizon Adaptation)**

R → αĀ₁₀ (↑)Avg. (↑)Forgetting (↓)Plasticity (↑)Stability (↑)P+S
0.40 → 0.9699159.6770.076.4156.3350.11106.44
0.45 → 0.9737359.0769.846.1055.4449.77105.21
0.50 → 0.9771560.7370.926.7757.8051.15108.95
0.55 → 0.9802660.7371.146.4157.4351.11108.54
0.60 → 0.9831160.6071.075.3656.8351.71108.54
0.65 → 0.9857459.7370.566.2955.6549.93105.58
0.70 → 0.9881860.3370.726.9257.6650.62108.28
0.75 → 0.9904559.6070.396.5956.1649.97106.13
0.80 → 0.9925860.4770.355.3756.4151.01107.42
0.85 → 0.9945959.4069.875.7055.2749.87105.14
0.90 → 0.9964959.2069.475.4454.4849.02103.50
  1. To address the mismatch in parameter ranges in the sensitivity comparison, we have revised Appendix A.7 by removing the direct sensitivity comparison between RKC-EMA and standard EMA, as well as the normalized fluctuation analysis. Instead, we independently discuss the performance variations of the two methods across their respective parameter ranges. We have also removed the claim that RKC-EMA is less sensitive to hyperparameter variations than standard EMA, retaining only the empirical observation that RKC-EMA exhibits relatively stable performance across the evaluated range of retention ratios.

  2. To address the missing EMA-based baselines, we have added comparisons with EMA-LoRA (Gao et al., 2024) and CaRE (Lou et al., 2026). To ensure a fair comparison, we made the following adaptations: for EMA-LoRA, we adopted its corresponding SFA framework, replaced the original CLIP backbone with the same ViT backbone used in RSK, and used the same classification head as RSK and CaRE. Furthermore, considering the architectural differences between these methods and RSK, we replaced their standard EMA with RKC-EMA while keeping all other components unchanged, in order to investigate the applicability of RKC-EMA across different architectures. The experimental results are presented in the following table. Incorporating RKC-EMA improves several evaluation metrics for both SFA and CaRE, further demonstrating its applicability across different PECL architectures.

MethodImageNet-A Ā₁₀ (↑)ImageNet-A Avg. (↑)ImageNet-R Ā₁₀ (↑)ImageNet-R Avg. (↑)
RSK (only RKC-EMA)59.0069.3276.8780.12
RSK59.4069.3477.0280.32
SFA (standard EMA)54.0766.9871.0278.30
SFA (with RKC-EMA)55.1366.8971.1578.39
CaRE (standard EMA)61.4972.4879.9883.69
CaRE (with RKC-EMA)62.2873.0680.1883.59
  1. The storage claim remains inconsistent. The conclusion now refers specifically to the shared branch, but Section 4.2 (line 430) still makes the broader claim about RSK, whereas Algorithm 3 retains and returns all task-specific factors. Regarding the inconsistency in the storage-efficiency claims, we have revised the relevant statements in the Conclusion and Section 4.2 to clarify that the storage-efficiency claim applies specifically when RSK is used independently, without being integrated with other PECL methods. Furthermore, we have revised Section 3.4 and Algorithm 3 to explicitly clarify that when RSK is combined with other PECL methods, both shared and task-specific parameters are retained and returned, ensuring consistency between Algorithm 3 and the description in Section 3.4.

We sincerely thank the reviewer again for the constructive feedback, which has helped us improve the rigor and clarity of our manuscript.

Reviewer_nxz5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses parameter-efficient continual learning and proposes two components: ‌RKC-EMA‌, which sets the EMA coefficient from a prescribed retention target so that retention does not depend on the optimization horizon, and ‌SKD‌, a structured shared–specific decomposition with asymmetric priors on the two LoRA branches. It is evaluated on four benchmarks over four PECL backbones, with ablations and efficiency measurements. The motivation is reasonable and the components are simple, but the evidence does not yet support the main claims.

Critical strengths

‌1、The problem is well posed.‌ Separating "how much interference is suppressed" from "how shared knowledge is retained and structured" is a useful distinction. 2‌、The design is simple and portable.‌ RKC-EMA is a one-line change to the consolidation rule, and SKD reduces to three regularizers with closed-form gradients; both plug into four different backbones. ‌

Critical weaknesses

1、How does RSK differ from soft-sharing-based continual learning? The paper should introduce soft-sharing-based continual learning in both the Introduction and the Related Work. 2、 Figure 1 shows that combining the baselines with SKD improves performance, but SKD is only one component of RSK; it is therefore not rigorous to attribute the improvement to RSK. At most, the figure demonstrates that the proposed method improves the performance of three baselines on image tasks—it does not substantiate the existence of the motivating phenomenon. More direct observational experiments are needed to support the research question. 3、Why is a shared LoRA designed? Would this not exacerbate catastrophic forgetting? 4、Missing related work.‌ Several relevant works are not discussed, for example CaLoRA. 5、Baselines are incomplete‌: the nearest neighbors from related work are absent, and CL-LoRA (78.72) exceeds every RSK variant on ImageNet-R. 6、The ablation and theory are thinner than advertised‌: no "SKD only" row, no numeric SKD weights, forgetting/stability never reported numerically, and Theorem 1 is a constructive identity whose bound is never linked to forgetting or accuracy.

Questions for authors

Please see the weaknesses.

4 comments

We sincerely thank the reviewer for the thorough evaluation and constructive feedback. We have carefully considered all the concerns and questions and provide our point-by-point responses below (all revisions in the manuscript are highlighted in red).

Question 1 (Positioning of RSK with respect to soft-sharing-based continual learning):

Following the your suggestion, we have expanded the Introduction and Related Work sections to include an overview of soft parameter sharing and representative methods, as detailed in Section 2.3, "Parameter Sharing in Continual Learning." Hard parameter sharing facilitates knowledge transfer by directly reusing shared parameters across tasks, whereas soft parameter sharing encourages knowledge transfer between task-specific parameters through regularization or structural constraints. RSK adopts an explicit shared–specific parameter structure, in which the shared pathway follows a hard parameter sharing mechanism, distinguishing it from typical soft parameter sharing approaches.

Question 2 (Insufficient empirical support for the motivating research question):

(1) The label Baseline + SKD in Fig. 1 was incorrect. The figure actually shows the results of Baseline + RSK. This typographical error has been corrected in the revised manuscript. We apologize for this oversight.

(2) To further substantiate our motivation and the rationale behind the Shared--Specific design, we have added a representation similarity analysis based on Linear CKA in Appendix A.6. The results show that the Shared pathway exhibits high cross-task similarity (0.903/0.904 for the K/V branches), whereas the Specific pathway exhibits substantially lower cross-task similarity (0.085/0.078 for the K/V branches). Moreover, the update representations of the two pathways also exhibit low similarity (0.0327--0.1617 for the K branch). These findings provide representation-level evidence that shared and task-specific knowledge exhibit distinct characteristics. In addition, we have conducted a more thorough literature review and expanded the Introduction and Related Work sections to discuss how existing methods model shared and task-specific knowledge, thereby further clarifying our research motivation.

Question 3 (Motivation and potential forgetting associated with shared LoRA):

Shared LoRA is introduced to facilitate cross-task knowledge reuse and reduce redundant task-specific parameters by maintaining a common adaptation pathway. Although parameter sharing may introduce interference between tasks, this risk depends on how shared parameters are updated. RSK addresses this issue through RKC-EMA, which controls the cumulative drift of shared parameters, and SKD, which structurally distinguishes shared and task-specific knowledge. Experimental results show that RSK achieves effective cross-task knowledge sharing while maintaining relatively low forgetting.

Question 4 (Incomplete related work):

We have incorporated a discussion of CaLoRA and other relevant recent methods, along with the corresponding citations, into the Introduction and Related Work sections to provide a more comprehensive review of related studies.

Question 5 (Incomplete baseline comparisons and performance on ImageNet-R):

Following the your suggestion, we have added three recent representative baselines, LoDA, SA-LoRA, and JANUS-LoRA, and expanded the Related Work section to cover these methods and CL-LoRA. We have also included additional experimental comparisons and revised our discussion to acknowledge that CL-LoRA achieves higher accuracy than all RSK variants on ImageNet-R.

Question 6 (Insufficient ablation studies and theoretical analysis):

(1) Regarding the missing SKD-only ablation, we have conducted an additional experiment using SKD alone to further evaluate its individual contribution. The results are reported in Table 3 of Section 4.3.

(2) Regarding the selection of SKD regularization weights, we have added the selection strategy for the three regularization coefficients, together with their joint sensitivity analysis. The default value for all three coefficients is 1×10−51\times10^{-5}. The results are reported in Table 12 of Appendix A.8.

(3) Regarding the lack of quantitative Forgetting and Stability results, we have added the corresponding numerical results in Table 5 of Appendix A.3 and Table 12 of Appendix A.8.

(4) Regarding the unclear connection between the theoretical bound in Theorem 1 and continual learning performance, we have added further analyses to examine the relationship between the theoretical bound, actual parameter drift, and continual learning performance. The results are reported in Tables 5 and 6 of Appendix A.3.

We sincerely thank the reviewer again for the constructive comments and suggestions, which have helped us improve the motivation, experimental validation, and theoretical analysis of our work. We look forward to further discussion with you.

Thank you for your reply, but I haven't seen the results of the comparative experiment you added? There is no way to prove that all your improvements are effective, and there are many mistakes in this paper that should not have occurred.

I cannot raise the score.

We thank you for your further comments. We understand your concerns regarding the newly added experimental results, the effectiveness of the proposed components, and the overall rigor of the manuscript. We provide further clarifications on these issues below.

(1)Regarding the newly added comparative experimental results:

In the revised manuscript (All revisions are highlighted in red), we have included three recent representative methods, JANUS-LoRA (Chen et al., 2026), SA-LoRA (Bing et al., 2026), and LoDA (He et al., 2026), in our comparisons across CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A. The corresponding results have been added to Table 1 in Section 4.2. In addition, Tables 13–15 in Appendix A.10 report the results obtained with three random seeds, providing further insight into the performance across different random seeds. The specific results are presented below:

MethodsCIFAR-100 Ā₁₀ (↑)CIFAR-100 Avg. (↑)DomainNet Ā₅ (↑)DomainNet Avg. (↑)ImageNet-R Ā₁₀ (↑)ImageNet-R Avg. (↑)ImageNet-A Ā₁₀ (↑)ImageNet-A Avg. (↑)
JANUS-LoRA87.93±0.3492.15±0.1672.25±0.1878.73±0.0770.91±0.3875.75±0.2258.67±0.9368.35±0.98
SA-LoRA88.08±0.1792.26±0.2171.49±0.1178.04±0.0875.04±0.3578.13±0.2057.98±0.9069.09±1.06
LoDA88.35±0.6991.92±0.4674.03±0.0779.89±0.0378.14±0.2281.75±0.3053.91±1.4968.58±1.00

References

[1] Chen, C., Zeng, P., Guo, Y., Gao, L., Shen, H. T., and Song, J. JANUS-LoRA: A Balanced Low-Rank Adaptation for Continual Learning. In Proceedings of the Forty-Third International Conference on Machine Learning (ICML), 2026.

[2] Bing, X., Xi, Y., and Yahyapour, R. SA-LoRA: Shared-A Decoupled Low-Rank Adaptation for Class-Incremental Learning. Journal of King Saud University Computer and Information Sciences, 38(6):511, 2026.

[3] He, L., Cheng, D., Wang, H., Yang, X., Wang, N., and Gao, X. Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-Based Continual Learning. In Proceedings of the Forty-Third International Conference on Machine Learning (ICML), 2026.

(2)Regarding the effectiveness of the proposed components:

To test the effects of individual components and their combination across different baseline methods, we have added SKD-only ablation experiments for multiple baselines in Table 3 of Section 4.3. We further compare the performance changes after introducing RKC-EMA, as well as the results of the complete RSK framework. Meanwhile, Tables 13–15 in Appendix A.10 report the results across three random seeds, allowing further examination of the performance of different configurations under different random seeds. The specific results are presented below:

MethodsCIFAR-100 Ā₁₀ (↑)CIFAR-100 Avg. (↑)DomainNet Ā₅ (↑)DomainNet Avg. (↑)ImageNet-R Ā₁₀ (↑)ImageNet-R Avg. (↑)ImageNet-A Ā₁₀ (↑)ImageNet-A Avg. (↑)
SD-LoRA87.99±0.0392.13±0.0871.86±0.0878.34±0.1570.38±0.5375.61±0.2357.15±0.5669.19±0.47
+ SKD88.96±0.0492.76±0.0973.39±0.1779.46±0.0870.09±0.5176.26±0.2058.00±1.3469.13±0.41
+ RKC-EMA89.07±0.0692.91±0.0573.34±0.1679.63±0.0672.21±0.1477.61±0.0957.44±0.0469.58±0.34
+ RSK89.12±0.0492.89±0.0773.44±0.0879.70±0.0373.15±0.4377.56±0.0458.55±1.1769.98±0.65
SplitLoRA88.01±0.0392.33±0.1271.04±0.5077.53±0.5172.10±0.0475.76±0.1758.31±1.0265.37±1.66
+ SKD88.56±0.3692.13±0.0472.44±0.0578.80±0.1076.64±0.4380.32±0.3060.49±0.9070.41±0.65
+ RKC-EMA88.64±0.0892.16±0.1072.13±0.1478.75±0.0776.91±0.0780.36±0.0660.82±0.1470.62±0.39
+ RSK88.51±0.2592.21±0.1572.69±0.1479.03±0.0377.43±0.0980.70±0.0961.04±0.3271.03±0.32
EWC-LoRA88.03±0.0892.15±0.1273.30±0.1779.48±0.1472.38±0.5177.47±0.2358.77±0.7269.44±0.19
+ SKD89.57±0.2493.13±0.1473.20±0.0379.44±0.0470.33±1.4174.68±1.1659.86±0.8170.17±0.43
+ RKC-EMA89.59±0.1693.06±0.0773.54±0.0679.87±0.0470.78±0.0377.46±0.5159.68±0.1470.09±0.56
+ RSK89.50±0.1093.04±0.1173.98±0.0580.05±0.0870.62±0.1177.51±0.4760.13±0.2970.17±0.51
Vanilla LoRA85.00±0.0390.29±0.5768.61±0.1676.51±0.1175.30±0.5679.66±0.1058.51±0.9068.92±0.60
+ SKD88.22±0.6191.92±0.9472.55±0.1278.86±0.0376.70±0.3780.27±0.4059.78±0.5269.40±0.58
+ RKC-EMA87.68±0.2391.55±0.9072.18±0.0378.81±0.0376.83±0.0580.33±0.1959.47±0.4169.73±0.35
+ RSK87.94±0.3291.81±0.8373.29±0.0579.45±0.0876.95±0.0780.55±0.2060.00±0.5269.75±0.36

As can be seen from the results, the effects of SKD and RKC-EMA vary across different baseline methods and datasets. Performance improvements are observed in some settings after introducing the corresponding components, while the complete RSK framework also exhibits advantages on several datasets and evaluation metrics. These comparisons provide further experimental evidence for examining the individual contributions of the components and their combined effects.

(3)Regarding errors in the manuscript and overall rigor:

We have further reviewed the manuscript for typographical, labeling, and wording issues, with particular attention to the main text, figures, tables, equations, and references. We have made corresponding corrections to the identified issues to further improve the accuracy and consistency of the revised manuscript. We sincerely apologize for these oversights in the original submission.

We hope that the above additions and revisions address your main concerns and provide useful information for your further assessment of the manuscript. Thank you again for your valuable comments.

Reviewer_6fuAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that existing parameter-efficient continual learning (PECL) methods are designed primarily from an interference-control perspective and do not explicitly address how retained knowledge should be consolidated over time, nor how shared and task-specific adaptations should be structurally organized. A fixed EMA coefficient does not correspond to the same cumulative retention across tasks with different optimization horizons, and architectural separation alone does not necessarily produce differentiated structural rules.

Critical strengths
  1. A worthwhile reframing toward explicit retention and structural organization. Moving from interference suppression to explicit retention plus structural organization is a useful perspective for PECL. RKC-EMA parameterizes consolidation by a task-level retention target rather than a fixed decay coefficient, and Theorem 1 maps prescribed R to realized cumulative retention and the consolidated-state drift bound, which is more principled than ad-hoc fixed EMA coefficients and gives a direct handle on the stability–plasticity trade-off at the trajectory level. Importantly, the consolidation component can be plugged into existing interference-control backbones, and Table 3's ablation shows RKC-EMA and SKD contribute complementary gains across SD-LoRA, SplitLoRA, and EWC-LoRA, supporting the claim that RSK is compatible rather than single-mechanism dependent.
  2. Reasonably broad experimental surface and multi-backbone compatibility evidence. Experiments cover four continual-learning benchmarks and roughly ten PECL and replay-free baselines, with final and average accuracy plus stability and forgetting metrics, and sensitivity analyses over R and direct EMA decay. The multi-backbone combination results (EWC-LoRA+RSK, SD-LoRA+RSK, SplitLoRA+RSK) give evidence that the added retention and structure components are not tied to one particular LoRA architecture.
Critical weaknesses
  1. Moderate novelty increment; theoretical evidence stays at the level of standard linear algebra. RKC-EMA is mechanically an analytic rescaling of the EMA coefficient against task step count; its relationship to existing horizon-aware exponential moving averages and time-weighted consolidation methods needs a clearer literature positioning and a sharper statement of what is not already captured by those schemes. SKD's three regularizers—shared Frobenius compactness, historical L2 fidelity, and task-specific L1 sparsity—are standard regularizers configured asymmetrically rather than a new structural mechanism, and the derivations in Appendix A.2.2 (rank inequalities, first-order stationarity, contraction) are standard linear-algebra exercises. The paper does not provide observable evidence that the shared and specific LoRA factors actually form a structurally distinct decomposition in representation space, which would strengthen the structural claim beyond the loss formulation.
  2. Gain boundaries and metric statements need stricter presentation. On ImageNet-R, Table 1 reports RSK at 71.71%, above InLoRA's 70.38% but below BiLoRA's 74.28%; the introduction nevertheless states RSK "surpassing InLoRA (70.38%) and BiLoRA (74.28%)", which appears inconsistent with the table. Table 2 shows that at 20 tasks, SD-LoRA+RSK drops from 53.60 to 52.33, a negative combination gain that the current discussion does not explain as a boundary condition. The retention target R is fixed at 0.5 across datasets, while Figure 3b shows the optimal direct EMA decay varies substantially across datasets (0.999 on CIFAR-100 versus 0.993 on ImageNet-R); evidence that a single R is dataset-independent would need to be stronger before the horizon-invariance claim generalizes.
Questions for authors
  1. Table 1 shows BiLoRA at 74.28% on ImageNet-R, above RSK's 71.71%. Is the sentence stating RSK "surpassing InLoRA (70.38%) and BiLoRA (74.28%)" a typo? Should RSK be positioned as a complementary plug-in to interference-control methods rather than a replacement for BiLoRA-style shared-subspace methods?
  2. How was the cumulative retention target R = 0.5 chosen? Can the drift bound from Theorem 1, (1 - R)\delta_t, be measured on the actual LoRA factors and matched against the reported forgetting metric across benchmarks, so the theoretical bound can be verified empirically?
  3. How are \lambda_{\mathrm{cmp}}, \lambda_{\mathrm{fid}}, and \lambda_{\mathrm{sp}} tuned? Is there a risk that the shared-path compactness term indirectly constrains the expressive space available to task-specific innovations, partially offsetting SKD's intended structural split? Could the authors report a joint sensitivity sweep over the three coefficients?
2 comments

We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We have carefully considered the comments and provide our point-by-point responses below (all revisions in the manuscript are highlighted in red).

Weakness 1 (Novelty and theoretical contribution of RKC-EMA and SKD):

(1) Regarding the relationship between RKC-EMA and existing horizon-aware EMA and time-weighted consolidation methods: We have expanded the discussions in the Introduction and Related Work to clarify the differences between RKC-EMA and existing methods, particularly EMA-LoRA and CaRE, and to further explain its motivation for controlling cumulative knowledge retention.

(2) Regarding the lack of empirical evidence for the structural distinction between Shared and Specific pathways: We have added Linear CKA analysis in Appendix A.6. The results show high cross-task similarity for Shared representations (0.903/0.904), lower similarity for Specific representations (0.085/0.078), and low similarity between their updates (0.0327–0.1617 for the K branch), providing representation-level evidence for the proposed structural decomposition.

Weakness 2 (Gain boundaries and metric statements need stricter presentation):

(1) Regarding the negative combination gain of SD-LoRA+RSK under the 20-task setting: We have added a discussion to clarify this result and its implications for the applicability of RSK.

(2) Regarding the dataset-independence of the cumulative retention ratio: Unlike standard EMA with a fixed decay coefficient, RKC-EMA adjusts the decay coefficient according to the number of optimization steps in each task via αt=R1/Kt\alpha_t=R^{1/K_t}, thereby maintaining a unified cumulative retention ratio. We have further added cross-dataset sensitivity analyses of RR in Tables 8–10 of Appendix A.7, providing empirical support for adopting a unified R=0.5R=0.5 without dataset-specific tuning.

Question 1 (Comparison with BiLoRA and the positioning of RSK):

(1) Regarding the incorrect statement that RSK outperforms BiLoRA on ImageNet-R: This was a typographical error in the original manuscript. We have corrected the relevant values and comparative descriptions in Table 1 and the Introduction.

(2) Regarding whether RSK should complement existing interference-control methods: BiLoRA focuses on task-specific parameter-space isolation, whereas RSK focuses on shared knowledge management. Therefore, the two approaches are potentially complementary, and their integration is left for future work.

Question 2 (Selection of the cumulative retention target and empirical verification of the drift bound):

(1) Regarding the selection of the cumulative retention target R=0.5R=0.5: We conducted sensitivity experiments with different retention targets (see Fig. 3(a) in Section 4.2, Table 6 in Appendix A.3, and Table 8 and Fig. 6 in Appendix A.7). Considering multiple evaluation metrics, R=0.5R=0.5 provides a favorable balance in final accuracy, stability, forgetting, average accuracy, and plasticity. We therefore adopt it as the default setting.

(2) Regarding the empirical validation of the theoretical drift bound: We additionally measured the actual Shared LoRA parameter drift DtD_t and the maximum online parameter deviation δt\delta_t under different values of RR, with results reported in Table 6 of Appendix A.3. All nine tasks employing RKC-EMA satisfy the theoretical bound Dt≤(1−R)δtD_t\leq(1-R)\delta_t. Furthermore, our analysis shows that reduced parameter drift does not necessarily translate into reduced forgetting, indicating that Theorem 1 characterizes the control of Shared LoRA parameter drift rather than directly bounding forgetting. These results have been added to the revised manuscript.

Question 3 (SKD hyperparameter selection and potential interactions between the Shared and Specific pathways):

(1) Regarding the tuning strategy and joint sensitivity of the three coefficients: We first conducted a joint sensitivity analysis by varying all three coefficients simultaneously, followed by individual sensitivity analyses in which one coefficient was varied while the other two were fixed at 10−510^{-5}. The results are reported in Table 12 of Appendix A.8.

(2) Regarding the potential restriction of task-specific innovations by the compactness constraint: The results in Table 12 show that Plasticity varies non-monotonically with λcmp\lambda_{\mathrm{cmp}}, suggesting that the effect of the compactness constraint on task-specific plasticity is not uniformly suppressive across the tested settings.

We sincerely thank the reviewer again for the valuable comments and suggestions, which have helped us improve the clarity, theoretical justification, and empirical validation of our work. We look forward to further discussion with you.

We would like to express our sincere gratitude once again for the reviewer’s time, careful consideration, and valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.