- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the limitation of parameter-efficient continual learning (PECL), this paper proposes the RSK framework, comprising two components:
- RKC-EMA: Rather than directly setting a local EMA coefficient , RKC-EMA takes a task-level cumulative retention target and derives the local coefficient as from the task optimization horizon . It is claimed to achieve horizon-invariant cumulative retention, supported by Theorem 1.
- SKD: SKD imposes an asymmetric structural prior on the shared and task-specific LoRA paths. The shared branch uses a Frobenius-norm compactness term plus a fidelity term to the consolidated state ; the task-specific branch applies sparsity only to , leaving unconstrained.
Experiments on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A apply RSK as a plug-in on top of multiple LoRA CL methods. Results report final/average accuracy, forgetting, and stability, together with ablations, sensitivity analyses on and EMA decay, and computational overhead comparisons.
Critical strengths
-
Sound problem framing and a general framework: treating “retention” and “structural organization” as dimensions distinct from interference control and designing RSK as a plug-and-play mechanism yields consistent positive gains over baselines on four benchmarks and across four backbone architectures.
-
A comprehensive analysis of overhead and robustness. Fig. 3(c) shows that training time and memory are comparable to those of Vanilla LoRA. Tab. 2 reports performance under varying task lengths, and Tab. 3 provides ablation results across four benchmarks.
Critical weaknesses
-
The core mechanism is essentially inactive in several experimental settings, so the claimed horizon-invariance advantage lacks effective evidence. On balanced benchmarks such as CIFAR-100, since (batch * epoch), all tasks share the same , so is constant within a run and RKC-EMA is mathematically equivalent to standard EMA with a fixed . Thus the distinctive design feature may never be activated.
-
The sensitivity comparison (Fig. 3a vs. 3b, Fig. 5) is confounded because the sweep 0.4, 0.9 and the decay sweep 0.9, 0.99999 cover different effective-retention ranges, the latter's low end () maps to , below the swept range. As and are a monotone bijection for fixed , the claimed advantage of is an artifact of the mismatched intervals, and the sweeps must be compared over the same effective-retention range.
Questions for authors
-
The consolidated-drift bound is circular and does not support its stated implication. Theorem 1 bounds the consolidated drift only conditionally: assuming that every online shared state stays within distance of the pre-task consolidated state. The premise thus already assumes the bounded deviation that the bound is meant to establish, so the result is close to being a restatement of its own hypothesis. Moreover, is neither measured nor explicitly defined, and it is likely to grow with , since retaining more of the pre-task state may require the online states to move farther to fit new tasks. Consequently, the claimed monotonicity that a larger yields a tighter bound is not established.
-
The most relevant baselines are missing. EMA-LoRA (Gao et al., 2024) and Lou et al. (2026) are prior EMA-based LoRA-merging methods that the paper itself cites as motivation. RKC-EMA is essentially a reparameterization of EMA, yet it is not compared with existing EMA methods.
-
RSK introduces an additional shared branch on top of all base methods, whereas Algorithm 3 explicitly retains and returns all task-specific factors. Yet Section 4.2 claims: “...despite requiring neither historical task-specific parameters nor accumulated subspaces or Fisher statistics,” and the Conclusion reiterates: “...without explicitly maintaining historical task-specific parameters.”
4 comments
Thank you for the detailed response. The revised definition of δₜ and the qualification of the monotonicity claim address my concerns about the theorem’s formal statement. However, my main experimental concerns remain unresolved.
- The benefit of horizon adaptation is still unclear. Different optimal decay values across datasets do not show that adapting the decay across tasks explains the gains. Tab.9 reports average coefficients, while Tab.2 evaluates the full framework, neither isolates the benefit of horizon adaptation.
- The sensitivity comparison is still confounded. Tabs. 8 and 10 retain the unmatched retention ranges. Tab. 9 documents the mapping but does not correct the comparison. Normalizing accuracy does not fix this either, so the claim that RKC-EMA is less sensitive remains unsupported.
- The relevant EMA-based baselines are still missing. Comparing against standard EMA within the proposed framework does not replace comparisons with EMA-LoRA (Gao et al., 2024) and Lou et al. (2026) . Differences in their architectures do not resolve the question of how RSK compares with these prior methods.
- The storage claim remains inconsistent. The conclusion now refers specifically to the shared branch, but Section 4.2 (line 430) still makes the broader claim about RSK, whereas Algorithm 3 retains and returns all task-specific factors.
As these issues do not alter the main grounds for my original assessment, I regret that I must maintain the original score.
We sincerely appreciate the reviewer’s valuable time and thoughtful effort in reviewing our manuscript and providing constructive feedback. Our revisions are detailed below.
- To clarify the benefit of horizon adaptation, we have conducted an additional controlled ablation study to directly compare RKC-EMA with fixed EMA. Specifically, RKC-EMA dynamically adjusts the EMA decay factor according to the optimization steps of each task, using a predefined global retention target (R). In contrast, fixed EMA applies a uniform decay factor across all tasks, calculated as the arithmetic mean of the task-wise decay factors corresponding to the same retention target (R). This design enables a comparison under a relatively consistent range of EMA decay factors while isolating the effect of task-level horizon adaptation. The corresponding results have been added to Tables 11 and 12 in Appendix A.7, as detailed below.
(1). Results with RKC-EMA (Horizon Adaptation)
| R | Ā₁₀ (↑) | Avg. (↑) | Forgetting (↓) | Plasticity (↑) | Stability (↑) | P+S |
|---|---|---|---|---|---|---|
| 0.40 | 60.47 | 70.93 | 6.32 | 57.10 | 51.28 | 108.38 |
| 0.45 | 59.67 | 70.38 | 6.38 | 56.50 | 50.09 | 106.59 |
| 0.50 | 60.87 | 70.81 | 5.98 | 57.64 | 52.00 | 109.64 |
| 0.55 | 60.00 | 70.45 | 7.76 | 57.98 | 50.36 | 108.34 |
| 0.60 | 59.87 | 70.05 | 6.18 | 56.20 | 50.07 | 106.27 |
| 0.65 | 60.00 | 70.29 | 6.21 | 56.52 | 50.40 | 106.92 |
| 0.70 | 60.40 | 70.85 | 6.42 | 57.02 | 50.63 | 107.65 |
| 0.75 | 58.93 | 69.66 | 6.46 | 55.47 | 49.22 | 104.69 |
| 0.80 | 59.87 | 69.75 | 5.09 | 54.97 | 49.81 | 104.78 |
| 0.85 | 59.47 | 69.53 | 5.61 | 55.22 | 49.45 | 104.67 |
| 0.90 | 58.93 | 69.38 | 6.08 | 55.30 | 49.41 | 104.71 |
(2). Results with Fixed EMA (Without Horizon Adaptation)**
| R → α | Ā₁₀ (↑) | Avg. (↑) | Forgetting (↓) | Plasticity (↑) | Stability (↑) | P+S |
|---|---|---|---|---|---|---|
| 0.40 → 0.96991 | 59.67 | 70.07 | 6.41 | 56.33 | 50.11 | 106.44 |
| 0.45 → 0.97373 | 59.07 | 69.84 | 6.10 | 55.44 | 49.77 | 105.21 |
| 0.50 → 0.97715 | 60.73 | 70.92 | 6.77 | 57.80 | 51.15 | 108.95 |
| 0.55 → 0.98026 | 60.73 | 71.14 | 6.41 | 57.43 | 51.11 | 108.54 |
| 0.60 → 0.98311 | 60.60 | 71.07 | 5.36 | 56.83 | 51.71 | 108.54 |
| 0.65 → 0.98574 | 59.73 | 70.56 | 6.29 | 55.65 | 49.93 | 105.58 |
| 0.70 → 0.98818 | 60.33 | 70.72 | 6.92 | 57.66 | 50.62 | 108.28 |
| 0.75 → 0.99045 | 59.60 | 70.39 | 6.59 | 56.16 | 49.97 | 106.13 |
| 0.80 → 0.99258 | 60.47 | 70.35 | 5.37 | 56.41 | 51.01 | 107.42 |
| 0.85 → 0.99459 | 59.40 | 69.87 | 5.70 | 55.27 | 49.87 | 105.14 |
| 0.90 → 0.99649 | 59.20 | 69.47 | 5.44 | 54.48 | 49.02 | 103.50 |
-
To address the mismatch in parameter ranges in the sensitivity comparison, we have revised Appendix A.7 by removing the direct sensitivity comparison between RKC-EMA and standard EMA, as well as the normalized fluctuation analysis. Instead, we independently discuss the performance variations of the two methods across their respective parameter ranges. We have also removed the claim that RKC-EMA is less sensitive to hyperparameter variations than standard EMA, retaining only the empirical observation that RKC-EMA exhibits relatively stable performance across the evaluated range of retention ratios.
-
To address the missing EMA-based baselines, we have added comparisons with EMA-LoRA (Gao et al., 2024) and CaRE (Lou et al., 2026). To ensure a fair comparison, we made the following adaptations: for EMA-LoRA, we adopted its corresponding SFA framework, replaced the original CLIP backbone with the same ViT backbone used in RSK, and used the same classification head as RSK and CaRE. Furthermore, considering the architectural differences between these methods and RSK, we replaced their standard EMA with RKC-EMA while keeping all other components unchanged, in order to investigate the applicability of RKC-EMA across different architectures. The experimental results are presented in the following table. Incorporating RKC-EMA improves several evaluation metrics for both SFA and CaRE, further demonstrating its applicability across different PECL architectures.
| Method | ImageNet-A Ā₁₀ (↑) | ImageNet-A Avg. (↑) | ImageNet-R Ā₁₀ (↑) | ImageNet-R Avg. (↑) |
|---|---|---|---|---|
| RSK (only RKC-EMA) | 59.00 | 69.32 | 76.87 | 80.12 |
| RSK | 59.40 | 69.34 | 77.02 | 80.32 |
| SFA (standard EMA) | 54.07 | 66.98 | 71.02 | 78.30 |
| SFA (with RKC-EMA) | 55.13 | 66.89 | 71.15 | 78.39 |
| CaRE (standard EMA) | 61.49 | 72.48 | 79.98 | 83.69 |
| CaRE (with RKC-EMA) | 62.28 | 73.06 | 80.18 | 83.59 |
- The storage claim remains inconsistent. The conclusion now refers specifically to the shared branch, but Section 4.2 (line 430) still makes the broader claim about RSK, whereas Algorithm 3 retains and returns all task-specific factors. Regarding the inconsistency in the storage-efficiency claims, we have revised the relevant statements in the Conclusion and Section 4.2 to clarify that the storage-efficiency claim applies specifically when RSK is used independently, without being integrated with other PECL methods. Furthermore, we have revised Section 3.4 and Algorithm 3 to explicitly clarify that when RSK is combined with other PECL methods, both shared and task-specific parameters are retained and returned, ensuring consistency between Algorithm 3 and the description in Section 3.4.
We sincerely thank the reviewer again for the constructive feedback, which has helped us improve the rigor and clarity of our manuscript.
We sincerely thank the reviewer for the thorough evaluation and constructive feedback. We have carefully considered all the concerns and questions and provide our point-by-point responses below.
Critical Weakness 1 (Limited activation of the horizon-aware mechanism):
Regarding the limited activation of the horizon-aware mechanism: For balanced datasets such as CIFAR-100, all tasks have the same number of training samples and epochs, resulting in a constant optimization horizon Kt. Consequently, αt=R1/Kt also remains constant across tasks, making RKC-EMA mathematically equivalent to standard EMA with a fixed decay coefficient. In contrast, \textcolor{red}{datasets such as ImageNet-R, ImageNet-A, and DomainNet exhibit different task-size distributions, which may lead to different optimization horizons across datasets and potentially across tasks.} RKC-EMA \textcolor{red}{accommodates such variations by determining} αt according to the actual optimization horizon Kt, \textcolor{red}{thereby} maintaining a consistent cumulative retention target R. Furthermore, Fig. 3(a) and Tables 8--10 in Appendix A.7 report the sensitivity of RKC-EMA to R across different datasets, while Fig. 3(b) shows that the optimal fixed EMA decay coefficient varies across datasets (0.999 for CIFAR-100 and 0.993 for ImageNet-R). These results highlight the dataset dependency of fixed decay coefficients and support parameterizing EMA decay through a unified cumulative retention target R.
Critical Weakness 2 (Unmatched effective-retention ranges in sensitivity analyses):
Regarding the mismatched ranges in the sensitivity analyses: For a given optimization horizon Kt, the cumulative retention target R and the EMA decay coefficient αt satisfy αt=R1/Kt, establishing a one-to-one correspondence between them. However, the original experiments used different sweep ranges for R (0.4--0.9) and α (0.9--0.99999), which do not cover the same effective-retention range. Since the optimization horizons vary across datasets, we provide the corresponding values of R and α for all four datasets in Table 9 of Appendix A.7.
Question 1 (Conditional nature and interpretation of the consolidated-drift bound):
(1) Regarding the circularity of the theoretical argument: We revised Theorem 1 and its proof by defining δt as the maximum deviation of the online shared parameters from the initial consolidated state along the finite optimization trajectory of the current task. By unrolling the EMA recurrence and applying the triangle inequality, we directly derive Dt≤(1−R)δt, without requiring any additional a priori boundedness assumption.
(2) Regarding the empirical measurement of δt: We added per-task drift measurements under different values of R on ImageNet-A (Appendix A.3, Table 5). All nine tasks employing RKC-EMA satisfy the theoretical bound. For example, on Task 1, as R increases from 0.01 to 0.99, Dt decreases from 3.15 to 0.03, while δt remains approximately 3.27, demonstrating that increasing R effectively reduces the actual EMA drift.
(3) Regarding the monotonicity claim: We clarify that the upper bound decreases with increasing R only when the optimization trajectory is fixed (i.e., δt remains unchanged). Across different training trajectories, neither the actual drift nor its theoretical upper bound is guaranteed to decrease monotonically.
Question 2 (Missing comparisons with existing EMA-based LoRA methods):
Existing EMA-based LoRA methods all employ standard EMA updates, but differ in their overall method designs, making it difficult to isolate the effect of the EMA strategy through direct method-level comparisons. Therefore, we have added multidimensional comparisons with standard EMA in Fig. 3(a)(b), Fig. 6, and Tables 8 and 10 in the Appendix, analyzing parameter sensitivity and update stability.
Question 3 (Inconsistent statements regarding historical task-specific parameter storage):
The original wording was imprecise and may have caused confusion. Our intention was to emphasize that the shared branch introduced by RSK does not itself require explicitly maintaining historical task-specific parameters, accumulated subspaces, or Fisher statistics. The task-specific factors retained and returned in Algorithm 3 are solely intended to preserve the original parameter structure of the integrated baseline methods, rather than indicating that the RSK shared branch depends on these historical parameters. We have clarified this distinction in the revised manuscript to avoid ambiguity.
We sincerely thank the reviewer again for the constructive comments and suggestions, which have helped us improve the experimental evaluation, theoretical analysis, and presentation of our work. We look forward to further discussion.