acceptodds
Under review as a conference paper at ICLR 2027

ECR-LoRA: Evolving Core LoRA and Complementary Residual LoRAs for Class-Incremental Learning

Abstract

Rehearsal-free parameter-efficient continual learning (PECL) enables frozen pretrained Vision Transformers (ViTs) to adapt to sequentially arriving tasks using lightweight adaptation modules, such as LoRA, without historical exemplars. Existing LoRA-based PECL methods face a fundamental trade-off between knowledge sharing and task specialization: sharing or merging adapters promotes reuse but blurs task-specific representations, whereas isolating updates mitigates forgetting at the cost of reusable knowledge transfer. We propose ECR-LoRA, a split-depth LoRA framework that organizes continual adaptation into an evolving Core LoRA and complementary task-specific Residual LoRAs. Early blocks maintain the fixed-rank Core, which evolves across tasks to accumulate transferable knowledge, while later blocks preserve Residual LoRAs for evidence not sufficiently captured by the Core. Core evolution is controlled by subspace-compatible learning and Fisher-guided consolidation, while core-error reweighting and residual-mix consistency encourage complementary residual learning and robustness to cross-residual variation. During inference, uncertainty from the consolidated core provides an entropy-guided prior for calibrating residual experts without task identities, additional training, or exemplar replay. Experiments on four class-incremental learning benchmarks show that ECR-LoRA achieves the best final accuracy on all four and the best average accuracy on three, with a favorable stability–plasticity trade-off. Code is available in the supplementary material.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_DTFqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ECR-LoRA, a class-incremental learning framework that partitions adaptation across transformer depth into an evolving shared core and task-specific residual LoRAs. The core uses subspace-compatible learning and Fisher-guided consolidation, while residual learning uses core-error reweighting (CER) and residual-mix consistency (RMC). At inference, core-only entropy calibrates competition among residual experts.

Critical strengths

The shared/specific organization is clearly motivated, and the paper acknowledges its relationship to CL-LoRA. The All-scan variant already improves over CL-LoRA on several datasets, indicating that the reported benefits are not entirely due to the final inference calibration.

Critical weaknesses
  1. The exact candidate initialization and Fisher normalization are unclear. If the candidate reuses the previous down-projection’s row space, the merged matrix already has rank at most r, leaving the role of SVD truncation unclear.
  2. CER updates the full Core–Residual path, and RMC is not compared with generic feature perturbations, leaving residual-specific complementarity insufficiently established.
  3. Table 2 and Figure A1 disagree for removing CER on ImageNet-R and RMC on ImageNet-A.
  4. Full residual scanning limits scalability, while the 50-session ImageNet-R gain over CL-LoRA is only 0.02 percentage points. Matched accuracy–latency comparisons are needed.
  5. The clarity and format of many figs.
Questions for authors
  1. How are the candidate down-projection and normalized Fisher support constructed, and does SVD discard nonzero singular directions in practice?
  2. Can the Table 2/Figure A1 discrepancies be reconciled?
  3. How much improvement remains over CL-LoRA when both methods use the same inference calibration?
9 comments
AuthorsOpenReview ↗

Response to Reviewer DTFq (1/3): W1 & Q1, W2

W1 & Q1 — Candidate Initialization, Fisher Support, and SVD.

Thank you for highlighting these ambiguities. Each shared Core is initialized with a row-orthonormal A0cA_0^c obtained via QR decomposition and B0c=0B_0^c=0. At task tt, the candidate inherits At−1cA_{t-1}^c, keeps it frozen, and optimizes only the zero-initialized BtnewB_t^{\mathrm{new}}.

We estimate Fisher-inspired support from squared cross-entropy gradients over four current-task mini-batches, averaging across batches and output rows and normalizing by the maximum rank-wise entry. Its mean and the historical EMA determine the absorption coefficient ηt\eta_t.

Since the candidate and the existing Core share the down-projection, their merged matrix already has rank at most rr. SVD therefore preserves the merged mapping exactly in exact arithmetic while providing balanced factors for the next task. Direct factor accumulation would preserve the same mapping but would not provide this reparameterization.

We have clarified these details in Section 3.3, Algorithm A1, and Appendix A.2.


W2: CER/RMC Complementarity and Appropriate Controls.

Thank you for this careful reading. We agree that CER derives detached example weights from shared-only (Core) predictions, while the weighted loss is optimized through the full Core–Residual path (Eq. 6). Thus, CER is Core-conditioned rather than Residual-restricted: it emphasizes examples insufficiently explained by the Core, while Residual specificity comes from the split-depth architecture rather than gradient isolation.

Following your suggestion, we conducted matched controls on 10-session ImageNet-A (NVIDIA L20, three seeds).

Table R1. CER/RMC controls on ImageNet-A. Accuracy (%) is reported as mean ± SD over three seeds. Old-task accuracy is measured after the final session; new-task accuracy is averaged across sessions. Parentheses report seed-paired differences from the full model (mean ± SD) for the four primary comparisons discussed below. Also reported as Table A6 in Appendix D.3.

VariantAll-scan ALastA_{Last}All-scan AAvgA_{Avg}CP ALastA_{Last}CP AAvgA_{Avg}Old-taskNew-task
ECR-LoRA (full)61.33 ± 1.3770.86 ± 0.7563.49 ± 0.4171.81 ± 0.5261.94 ± 1.2370.40 ± 0.49
w/o CER61.22 ± 0.7470.35 ± 0.75 (−0.51 ± 0.04)63.27 ± 0.3471.50 ± 0.3561.79 ± 0.9569.46 ± 0.31 (−0.95 ± 0.45)
Full-path weighting61.31 ± 1.3670.64 ± 0.7363.93 ± 1.4371.77 ± 1.0161.71 ± 1.2870.13 ± 0.12
w/o RMC61.33 ± 1.6370.86 ± 0.8563.27 ± 0.6771.46 ± 0.45 (−0.35 ± 0.09)61.75 ± 1.6370.54 ± 0.35
Norm-matched random shift61.49 ± 1.1470.61 ± 0.8463.29 ± 0.6371.53 ± 0.58 (−0.28 ± 0.11)61.99 ± 1.0570.33 ± 0.46

CER. Removing CER reduces new-task accuracy and All-scan AAvgA_{Avg} in all three paired seeds. On a fixed subset of samples misclassified by the Core-only path, its removal also reduces within-task accuracy by 0.46±0.130.46\pm0.13 pp. Full-path weighting performs comparably, suggesting that CER-style example reweighting benefits new-task learning without establishing the unique superiority of Core-derived weights.

RMC. Removing RMC or replacing historical-Residual shifts with norm-matched random perturbations reduces CP AAvgA_{Avg} by 0.35±0.090.35\pm0.09 and 0.28±0.110.28\pm0.11 pp, respectively, with decreases in all three seeds. This provides modest evidence for the usefulness of structured historical-Residual exposure. It is also qualitatively consistent with the original coefficient sweep (Figure A1), where setting λmix=0\lambda_{mix}=0 lowers ALastA_{Last} on both datasets. However, the changes in final accuracy in the matched L20 reruns remain within seed variation.

We focus the detailed comparison on ImageNet-A, where the Core-only path misclassifies a larger proportion of samples (approximately 48% versus 34% on ImageNet-R). On ImageNet-R, the same controls differ from the full model by less than 0.2 pp across the reported metrics. Complete results for both datasets are provided in Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer DTFq (2/3): W3 & Q2, W4

W3 & Q2: Numerical consistency. We agree that the numerical differences between Table 2 and Figure A1 require clarification. Table 2 reports component ablations from an earlier experimental batch, while Figure A1 presents a separate three-seed coefficient sweep. Since these experiments were conducted independently, corresponding settings yield slightly different results (ImageNet-A w/o RMC: 60.76 ± 1.29 vs. 61.36 ± 1.02; ImageNet-R w/o CER: 80.29 ± 0.41 vs. 79.92 ± 0.10).

We also identified a plotting error: the default-setting points in Figure A1 were taken from the main-table results rather than the original sweep. After checking the sweep logs, we corrected these points in the revised Figure A1. All other points were consistent with the original records, and each curve now uses results from the same experimental batch. We have also updated the corresponding analysis in Appendix C.2 to clarify the experimental sources and reflect the corrected results.

The qualitative conclusions remain unchanged: removing SAR produces the largest accuracy drop, while CER and RMC show smaller variations across nearby nonzero weights.


W4: Scalability and accuracy–efficiency trade-off.

To assess the accuracy–efficiency trade-off, we conducted matched comparisons of ECR-LoRA and CL-LoRA on 10- and 20-session ImageNet-R. Both methods use the same shared-prefix reuse optimization for latency measurements mentioned in Appendix D.1, ensuring a consistent inference setup.

Table R2. Accuracy–latency comparison on 10- and 20-session ImageNet-R.

SessionsMethodALastA_{\mathrm{Last}} (%)Latency (ms/image)
10CL-LoRA (All-scan)79.76 ± 0.2838.5
10ECR-LoRA (All-scan)80.28 ± 0.2943.1
10CL-LoRA + CP80.47 ± 0.1245.4
10ECR-LoRA + CP80.84 ± 0.0848.1
20CL-LoRA (All-scan)76.26 ± 0.2174.5
20ECR-LoRA (All-scan)77.83 ± 0.4980.8
20CL-LoRA + CP76.36 ± 0.7479.9
20ECR-LoRA + CP78.56 ± 0.3191.9

Accuracy is reported as mean ± SD over three seeds. Latency is measured on the same NVIDIA L20 using FP32, batch size 1, and shared-prefix reuse for both methods, reporting the median of three measurement rounds.

Under All-scan inference, ECR-LoRA improves final accuracy over CL-LoRA by 0.52 and 1.57 pp at 10 and 20 sessions, respectively, with approximately 12% and 8% additional latency. When CP is applied to both methods, ECR-LoRA maintains gains of 0.37 and 2.20 pp, with approximately 6% and 15% additional latency. Notably, although the absolute accuracies differ between experimental batches, the 20-session All-scan improvement is similar to that reported in Table 3 (+1.39 pp).

At 50 sessions, the original three-seed results show comparable final accuracy between ECR-LoRA and CL-LoRA (71.73 ± 0.26 vs. 71.71 ± 0.67), with a difference of only 0.02 pp. However, ECR-LoRA achieves substantially higher average accuracy across sessions (78.63 ± 1.63 vs. 75.89 ± 1.68), an improvement of 2.74 pp. This suggests that, despite similar performance at the final session, ECR-LoRA maintains better classification accuracy throughout the incremental learning process. We additionally conducted a matched 50-session rerun with the same configuration on an L20 GPU. In this single-seed experiment, ECR-LoRA outperforms CL-LoRA in both final accuracy (73.70 vs. 70.65, +3.05 pp) and average accuracy (80.81 vs. 76.49, +4.32 pp). These results provide further evidence of ECR-LoRA's effectiveness in long-sequence continual learning, particularly in maintaining higher accuracy across incremental sessions.

Nevertheless, shared-prefix reuse does not eliminate the linear cost of evaluating task-specific Residuals. We explicitly acknowledge this limitation and discuss sparse expert selection as future work in Appendix D.1.

AuthorsOpenReview ↗

Response to Reviewer DTFq (3/3): Q3, W5

Q3 — Matched Core-Prior Calibration on CL-LoRA.

Thank you for suggesting a matched calibration comparison. We reran CL-LoRA and ECR-LoRA using matched task splits and three seeds on NVIDIA L20 GPUs. Both methods use the same Core-prior (CP) calibration rule with βH=4\beta_H=4, without method-specific retuning. CP is a score-calibration rule rather than a separately trained routing module. It can therefore be applied to CL-LoRA using its shared-only pathway as the reference, allowing a matched comparison under the same inference calibration.

Table R3. Matched CP comparison (ALastA_{\mathrm{Last}}, mean ± SD, %; three seeds on NVIDIA L20 GPUs). Full results in Table A9, Appendix D.5.

MethodImageNet-RImageNet-ACIFAR-100
CL-LoRA (All-scan)79.76 ± 0.2858.09 ± 0.9186.15 ± 0.22
CL-LoRA (+CP)80.47 ± 0.1259.96 ± 0.7487.81 ± 0.28
ECR-LoRA (All-scan)80.28 ± 0.2961.33 ± 1.3787.40 ± 0.20
ECR-LoRA (+CP)80.84 ± 0.0863.49 ± 0.4189.25 ± 0.23

CP improves CL-LoRA's final accuracy by 0.71, 1.87, and 1.66 percentage points on ImageNet-R, ImageNet-A, and CIFAR-100, respectively. When both methods use CP, ECR-LoRA retains margins of 0.37, 3.53, and 1.44 points. Thus, its gains are not solely attributable to CP, although the margin on ImageNet-R is modest.

We have clarified this comparison in Sec. 4.4 and added the CL-LoRA implementation, matched protocol, and complete final/average accuracy results to Appendix D.5 (Table A9).


W5: Figures and Presentation.

Thank you for your helpful suggestions and careful attention to these details. We have corrected the plotting error in Figure A1 and revised the corresponding descriptions to improve clarity and consistency. We have also added further explanations and experimental details in the relevant sections and appendix. We sincerely appreciate your feedback and would be happy to provide additional clarification or make further improvements if any points remain unclear.

Reviewer_Th1UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies rehearsal-free class-incremental learning with a frozen pretrained ViT. It uses shared Core LoRAs in earlier blocks and task-specific Residual LoRAs in later blocks. A candidate Core update is trained with output-subspace alignment and then consolidated using branch-level Fisher-like support statistics. Residual training combines shared-path-error-based reweighting (CER) with feature perturbations induced by historical residuals (RMC). At inference, the method scans residual experts and calibrates their predictions using shared-path entropy (Core-prior, CP). The complete method achieves the highest final-accuracy means on all four main settings and the highest average incremental accuracy on three, among the reported baselines. These are encouraging results. My main questions concern how specifically the proposed mechanisms account for the improvements, particularly the choice of shared update directions and the interpretation of the residual-learning objectives.

Critical strengths
  1. The problem is relevant, and the overall design coherently combines shared adaptation, task-specific preservation, and inference calibration.
  2. Evaluation covers four benchmarks, three seeds with reported standard deviations, longer task sequences, depth allocations, LoRA ranks, and additional self-supervised backbones.
Critical weaknesses
  1. The alignment term encourages a candidate increment (\Delta) to remain within the output column space of the previous Core. This is a plausible knowledge-reuse bias. Nevertheless, keeping the output subspace similar does not by itself preserve the mapping within that subspace, so the connection to old-task retention could be clearer. The zero-weight ablation in the appendix usefully shows a benefit over unconstrained updates; a comparison with the opposite directional preference would help explain why this particular choice works. A direct comparison with the mirror penalty would be particularly helpful here, using the same old-subspace basis and otherwise matched settings, including a comparable regularization-selection budget. Reporting old-task retention and current-task accuracy under All-scan could show how the two directional preferences affect the tradeoff. This could clarify whether reuse improves the balance or favors new-task adaptation at some retention cost. Either outcome would help explain the design. The mirror penalty would serve as a diagnostic control; OGD and InfLoRA may constrain different spaces.

  2. A concise report of peak memory and these time costs would be appreciated if obtainable from existing runs or logs. This would complement the analysis already provided and make the practical accuracy-cost tradeoff easier to judge. Table A4 usefully reports adapter storage, inference FLOPs, and latency, and acknowledges the cost of scanning the residual bank. At 20 sessions, CP latency is approximately 254 ms/sample on an A40 in FP32 with batch size 1. Although the Core size is fixed, residual storage and inference computation grow with the number of tasks. The remaining gap is that adapter storage excludes the backbone, classifiers, optimizer states, and temporary training parameters, so it does not describe peak GPU memory. Training time and task-end processing costs are also less clear, particularly for the auxiliary paths, support-statistic estimation, and refactorization.

Questions for authors
  1. How should the output-subspace reuse bias be understood in terms of the retention/adaptation tradeoff? The matched directional comparison above would be particularly helpful.
  2. Could the authors clarify the interpretation of the branch-support statistics, the scope of the quadratic surrogate's optimality, and the candidate-factor initialization?
  3. Does shared-path-guided weighting produce more complementary corrections than full-path hard-example weighting, and do historical perturbation directions provide a benefit beyond norm-matched randomized directions?
  4. How should the respective contributions of training and inference calibration be interpreted from the existing results?
  5. Could peak GPU memory and training/task-end processing costs be reported if these measurements are readily available?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Th1U (1/2): W1 & Q1, Q2, Q3

W1 & Q1: Rationale for subspace reuse.

We agree that output-subspace alignment alone does not guarantee preservation of the historical input–output mapping. We therefore clarify that SAR encourages historical subspace reuse rather than enforcing functional invariance. Following your suggestion, we compare SAR against a mirror penalty that uses the same historical basis Ut−1U_{t-1} but penalizes the in-subspace component ∥Ut−1Ut−1⊤Δtnew∥F2\lVert U_{t-1}U_{t-1}^{\top}\Delta_t^{\mathrm{new}}\rVert_F^2, instead of the out-of-subspace component. The mirror coefficient is selected from {0.2, 0.5, 1.0} in a pilot sweep and fixed at 0.2 for the three-seed comparison; SAR uses the dataset-specific settings reported in the paper.

Table R4. Subspace-direction controls on 10-session ImageNet-A and ImageNet-R (L20, three seeds).

DatasetPenaltyOld-task acc.New-task acc.All-scan ALastA_{Last}CP ALastA_{Last}
ImageNet-ANone60.0670.5859.7361.36
ImageNet-AMirror60.4770.8760.1661.56
ImageNet-ASAR61.9470.4061.3363.49
ImageNet-RNone78.0685.7278.3579.03
ImageNet-RMirror76.6985.5177.1177.69
ImageNet-RSAR80.2085.5380.2880.84

Values are three-seed means. Old-task accuracy is measured on previously learned classes after the final session; new-task accuracy is averaged across sessions.

SAR achieves the highest final old-task accuracy and ALastA_{Last} among the tested settings on both datasets, while new-task accuracy varies by less than 0.5 pp. On ImageNet-R, replacing SAR with mirror reduces old-task accuracy by 3.51±0.463.51\pm0.46 pp and ALastA_{Last} by 3.17±0.433.17\pm0.43 pp (All-scan) and 3.14±0.133.14\pm0.13 pp (CP), with negligible change in new-task accuracy (−0.02±0.18-0.02\pm0.18 pp). ImageNet-A shows the same overall preference for SAR. Removing directional regularization also reduces ALastA_{Last} by 1.6–2.1 pp across the two datasets.

These results support encouraging historical subspace reuse as an effective retention-oriented preference, without claiming exact preservation of past mappings.


Q2: Fisher support, surrogate optimality, and initialization. We estimate Fisher-like support from squared gradients of the trainable Core candidate factor BtnewB_t^{new}, averaged over four current-task mini-batches and output dimensions, then max-normalized. Its branch mean and historical EMA determine the absorption coefficient (Eq. 3). This coefficient minimizes the stated local quadratic surrogate, not the true joint historical/current task losses. For initialization, the first Core uses a row-orthonormal QR factor and zero up-projection; subsequent candidates inherit the preceding Core factor AtnewA_t^{new}, which remains frozen within each task, while BtnewB_t^{new} is trained. Core factors can change between tasks through refactorization. We have clarified these details in Section 3.3 and Appendix A.2.


Q3: CER/RMC Mechanisms and Matched Controls.

Thank you for suggesting these controls. We conducted both experiments on 10-session ImageNet-A using three seeds. Detailed results are presented in Table R1 of our response to Reviewer DTFq (W2, Part 1/3).

(1) Shared-path vs. generic hard-example weighting. Full-path weighting performs comparably to shared-path weighting (new-task 70.13 vs. 70.40; CP AAvgA_{Avg} 71.77 vs. 71.81), so we do not claim that the shared-only signal is uniquely superior. Removing CER reduces new-task accuracy by 0.95 pp and All-scan AAvgA_{Avg} by 0.51±0.040.51\pm0.04 pp, with decreases in all three seeds. These results support CER's contribution to new-task learning.

(2) Historical vs. random perturbations. Historical-Residual perturbations achieve a CP AAvgA_{Avg} of 71.81, compared with 71.53 using norm-matched random directions and 71.46 without RMC. The corresponding seed-paired differences are −0.28±0.11-0.28\pm0.11 and −0.35±0.09-0.35\pm0.09 pp, respectively, with decreases in all three seeds. This provides modest support for RMC's intended role in improving cross-expert comparability under CP.

On ImageNet-R, all matched controls remain within 0.2 pp of the full model. Complete results for both datasets are provided in Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer Th1U (2/2): Q4, W2 & Q5

Q4 — Training versus Inference Calibration.

We agree that differences between the trained methods and gains from inference-time calibration should be distinguished. We therefore evaluated CL-LoRA and ECR-LoRA with and without the same CP rule in matched three-seed NVIDIA L20 reruns. CP operates on the predictions of the same trained Residual experts, adjusting their cross-task scores without changing model parameters or making a hard expert-selection decision. Therefore, the difference between All-scan and CP isolates the contribution of inference-time calibration.

Table R3 (same table as in our response to Reviewer DTFq, Q3). Matched CP comparison (ALastA_{\mathrm{Last}}, mean ± SD, %; three seeds on NVIDIA L20 GPUs). Full results in Table A9, Appendix D.5.

MethodImageNet-RImageNet-ACIFAR-100
CL-LoRA (All-scan)79.76 ± 0.2858.09 ± 0.9186.15 ± 0.22
CL-LoRA (+CP)80.47 ± 0.1259.96 ± 0.7487.81 ± 0.28
ECR-LoRA (All-scan)80.28 ± 0.2961.33 ± 1.3787.40 ± 0.20
ECR-LoRA (+CP)80.84 ± 0.0863.49 ± 0.4189.25 ± 0.23

Under All-scan inference, ECR-LoRA exceeds CL-LoRA by 0.52, 3.24, and 1.25 percentage points on ImageNet-R, ImageNet-A, and CIFAR-100, respectively. CP additionally improves ECR-LoRA by 0.56, 2.16, and 1.85 points, while also improving CL-LoRA by 0.71, 1.87, and 1.66 points. These comparisons separate the inference-calibration effect from method-level performance differences without attributing the latter to any single training component.

We have clarified this distinction in Sec. 4.4 and added the full comparison, including average accuracy and implementation details, to Appendix D.5 (Table A9).


W2 & Q5: GPU memory and computational costs.

We agree that the adapter storage reported in Table A4 does not represent peak GPU memory, and that training-time and task-end costs should be distinguished. We therefore provide additional measurements on 10-session ImageNet-R.

Table R5. GPU memory and training-time comparison on ImageNet-R (10 sessions, L20, FP32). Training uses batch size 32; inference memory is measured with batch size 1.

MetricCL-LoRAECR-LoRAOverhead
Peak training memory (MiB)3,5173,662+4.1%
Task-training time (min)59.461.0+2.7%
Peak inference memory (MiB)389394+1.3%

The task-training time includes auxiliary forward computations for CER/RMC and support-statistic estimation, but excludes post-task Core consolidation and prototype recomputation. The reported time difference therefore reflects training-stage overhead, not total end-to-end overhead.

Task-end cost breakdown (Table R6). We further analyzed the individual operations using L20 logs (ImageNet-R, 10 sessions, three seeds). The following values are averaged over seeds and summed across all ten sessions.

OperationTime (min)Relative to epoch-training time
Training epochs (including CER/RMC)151.0—
Fisher-based support estimation0.660.4%
Core consolidation and SVD refactorization0.520.3%
Additional prototype recomputation12.28.1%

Both support estimation and Core refactorization account for less than 0.5% of epoch-training time. The refactorization timing is also a conservative upper bound because it includes logging of Core snapshots. The main additional task-end cost is prototype recomputation after Core consolidation: since consolidation changes the shared representation, ECR-LoRA updates the prototype classifier a second time, whereas CL-LoRA requires only the initial computation.

Overall, the measurements show modest additional training-stage memory and computation, with prototype recomputation accounting for the main post-task overhead. For inference, we provide matched accuracy–latency comparisons at 10 and 20 sessions in our response to DTFq (W4, Part 2/3; Table R2), using the same shared-prefix optimization for both methods. We also explicitly acknowledge the remaining linear Residual-scanning cost in Appendix D.1.

Dear Reviewer,

Thank you for your time and consideration. We would be grateful if you could let us know whether our rebuttal has adequately addressed your concerns or if there are any remaining questions. If our responses have clarified your concerns, we would sincerely appreciate your consideration of a score adjustment. Thank you again for your valuable feedback!

The author has addressed most of my questions; I will raise my current score and am inclined to accept this paper.

Reviewer_FhwEOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In continual learning, the stability-plasticity dilemma is a well-known challenge, and LoRA-based methods are no exception. As explained in the paper, these methods aim to continuously acquire new knowledge by seeking a trade-off between modifying and reusing previously acquired knowledge (normally in one set of weights) and freezing what has been learned (one LoRA weight set per task). This work presents ECR-LoRA, which divides what has been learned into two components: a shared core that is updated by all tasks, and a task-specific residual LoRA. It proposes several components and loss functions designed to encourage reuse of the shared core while minimising unnecessary modifications. It also uses the core’s uncertainty to calibrate its outputs and improve predictions when selecting which task-specific residual to use. Favourable results are presented across four benchmarks, along with ablation studies of key components.

Critical strengths
  • The paper is well written, clearly identifying the research questions and motivations behind the problem. It successfully identifies the limitations of previous studies and presents a compelling case for the proposed method, which is well aligned with the stated motivations. It also provides an ablation study for the method’s components.
  • The paper presents an interesting method that mitigates forgetting by keeping the "core" largely stable while adding plasticity mostly through the residuals. Furthermore, it uses an entropy-based calibration that removes the need to train a router or perform task-specific LoRA merging.
Critical weaknesses
  • As mentioned in Section D.1 of the appendix, the cost of the proposed method is high, both during training and inference. During training, multiple passes over each batch are required, while during inference, the cost increases linearly with the number of tasks due to the need to compute the output of each task’s LoRA residual. The inference cost is difficult to mitigate because of the use of one classifier per task. This is a critical issue because, in real-world applications, users may be willing to accept longer training times but typically expect fast inference.
    • Given that the classifier is based on cosine similarity, how would performance be affected if the LoRA residuals were merged into a single representation? This would help reduce computational cost, but it would likely come at the cost of performance.
Questions for authors
  • What does “Q/V LoRA” in line 188 mean?
  • Reuse of previous knowledge is encouraged by initialising A from previous tasks, freezing it, and training B to exploit this knowledge. However, it is never explained whether A is trained at any point. Is A ever trained? How can we be sure that A contains useful information?
  • Equation 7 is essentially a form of “data augmentation” that increases the variability of the representation, which can improve the robustness of the model. Is there an ablation on the alpha values?
  • There is an ablation of the proposed components that form the method. Is there also an ablation of the terms in the objective function and their coefficients (Eq. 9)?
  • In Figure 3, what does the “Core prior” mean?
  • In Table 3, ImageNet-R shows little improvement over previous methods. What happens on the other benchmarks?
  • According to Table 1, entropy-guided inference seems critical to achieving state-of-the-art results. Can it be applied to other LoRA-based approaches to verify its compatibility?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FhwE (1/2): W1, Q1 & Q2, Q3 & Q4

W1: Inference efficiency and Residual merging.

We acknowledge that retaining separate Residual experts introduces inference overhead that increases with the number of sessions. We investigated two ways to reduce this cost.

(1) Shared-prefix reuse. The first six blocks are shared across experts and can be computed once without changing predictions. This reduces the theoretical backbone cost from approximately 2T2T to T+1T+1 half-backbone passes. Repeated L20 measurements show a 47–55% reduction in All-scan latency across 10, 20, and 50 sessions, consistent with the expected reduction. We have added the theoretical analysis to Appendix D.1. This optimization also applies to CL-LoRA and does not eliminate the linear cost of evaluating individual Residuals.

(2) Residual merging. Following the suggestion, we averaged the task-specific Residual updates into a single branch, requiring only one backbone forward pass:

Table R7. Residual merging versus All-scan (ALastA_{\mathrm{Last}}, 10 sessions, L20).

Inference (ALastA_{\mathrm{Last}}, 10 sessions)ImageNet-AImageNet-R
All-scan61.33 ± 1.3780.27 ± 0.30
Mean merging57.44 ± 0.3575.89 ± 0.29
Core only51.42 ± 0.3065.85 ± 0.34

Mean merging retains useful Residual information but reduces final accuracy by 3.89 and 4.38 pp relative to All-scan. Thus, merging offers lower inference cost at a substantial accuracy cost.

We retain separate Residuals to preserve their task-specific contributions while using shared-prefix reuse to eliminate redundant computation. Compared with single-backbone merged-adapter methods, the remaining linear scanning cost is an explicit limitation, as discussed in Appendix D.1.


Q1 & Q2: LoRA implementation and Core factorization.

(Q1) Q/V denotes the query and value projections in self-attention. LoRA is applied to both, with shared Core adapters in blocks 1–6 and task-specific Residual adapters in blocks 7–12 (rank 10 by default; 30 for DomainNet).

(Q2) The initial Core factor A0cA_0^c is row-orthonormalized via QR, with B0c=0B_0^c=0. At each task, AtnewA_t^{\mathrm{new}} inherits At−1cA_{t-1}^c and remains frozen; only BtnewB_t^{\mathrm{new}} receives gradients. Task-end SVD may change the numerical factors but preserves the merged mapping without rank truncation.

Importantly, inheriting AA maintains compatible input-side coordinates rather than assuming AA independently stores learned knowledge. Historical information is represented by the consolidated mapping Ct−1=Bt−1cAt−1cC_{t-1}=B_{t-1}^cA_{t-1}^c, whose output subspace guides subsequent adaptation.

These procedures are specified in Sec. 3.3, Algorithm A1, and Appendix A.2.


Q3: Sensitivity to the mixing coefficient. In Eq. (7), α\alpha is sampled independently for each example from Uniform[0.05,0.2]\mathrm{Uniform}[0.05,0.2]. To examine its sensitivity, we tested two alternative intervals on 10-session ImageNet-A using three seeds, with all other settings unchanged:

Table R8. Sensitivity to the α\alpha range on 10-session ImageNet-A (three seeds).

α\alpha rangeAll-scan ALastA_{\mathrm{Last}}CP ALastA_{\mathrm{Last}}
[0.02, 0.1]60.98 ± 0.8563.96 ± 0.45
[0.05, 0.2] (default)61.33 ± 1.3763.49 ± 0.41
[0.1, 0.4]61.04 ± 0.9063.71 ± 0.47

The default range achieves the highest All-scan accuracy, while CP accuracy varies by at most 0.47 percentage points across the tested intervals. Thus, performance is relatively stable within this range, although the default is not uniformly optimal under CP.

Q4: Objective ablations and loss-weight sensitivity. Table 2 evaluates the contributions of the shared Core, CER, and RMC, while Figure A1 provides three-seed sensitivity analyses of λcer\lambda_{\mathrm{cer}}, λsar\lambda_{\mathrm{sar}}, and λmix\lambda_{\mathrm{mix}} on ImageNet-R and ImageNet-A. Each coefficient is varied independently with the remaining settings fixed.

Removing SAR produces the largest accuracy drop: 2.05 pp on ImageNet-R and 3.60 pp on ImageNet-A under Core-prior inference. In comparison, CER and RMC exhibit more modest, dataset-dependent variations around their selected weights. The default values are not always the best-performing settings, but generally lie in high-performing regions. These results indicate reasonable sensitivity to the selected coefficients, rather than establishing their joint optimality. The detailed results and analysis are provided in Appendix C.2 and Figure A1.

AuthorsOpenReview ↗

Response to Reviewer FhwE (2/2): Q5 & Q7, Q6

Q5 & Q7 — Core-Prior Definition and Transferability.

Core-prior (CP) is the entropy-based inference calibration in Eqs. (10)–(12), not an additional trained module. It uses shared-only predictions to adjust task-expert scores without test-time task identities. In task-agnostic continual learning, selecting a single Residual expert through hard routing would require making an expert-selection decision before evaluating the remaining expert predictions. CP instead uses the shared Core's task-level uncertainty as a common reference to calibrate all Residual scores, without requiring task identities or an additional learned router. This avoids committing to a single expert in advance, although all Residuals must still be evaluated.

To apply CP to CL-LoRA, we temporarily disable its task-specific LoRA contributions to obtain shared-only logits from the stored classifiers, while retaining each expert's original All-scan logits. We use the same normalized entropy, within-task softmax, βH=4\beta_H=4, and all-expert evaluation as in ECR-LoRA, without retuning or top-kk pruning.

Table R3 (same table as in our response to Reviewer DTFq, Q3). Matched CP comparison (ALastA_{\mathrm{Last}}, mean ± SD, %; three seeds on NVIDIA L20 GPUs). Full results in Table A9, Appendix D.5.

MethodImageNet-RImageNet-ACIFAR-100
CL-LoRA (All-scan)79.76 ± 0.2858.09 ± 0.9186.15 ± 0.22
CL-LoRA (+CP)80.47 ± 0.1259.96 ± 0.7487.81 ± 0.28
ECR-LoRA (All-scan)80.28 ± 0.2961.33 ± 1.3787.40 ± 0.20
ECR-LoRA (+CP)80.84 ± 0.0863.49 ± 0.4189.25 ± 0.23

CP increases CL-LoRA's final accuracy by 0.71, 1.87, and 1.66 percentage points on ImageNet-R, ImageNet-A, and CIFAR-100, respectively. This demonstrates transfer to a compatible LoRA architecture, without assuming applicability to every LoRA method.

We have added the precise inference construction, evaluation protocol, and full results in Appendix D.5 (Table A9) and clarified the matched comparison in Sec. 4.4.


Q6: Performance across benchmarks and task horizons. The improvements are not limited to ImageNet-R. In Table 1, ECR-LoRA with CP outperforms CL-LoRA in final accuracy by 6.56 pp on ImageNet-A, 3.62 pp on DomainNet, 2.52 pp on ImageNet-R, and 1.57 pp on CIFAR-100.

For longer sequences (Table 3), the gains over CL-LoRA remain 3.29 pp on 20-session ImageNet-A and 1.39 pp on 20-session ImageNet-R. However, the advantage is smaller at 5 sessions on ImageNet-R (+0.68 pp), where SplitLoRA performs slightly better (82.43 vs. 82.27). At 50 sessions, ECR-LoRA and CL-LoRA achieve comparable accuracy (71.73 vs. 71.71).

These results suggest that the benefits extend across datasets and task horizons, although their magnitude depends on the experimental setting.

Dear Authors,

Thanks for your response to my queries.

Although the main limitation of the proposal is still present (the increased time in both training and inference), you acknowledge it and provide additional experiments to address it. Moreover, this is not specific to your proposal but a general limitation of these types of methods.

Overall, most of my concerns and questions have been addressed. I will increase my score to 4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has helped us substantially improve the paper. We address each reviewer's comments in the individual responses below.

Revised PDF. All changes are highlighted in blue. The main revisions are:

  1. Sec. 3.3, Algorithm A1, and Appendix A.2: clarified Core initialization, Fisher-based support estimation, and SVD refactorization.
  2. Sec. 4.4 and Appendix D.5 (Table A9): matched comparison applying Core-prior calibration to CL-LoRA.
  3. Appendix D.3 (Tables A6–A7): matched CER/RMC controls on ImageNet-A and ImageNet-R.
  4. Appendix C.2 and Figure A1: corrected the default-setting points and clarified the experimental sources.
  5. Appendix D.1: analysis of shared-prefix reuse for inference.

Note on experimental setup. Owing to cluster availability during the rebuttal period, the new experiments were run on NVIDIA L20 GPUs, whereas the original submission used A40 GPUs. To ensure fair comparisons, all methods within each new table were rerun on the same hardware with identical data splits, random seeds, and evaluation protocols, and these results are reported separately from the original A40 results.

Dear Reviewers,

Could you take a moment to review the authors' responses and let them know whether any concerns remain or require further clarification?

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.