acceptodds
Withdrawn from ICLR 2027

RE-LoRA: Residual Evolving LoRA for Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rehearsal-free class-incremental learning (CIL) with frozen pre-trained models is commonly realized through Low-Rank Adaptation (LoRA), yet existing methods still face a fundamental stability-plasticity-efficiency dilemma: they either allocate task-specific low-rank components whose parameter footprint grows with the number of tasks, or keep a compact shared adapter and rely on a regularization loss term to reduce inter-task interference. We propose Residual Evolving LoRA (RE-LoRA), a rehearsal-free framework built upon a single evolving LoRA that attains strong stability and sustained plasticity under a constant parameter budget. RE-LoRA comprises three components. Residual Adapter Learning (RAL) trains each task's low-rank update as a residual that is additively composed with the frozen consolidated adapter in weight space, so that a new task refines rather than overwrites historical knowledge without any auxiliary regularization. Historical Knowledge Consolidation (HKC) then merges the residual update into the historical adapter and re-compresses the result via truncated SVD, which yields the optimal low-rank approximation under a fixed rank budget, so that the single adapter evolves across tasks while its parameter and memory footprints remain constant. Finally, Prototype-Guided Task Routing (PGTR) uses stable prototypes from the frozen backbone to identify the task and masks the logits to its classes, eliminating prediction bias in inference. We prove that HKC achieves the optimal approximation of the RAL combined adaptation in the Frobenius norm under a given rank constraint, and that its truncation error grows at most linearly without compounding over the task stream. Extensive experiments on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A across ViT-B/16, DINO, and iBOT backbones show that RE-LoRA consistently outperforms state-of-the-art LoRA-based CIL baselines on every evaluated dataset.

Primary area:
optimization
Keywords:
Continual learning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_EBtjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RE-LoRA studies rehearsal-free class-incremental learning with a frozen pretrained backbone. It trains a residual LoRA alongside a frozen historical adapter, then compresses their summed update to a fixed rank using truncated SVD. At inference, PGTR uses frozen-backbone class prototypes to infer a task and masks all logits outside that task. The paper evaluates this combination on four benchmarks and analyzes the weight approximation error introduced by compression.

Critical strengths
  • The residual-and-compression procedure is simple and maintains a fixed deployed adapter rank without replay or an auxiliary regularization loss.

  • The experiments span several datasets, task splits, and pretrained backbones, and report parameter, storage, and inference costs.

  • The ablations expose the large effect of task routing and compare prototypes from frozen and adapted features.

Critical weaknesses
  • The headline accuracy primarily reflects task routing rather than the proposed adapter mechanism. Before PGTR, final class-incremental accuracy is 39.26% on ImageNet-R and 29.37% on ImageNet-A; with PGTR, it becomes 90.72% and 66.50%. PGTR exploits the benchmark’s known class-to-task partition to restrict every prediction to one task. Without applying the same router to the baselines, the reported comparisons do not establish an advantage in continual adaptation or retention.

  • HKC is a standard fixed-rank approximation of the summed updates, not an objective designed to preserve old-task behavior. Its Frobenius-norm guarantee does not imply an accuracy guarantee, and adding HKC to RAL does not improve final task-identified accuracy on either ImageNet-R or ImageNet-A. The evidence supports its role in controlling adapter size more clearly than its claimed role in preventing forgetting.

  • The dependence on routing limits the conclusions. The paper reports perfect task identification, but Figure 6 shows aggregated prototype similarities rather than per-sample routing errors. Its behavior under different class groupings, unequal task sizes, or imperfect task identification is unclear; PGTR also requires approximately an additional backbone pass.

Questions for authors
  • How do all main baselines perform with the identical PGTR router?

  • What are PGTR’s per-sample task-identification errors under alternative class-to-task partitions, and how quickly does final accuracy degrade when routing is imperfect?

  • How does HKC change old-task accuracy immediately before and after compression across different rank budgets?

Reviewer_z9b6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RE-LoRA for rehearsal-free class-incremental learning. It learns task-specific residual updates, consolidates historical and new updates into a fixed-rank adapter through truncated SVD, and uses frozen-backbone class prototypes for task routing and logit masking. Experiments report strong performance across four datasets, multiple task schedules, and supervised and self-supervised backbones.

Critical strengths

Fixed-rank consolidation provides a practical mechanism for limiting adapter growth across tasks. The evaluation covers multiple datasets, task lengths, and pretrained backbones, with statistics over five runs. The Task-IL ablations also support RAL’s contribution to within-task classification and HKC’s ability to largely maintain accuracy after compression.

Critical weaknesses

1.Performance attribution and routing validation remain incomplete. PGTR produces substantial gains in Table 5, but comparisons against strong baselines using identical routing and masking are missing. Moreover, the prototype-similarity analysis does not establish the claimed 100% sample-level routing accuracy, which is central to interpreting the results.

2.The knowledge-preservation claims exceed the current guarantees. Freezing the historical branch does not prevent residual updates or classifier training from changing historical predictions. The theoretical analysis establishes weight-reconstruction properties without connecting them to historical-task performance.

3.The scalability conclusions require stronger evidence. Longer task sequences also produce smaller routed candidate sets, confounding the interpretation of improved accuracy. Efficiency accounting primarily concerns inference and does not fully characterize training, consolidation, prototype construction, or storage growth.

Questions for authors

Questions that need the author to answer 1.Matched-routing comparisons and performance attribution: In Table 5, the final Class-IL accuracy on ImageNet-R with 10 tasks increases from 39.26% with +HKC to 90.72% after adding PGTR, a gain of 51.46 percentage points. Without PGTR, performance is also substantially below most baselines in Table 1. This indicates considerable dependence on routing and logit masking. Although the Task-IL ablations support RAL’s contribution to within-task classification, they do not establish whether its advantage over strong existing baselines persists under identical routing. Please provide or verify the Class-IL results of vanilla LoRA, InfLoRA, and EWC-LoRA using the same PGTR and logit masking, to distinguish adaptation gains from inference-mechanism gains.

2.Evidence for 100% routing accuracy: The paper reports 100% sample-level task-identification accuracy, yet Figure 6 presents prototype similarities, whose diagonal dominance does not imply zero sample-level routing errors. Since incorrect routing directly excludes the true class, the current evidence is insufficient to substantiate this strong claim. Please provide actual task-identification confusion counts for each dataset and run, together with the corresponding evaluation code.

3.Why does freezing historical parameters protect historical predictions? Freezing the historical adapter only preserves its parameters; unconstrained residuals and the continually updated shared classifier can still alter old-task predictions. Interpreting this freezing as structural knowledge protection is therefore insufficiently justified. Please explain the protection mechanism and report old-task accuracy before residual training, after residual training, and after HKC, under identical routing conditions.

4.The gap between weight reconstruction and continual-learning stability: Proposition 2.1 establishes the Frobenius-optimal reconstruction of merged weights through truncated SVD. However, small singular components may still be important for distinguishing historical classes, so reconstruction optimality does not imply performance preservation. Proposition 2.2 fixes the residual sequence generated during actual training and compares its compressed and uncompressed accumulation. It neither controls how compression affects subsequent learning nor excludes forgetting in the reference model itself. Appendix B.3 acknowledges this limitation, yet the main text still claims that RAL and HKC guarantee feature-level historical knowledge preservation. Can the authors connect reconstruction error to historical-task performance, or explicitly restrict the guarantee to weight-compression fidelity? Furthermore, Appendix B.3 questions the well-posedness of Frobenius-distance comparisons because the adapters have different ranks, although their effective updates have the same matrix dimensions. Please clarify this reasoning.

5.Interpretation of task-length results: In Table 3, increasing ImageNet-R from 5 to 20 tasks reduces the routed candidate set from 40 to 10 classes. Higher accuracy may therefore reflect easier within-task classification and cannot directly establish robustness to longer task streams. Can the authors distinguish these effects by controlling masking granularity or reporting corresponding forgetting metrics?

6.Complete efficiency accounting: Table 6 primarily evaluates inference and is insufficient to establish overall efficiency. Training retains both historical and residual adapters, consolidation uses dense SVD, and prototype storage grows with the number of classes. Can the authors report peak training memory, consolidation and prototype-construction time, and total storage as the class count grows, and clarify precisely which components are covered by the “constant budget” claim?

Reviewer_1mCPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RE-LoRA addresses rehearsal-free class-incremental learning by learning residual LoRA updates, consolidating them at fixed rank through truncated SVD, and routing predictions using frozen-backbone prototypes and task-based logit masking. Experiments span four datasets, multiple task lengths, and supervised/DINO/iBOT backbones, alongside an analysis of compression error for a fixed sequence of realized residual updates.

Critical strengths
  1. Table 5 directly supports fixed-rank consolidation: with task identities provided, adding HKC changes final accuracy from 90.96% to 90.72% on ImageNet-R and 66.72% to 66.50% on ImageNet-A, largely preserving within-task performance.
  2. Tables 1 and 3–5 cover diverse settings, while ablations with and without task identities help distinguish representation learning from task identification.
Critical weaknesses
  1. PGTR raises ImageNet-R final accuracy from 39.26% to 90.72% in Table 5, matching the result with oracle task identities, and pages 8 and 18 report 100% routing. However, Appendix C's Figures 5–6 and Eq. (17) analyze aggregated class means; neither this geometry nor matching classification accuracy establishes perfect sample-level routing, leaving this central result insufficiently substantiated.
  2. Appendix B.3 aggregates errors over 24 modules on page 15 but applies Corollary 2.3's single-matrix constant C globally on page 16. A shared module-wise bound generally gives a global per-step bound of sqrt(24)·C; CIFAR-100's first nonzero global error, 0.93, exceeds C≈0.55. This requires correcting the numerical validation, while the single-matrix derivation and linear-growth form remain intact.
Questions for authors
  1. Could you provide per-task routing correct/total counts, the class-to-task partition, and the Eq. (5) inference code to verify that 100% routing comes from individual test samples querying all seen-class prototypes?
  2. Are α, C, and the errors in Appendix B.3 computed per module or globally, and what consistent aggregation explains 0.93 versus approximately 0.55?
Reviewer_YhrnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets rehearsal-free class-incremental learning with a frozen ViT adapted by LoRA. It argues that existing methods either grow per-task components or depend on regularization that costs memory and erodes plasticity. RE-LoRA trains each task as a residual LoRA on top of a frozen historical adapter (RAL), then merges the two and compresses the sum back to a fixed rank by truncated SVD (HKC). At inference, PGTR assigns each sample to a task via its nearest frozen-backbone class prototype and masks logits outside that task. Two propositions give Eckart–Young optimality of the compression and a linear bound on accumulated truncation error. Experiments on four benchmarks and three backbones claim consistent gains over prompt- and LoRA-based baselines at constant parameter cost.

Critical strengths
  1. The evaluation is broad and comprehensive: four benchmarks, three backbones and three sequence lengths (Tables 1 to 4). The efficiency table also discloses RE-LoRA’s doubled inference cost (Table 6).
  2. Table 5 reports Task-IL alongside Class-IL and includes a learnable-prototype routing variant. Appendix B.3 tests the assumption behind Corollary 2.3 and admits the bound is loose.
Critical weaknesses
  1. Without PGTR, RAL+HKC reaches 39.26 and 29.37 final accuracy on ImageNet-R and ImageNet-A (Table 5). That is below every Table 1 baseline, including Finetune.
  2. With PGTR, Class-IL equals Task-IL exactly (Table 5), and Appendix C reports 100% routing. RE-LoRA is therefore scored with effectively oracle task identity, while baselines are not. Table 3 shows the signature: RE-LoRA and Full Expansion, both routed, improve from 5 to 20 tasks while every baseline degrades.
  3. 100% sample-level routing on ImageNet-A is hard to reconcile with Fig. 6, where class-mean margins fall to 0.56 vs 0.52 (row 6). Individual test features are noisier than class means. Task leakage, e.g. through per-task test loaders, must be ruled out.
  4. RAL resembles frozen-accumulator schemes such as LAE and InfLoRA’s branch merging. HKC is SVD re-compression of summed adapters, close to KnOTS, TSV-M and MagMax.
  5. PGTR is the frozen nearest-class-mean classifier of SimpleCIL/APER, EASE and FeCAM, applied to task inference as in HiDe-Prompt. None of these is compared, nor are SLCA and TUNA, both cited in Section 2.4.
Questions for authors
  1. What are the sample-level routing accuracy and the frozen nearest-prototype class accuracy per dataset, including DomainNet? Is the test set evaluated per task, with any batch-level aggregation or access to the task index?
  2. Do you have results for SD-LoRA, InfLoRA, EWC-LoRA or MoCL with PGTR, or their Task-IL accuracy? Does RAL+HKC still lead under the same routing?
  3. Why does RAL+HKC without PGTR fall below Finetune? Do Vanilla LoRA and Finetune differ in classifier training or old-class logit masking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.