acceptodds
Withdrawn from ICLR 2027

RELORA-CL: FROM REUSABLE UPDATES TO RELI- ABLE REPRESENTATION MAINTENANCE IN CLASS- INCREMENTAL LEARNING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient class-incremental learn- ing, but successive updates shift historical class representations and render stored prototypes outdated. Without historical inputs, correcting this mismatch requires estimating their displacement from current-task samples. We investigate how continual updates can accommodate new classes while keeping historical represen- tation drift amenable to estimation. Our analysis reveals depth-dependent functional reuse: early-task input subspaces remain more sufficient for shallow-block updates, whereas deeper blocks increasingly require additional directions. Motivated by this observation, we propose reuse-aware LoRA (ReLoRA-CL), which preserves learned input subspaces in shallow blocks and expands complementary directions in deeper blocks. Semantic drift compensation and balanced classifier alignment subsequently maintain historical class statistics and decision boundaries. We formu- late prototype migration as finite-sample drift estimation and derive an error bound involving local drift regularity, support mismatch, and residual uncertainty. Under additional sensitivity assumptions, we connect drift regularity to consecutive-task parameter changes, providing a conditional rationale for structured adaptation. ReLoRA-CL improves final accuracy over the strongest baseline by 3.58, 1.64, and 4.53 percentage points on ImageNet-R with 20 tasks, CIFAR-100 with 10 tasks, and ImageNet-A with 10 tasks, respectively. Drift diagnostics further show lower local drift variation and prototype displacement estimation error. The code is available at: https://anonymous.4open.science/r/ReLoRA-8C3A/

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning, Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning, Representation Drift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dZirOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of old-class prototype mismatch caused by continual LoRA updates in exemplar-free class-incremental learning. ReLoRA-CL fixes the input subspace learned in the first task and updates the output factors in shallow layers, while adding new components orthogonal to historical directions in deeper layers. It also preserves knowledge of old classes through semantic drift compensation and classifier alignment. Its main contributions are an analysis of depth-dependent subspace reuse, corresponding adaptation strategies, and a conditional theoretical analysis of prototype transfer error.

Critical strengths

Modeling prototype compensation reliability has explanatory value. The conditional error bound distinguishes drift regularity, insufficient coverage of old classes by the current samples, and residual uncertainty, rather than simply assuming that limiting parameter updates can prevent forgetting.

Critical weaknesses
  1. The experiments do not include two closely related works: CL-LoRA already uses a fixed random input projection in shallow layers and task-specific LoRA in deeper layers; MACIL already combines LoRA, mean-shift compensation, and classifier alignment. Therefore, the paper’s novelty should lie primarily in its analysis of depth-dependent reuse of learned subspaces and its connection to drift estimability, rather than in layer-wise division of labor or the compensation pipeline itself.

  2. The central explanation that “drift is easier to estimate” has not been sufficiently validated. Proposition 2 compares only fixed reference updates and their projections, rather than the two solutions obtained through actual optimization. Although the authors explicitly acknowledge this limitation, the absolute prototype displacement estimation error in the experiments is also affected by drift magnitude and feature scale, and no controlled comparison of individual modules under consistent compensation and alignment conditions is clearly reported. It is therefore difficult at present to distinguish whether the gains come from more estimable drift, smaller drift, or the statistical maintenance itself.

3.The experimental performance is less than satisfactory. As far as I know, overall it should not be as effective as the LoDA method. If you could additionally explain the advantages of your method over LoDA, this would make your method more convincing.

CL-LoRA: CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning

MACIL: Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning

LoDA: Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Questions for authors

My main concerns are all listed in the weaknesses. Overall, this paper is fairly solid. If the author addresses my concerns well, I will consider raising my rating.

Reviewer_GX22OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies exemplar-free class-incremental learning with LoRA adaptation. The key contribution is a depth-aware LoRA update strategy that preserves reusable directions in shallow layers while expanding complementary task-specific directions in deeper layers. Experiments are conducted on ImageNet-R, CIFAR-100, and ImageNet-A. The method reports improvements over the listed baselines.

Critical strengths
  1. Preserving reusable directions in shallow blocks while expanding complementary directions in deeper blocks provides a meaningful connection between layer-wise representation behavior and parameter-efficient adaptation.
  2. Results on ImageNet-R, CIFAR-100, and ImageNet-A, including different task counts on ImageNet-R are good.
Critical weaknesses
  1. The contributions of SDC and CA are difficult to disentangle from those of RSA and CDE. SDC and CA may improve performance by compensating for prototype drift, independently of the proposed hierarchical LoRA update strategy. Therefore, the ablation should not be limited to removing SDC and CA from ReLoRA-CL. The paper should also augment representative comparison methods, such as standard or unrestricted LoRA, with the same SDC and CA modules under identical settings.
  2. The layer boundary is manually selected. The method uses a fixed boundary such as L_s=4. Although a sensitivity analysis is included, there is no principled rule for selecting this boundary for a new architecture or dataset.
  3. The core observation that shallow layers tend to encode reusable features while deeper layers capture task-specific semantics is well established in deep representation learning. ReLoRA-CL mainly operationalizes this observation by preserving LoRA input directions in shallow blocks and adding orthogonal directions in deeper blocks.
  4. SDC and CA appear to be adopted from existing prototype-migration and classifier-alignment techniques rather than introduced as new methodological contributions. Moreover, their connection to RSA/CDE is relatively weak: SDC and CA mainly compensate for prototype and classifier changes after representation drift, but they do not arise directly from the proposed hierarchical LoRA formulation. Thus, the paper combines a potentially incremental LoRA update strategy with existing auxiliary components and presents the combination as a unified contribution.
  5. The main contribution is the depth-aware update of LoRA factors. However, the theoretical section primarily analyzes prototype migration and estimation error, without establishing why the proposed RSA/CDE updates are preferable to uniform LoRA updates or how they directly induce the stated guarantees. As a result, the theory adds technical complexity but provides limited support for the core contribution. The paper would be stronger if the authors simplified the presentation, clearly separated supporting analysis from the main methodological contribution, and developed theory that directly explains the benefit or limitations of hierarchical LoRA updating.
Questions for authors

Refer to [Weaknesses].

Reviewer_sF5jOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on exemplar-free class-incremental learning on Vision Transformer (ViT) with low-rank adaptation (LoRA). Motivated by an observation of depth-dependent functional reuse, this paper proposes ReLoRA-CL. The proposed method applies Reusable Subspace Adaptation (RSA) in shallow Transformer blocks to preserve the Task-0 input subspace, and Complementary Direction Expansion (CDE) in deeper blocks to introduce new orthogonal directions. Semantic drift compensation is used for prototype migration and balanced classifier alignment is utilized to mitigate classifier bias. The theorectical derivation provides a conditional and generic bound for prototype displacement estimation, but the connection to the motivation and proposed method is weak. Experiments show improved performance over the compared baselines.

Critical strengths
  1. This paper provides a new theoretical deviation of a conditional bound of prototype displacement.
  2. Empirical results demonstrate good performance of proposed method under the settings in the manuscripts.
Critical weaknesses
  1. Connection between theorectical deviation and the proposed methods is weak. This paper provides several proposition of the conditional bound of prototype migration and magnitudes of consecutive updates. However, the theoretical analysis does not directly justify the actual RSA-CDE optimization procedure or key design choices such as the shallow/deep layer allocation, and several assumptions required by the analysis are strong and not empirically validated.
  2. Presentation of the derivation is somehow unclear and confusing. For example, several notations are used in similar forms and can easily lead to confusion. Also, the derivations and the introduction of several key expressions lack sufficient motivation and necessary assumptions.
  3. The key observation of depth-dependent update reuse lacks sufficient theorectical analysis. Why is the descriptive score formulated in that form? Why this metric can be useful? Also, this observation is only validated in the ImageNet-R (T=10), which leads to a problem whether this observation also holds universally.
  4. The selection of LsL_s is an empirical experience, making the metric hard to implement in different scenarios.
  5. Empirical study is not sufficient. The experiments only focus on three datasets in small task. It is highly recommended to validate the proposed method in large scales of tasks (T=50, 100, etc.). Also, the datasets in validation have fair number of classes, it is interesting to validate the proposed method in dataset with larger classes (e.g., OminiBench-1k [R1]).
  6. The section of related work/ literature review is missing.
  7. Minor concern of the layout. For example, the distance between the caption of Figure 1 and the text is too small.

[R1] Meng Lou, et al., Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts, ICML 2026.

Questions for authors

Please refer to the weaknesses. I think this paper is not ready, the connection between theorectical derivation and proposed method should be strenthened, further discussion on the metric for seperating swallow/deep layers should be provided, and the corresponding experiments should be included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.