- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AO-LoRA to reduce pretrained representation degradation during continual learning. It accumulates Householder-based low-rank updates anchored to pretrained weights, with learnable scaling and inter-task orthogonality regularization. Experiments show competitive performance across several continual learning benchmarks.
Critical strengths
-
Strong empirical performance. AO-LoRA improves final accuracy over the strongest reported continual-learning baseline by 2.18 percentage points on CIFAR-100 and 2.01 points on ImageNet-R with 20 tasks. Positive results across different pretrained backbones further support its practical value.
-
Informative representation analysis. Held-out linear probes and class-prototype similarity analyses provide complementary evidence about changes in transferability and representation structure. These diagnostics give the method a concrete empirical motivation and help assess pretrained knowledge retention beyond accuracy on previously learned tasks.
Critical weaknesses
-
The proposed mechanism is not fully supported. The weighted residual sum in Eq. (9) is generally not orthogonal, so preservation of feature geometry is not guaranteed. The ablations also leave the source of the gains unclear: anchored orthogonal updates improve final accuracy over LoRA by 0.40 points, while adding scaling brings another 2.99 points. A LoRA baseline with the same scaling and comparable regularization is needed to establish the benefit of the proposed design.
-
The results are sensitive to shuffling on ImageNet-A. Enabling shuffling reduces final accuracy from 64.44% to 58.66% and average accuracy from 74.97% to 67.21% (Tables 8 and 10). These changes are much larger than those of CL-LoRA and SD-LoRA. Although AO-LoRA still has the highest final accuracy, this sensitivity reduces confidence in the consistency of its gains and requires further explanation.
Questions for authors
-
What geometric property does Eq. (9) preserve, given that the weighted residual sum is generally not orthogonal?
-
Under matched training settings and parameter budgets, does AO-LoRA still outperform LoRA with the same learnable scaling and comparable orthogonality regularization?
5 comments
(Weakness 1 and Question 1 & 2)
(3) Controlled comparison with LoRA under matched experimental settings
Following the reviewer’s valuable suggestion, we introduce Scaled LoRA and LoRA with I-Orth. as controlled baselines to investigate the contributions of task-specific scaling and inter-task orthogonality, respectively, while retaining the standard additive low-rank update formulation of LoRA.
Scaled LoRA: Scaled LoRA adopts the same task-specific bounded scaling mechanism as AO-LoRA. Specifically, its task-specific update is defined as:
where is a learnable, task-specific scaling factor with the same parameterization and initialization as in AO-LoRA.
The results are summarized in Table 3. The results suggest that the effect of scaling depends on the underlying update mechanism. For the LoRA baseline, the scaling factor modulates the unconstrained low-rank update determined by the factors and . In particular, it does not impose additional structural constraints on the update directions of LoRA. In contrast, for A-Orth., the scaling factor modulates the contribution of a structurally parameterized update. It provides an additional task-specific degree of freedom for controlling the strength of the orthogonally induced adaptation.
Table 3. Comparison of LoRA baseline (LoRA), LoRA with scaling (Scaled LoRA), Anchored Orthogonal Adaptation (A-Orth.), and A-Orth. with scaling (Scaled A-Orth.) on ImageNet-R with 20 tasks.
| Method | Update mechanism | Avg. | Stability | Plasticity | Forgetting () | |
|---|---|---|---|---|---|---|
| LoRA | 74.13 | 81.11 | 80.24 | 81.70 | 8.15 | |
| Scaled-LoRA | 72.65 | 80.35 | 79.44 | 80.44 | 8.48 | |
| A-Orth. | 74.53 | 82.82 | 82.21 | 81.08 | 7.00 | |
| Scaled A-Orth. | 77.52 | 83.93 | 82.49 | 83.37 | 6.36 |
LoRA with I-Orth.: Regarding inter-task orthogonality, AO-LoRA constrains the overlap between the low-rank directions projected through the pretrained weight matrix. Since this constraint is specifically designed for the Householder-based parameterization, it cannot be directly applied to the LoRA baseline. Therefore, when constructing the inter-task orthogonality regularizer for LoRA, we follow the design of O-LoRA [1]. Specifically, we define the subspace spanned by the columns of as the update subspace of task : .
To encourage the update subspace of the current task to be orthogonal to those of previous tasks, we impose the following constraint:
Accordingly, we define the inter-task orthogonality loss as:
\left|\left|A^{(i)\top}A^{(t)}\right|\right|_F^2$$ The final training objective is therefore formulated as: $$L^{t} = L_{ce}^{(t)} + \lambda L_{orth}^{(t)}$$ where $\lambda$ controls the strength of the inter-task orthogonality regularization. The results are summarized in **Table 4**. **Inter-task orthogonality is also effective when applied to the LoRA baseline**, suggesting that encouraging task-specific update subspaces to have minimal overlap is an effective way to mitigate forgetting in continual learning. **Table 4.** Comparison of LoRA baseline (LoRA), LoRA with inter-task orthogonality (LoRA with I-Orth.), Anchored Orthogonal Adaptation (A-Orth.), and A-Orth. with inter-task orthogonality (A-Orth. with I-Orth.) on ImageNet-R with 20 tasks. Method | $\overline{A}_{20}$ | Avg. | Stability | Plasticity | Forgetting ($\downarrow$) -------|-----------|-----------|-----------|-----------|----------- LoRA | 74.13 | 81.11 | 80.24 | 81.70 | 8.15 LoRA with I-Orth. | **75.47** | 82.37 | 82.31 | **82.01** | 7.52 A-Orth. | 74.53 | 82.82 | 82.21 | 81.08 | 7.00 A-Orth. with I-Orth. | 75.21 | **83.10** | **82.49** | 81.75 | **6.22** [1] Orthogonal subspace learning for language model continual learning, EMNLP 2023. --- We sincerely thank the reviewer for this insightful question. In response, we have - **1)** Revised the explanation **(Section 3.2: Line 294-298)** to make this distinction explicit. - **2)** Further discussed the contributions of AO-LoRA in the extended ablation study **(Section 4.3: Line 463-467 and Table 4)**. - **3)** Included controlled evaluations of scaling and regularization using LoRA baselines in both the main text and the appendix **(Section 4.3: Line 481-485, Section D.7: Line 1322-1379)**. We sincerely appreciate the reviewer’s opportunity to clarify these concerns.(Weakness 2) Performance on ImageNet-A under class shuffling
We sincerely thank the reviewer for helping us identify this issue. We acknowledge that AO-LoRA exhibits sensitivity to class shuffling on ImageNet-A.
To better understand this phenomenon, we revisited the training logs and conducted additional analyses across 5 random seeds under both shuffling settings. As shown in Figure 1. We observe that, without shuffling, the class order and task composition remain fixed across different seeds, resulting in relatively consistent training behavior. In contrast, enabling shuffling changes the class composition of each incremental task, potentially leading to substantial variations in task difficulty.
Under the shuffled setting, the model tends to maintain relatively high training accuracy while exhibiting lower test accuracy. We conjecture that this behavior may be partly attributed to the limited and non-uniform number of samples per class in ImageNet-A, which may make the model more susceptible to overfitting under certain task compositions.
We referenced the experimental configurations provided in the official code of related works [1-3] for ImageNet-A, we noticed that these methods use different learning rates and numbers of training epochs. Motivated by this observation, we tried different training parameter settings to see if they could alleviate the problem shown in the figure. Our results show that, although a performance gap remains, appropriately reducing the number of training epochs can partially mitigate the performance degradation of AO-LoRA under the shuffle=True setting, as shown in Figure 2.
Figure 2. Training and test performance with different number of training epochs on ImageNet-A.
[1] CL-LoRA: Continual low-rank adaptation for rehearsal-free class-incremental learning, CVPR 2025.
[2] Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning, ICLR 2025.
[3] Revisiting weight regularization for low-rank continual learning, ICLR 2026.
We thank the reviewer again for raising this important question, which provides valuable insights and directions for our future research and we will pay closer attention to this issue. We have clarified this limitation and incorporated the additional analysis into the revised manuscript (Section D.1: Line 1095-1100).
We would like to express our sincere gratitude to Reviewer jynw for the positive comments on the strengths of our work and for providing us with the valuable opportunity to further clarify its contributions. Below, we address each point raised by the reviewer and describe the corresponding revisions made to the manuscript, which are highlighted in blue. We hope that our explanations can address the Reviewer's concerns, and we sincerely welcome any further comments or suggestions.
(Weakness 1 and Question 1 & 2) Geometric Properties and the Contributions of AO-LoRA
We sincerely thank the reviewer for raising this important concern regarding the underlying geometric properties and the contributions of AO-LoRA. We address the concern from the following three aspects:
(1) Geometric properties and design motivation of Eq. (9)
We agree that the scaled residual accumulation in Eq. (9) does not generally constitute an orthogonal transformation. This design choice is motivated by the key consideration: Enforcing strict orthogonality transformation constraints may limit the model’s ability to adapt to new tasks.
This observation is supported by our preliminary experimental results, as reported in Table 1 (OFT-based). These results suggest that strict orthogonality constraints may impose excessive restrictions on parameter updates in continual learning scenarios, thereby hindering the model’s adaptability to new tasks.
Table 1. Comparison between LoRA-based methods (InfLoRA, SD-LoRA, CL-LoRA), Strict Orthogonal Transformation (OFT-based), and Ours (AO-LoRA) on ImageNet-R with 20 tasks. All results were obtained using the same random seed.
Therefore, compared with LoRA-based methods, Eq. (9) is designed to mitigate arbitrary parameter changes induced by unconstrained additive updates. To preserve sufficient plasticity, AO-LoRA does not enforce strict orthogonality between the accumulated weights and the pretrained weights. Instead, it uses orthogonality as a structural constraint on task-specific update directions. This design helps mitigate the degradation of pretrained representations caused by unconstrained updates, while preserving sufficient flexibility for adaptation to new tasks.
(2) Explanation of improvement in the ablation study
To further disentangle the contributions of different components in AO-LoRA, we extend the ablation study in Table 2 (Numbered as Table 4 in the revised manuscript) by additionally reporting Stability, Plasticity, and Forgetting. These metrics provide further insights into the sources of performance improvement.
Table 2. Ablation study on ImageNet-R with 20 tasks. The first row shows the LoRA baseline. OFT denotes the results obtained by applying orthogonal fine-tuning alone.
Although anchored orthogonal adaptation (A-Orth.) achieves a final-task accuracy (A20) of 74.53% comparable to the 74.13% achieved by the LoRA baseline, it exhibits different stability and plasticity. Specifically, A-Orth. considerably improves model stability, suggesting that its structured low-rank updates provide a more stable adaptation mechanism than conventional LoRA updates. Furthermore, incorporating the task-specific scaling factor (Scal.) introduces additional flexibility in controlling the magnitude of task-specific updates, thereby improving the model's ability to adapt to new tasks and enhancing its plasticity.