- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SPACE-LoRA, a LoRA-based continual learning method. For each new task, it projects the LoRA input factor onto the orthogonal complement of the dominant historical activation directions. As a result, the new branch gives zero response on that subspace. The number of protected directions in each module is chosen by a Fisher-coverage rule. Experiments cover vision benchmarks with ViT-B/16 and a 15-task language benchmark with T5-large.
Critical strengths
-
Simple mechanism with an exact guarantee. A single projection after each optimizer step ensures zero response on the protected subspace, while keeping both LoRA factors trainable. This makes the method easy to implement.
-
Broad evaluation setting. The method is tested on vision and language tasks, under both class- and task-incremental protocols. This suggests the approach is not tied to one architecture or protocol.
Critical weaknesses
-
Limited novelty. Projecting updates onto the orthogonal complement of historical activation subspaces closely resembles GPM [1], applied here to the LoRA input factor. InfLoRA [2] and Adam-NSCL [3] also constrain new updates with respect to old input subspaces. The main new element appears to be the Fisher-coverage rule for choosing the protection size.
-
The Fisher-coverage rule is not fully validated.
- The rule is compared only with uniform allocation (Table 5). There is no comparison with energy-based allocation as in GPM, which the paper argues is inferior.
- In that ablation, the Fisher rule raises accuracy but also increases forgetting.
- The conclusion states that Fisher sizing does not consistently beat uniform sizing, which seems inconsistent with Section 5.4.
-
Gains are limited and comparisons are narrow.
- SPACE-LoRA trails InfLoRA and Janus-LoRA on ImageNet-R with 10 and 20 tasks (Table 1).
- Recent methods on the same benchmarks, such as SD-LoRA [4] and CoSO [5], are not compared.
References
[1] Gradient Projection Memory for Continual Learning. ICLR 2021.
[2] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. CVPR 2024.
[3] Training Networks in Null Space of Feature Covariance for Continual Learning. CVPR 2021.
[4] SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning. ICLR 2025.
[5] Continuous Subspace Optimization for Continual Learning. NeurIPS 2025.
Questions for authors
- How does the Fisher-coverage rule compare with an activation-energy threshold at a matched average protected dimension?
- Section 5.4 and the conclusion seem to disagree on whether the Fisher rule beats uniform allocation. Which reflects the full results?
6 comments
Response to Reviewer 3 (2/4)
Third, SPACE-LoRA differs from all three methods in how the protection capacity is determined. GPM primarily selects historical directions according to activation energy, while Adam-NSCL identifies an approximate null space using the singular values of historical feature covariance. InfLoRA constructs its adaptation subspace using historical gradient-subspace information together with current-task representations. In contrast, SPACE-LoRA separates the identification of historical activation directions from the determination of their protection coverage. Historical activations determine the candidate directions and their ordering, while Fisher sensitivity measures how much past-task loss sensitivity is captured by protecting those directions. A common Fisher-coverage target then determines the protection dimensionality independently for each module, while preserving sufficient input dimensions for the fixed-rank LoRA adapter. This enables the protection constraint introduced in Section 4.1 to be applied with module-specific capacity determined by historical task sensitivity, rather than solely by representation energy or covariance magnitude.
Taken together, these distinctions demonstrate that SPACE-LoRA is not simply an application of GPM-style gradient projection to a LoRA parameter. While the underlying orthogonality principle is shared with prior work, SPACE-LoRA formulates the preservation of historical behavior as a hard constraint on newly introduced low-rank residual responses, maintains this constraint while jointly learning both LoRA factors, and determines the protected subspace dimensionality through Fisher sensitivity coverage. We therefore respectfully disagree with the implication that the use of a standard orthogonal projection operator diminishes the novelty of our formulation. The contribution lies in the constrained low-rank adaptation framework and its integration with sensitivity-guided protection allocation, rather than in the projection operator considered in isolation.
Response to Reviewer 3 (3/4)
Weakness 2 and Questions 1, 2. Validation of the Fisher-coverage rule.
Comparison with activation-energy allocation (Question 1). We thank the reviewer for suggesting an additional comparison with activation-energy-based allocation. In response, we have conducted additional experiments using the energy-based allocation rule following GPM and included the results in General Response G2. As shown in Table R2, Fisher-guided allocation consistently achieves higher final predictive accuracy than GPM-style energy-based allocation across all evaluated configurations. Specifically, Fisher-guided allocation improves final accuracy by 0.98–2.52 percentage points across ImageNet-R and VTAB, while requiring substantially fewer protected dimensions. Under the evaluated energy thresholds, GPM-style allocation protects approximately 1.67–2.51 times as many directions as our Fisher-coverage rule, yet consistently yields lower final accuracy. These results further support the effectiveness of the proposed Fisher-coverage rule in determining module-wise protection capacity, demonstrating that it achieves better final predictive performance with a smaller protected subspace under the evaluated settings. We appreciate the reviewer's constructive suggestion, which allowed us to strengthen the experimental validation of our proposed allocation strategy.
Section 5.4 and the Conclusion (Question 2). The reviewer also pointed out an apparent inconsistency between Section 5.4 and the Conclusion in how we discuss the comparison with uniform allocation. However, we would like to clarify that this concern appears to stem from a misunderstanding of the intended interpretation of our experimental results. Specifically, the performance advantage of the proposed Fisher-coverage rule discussed in Section 5.4 explicitly refers to final predictive performance. At the same time, Section 5.4 also reports the forgetting results, which favor uniform allocation. These observations should be interpreted in the context of the stability–plasticity trade-off, where the relative advantage of each allocation strategy depends on the evaluation perspective. It was precisely with this consideration in mind that we included the corresponding statement in the Conclusion. Nevertheless, we understand that the original presentation of Section 5.4 places relatively greater emphasis on predictive accuracy, which may have led readers to interpret the results as suggesting an overall superiority of Fisher-guided allocation. To make our intended interpretation clearer, we have refined the relevant descriptions in both Section 5.4 and the Conclusion, explicitly distinguishing the comparison in final predictive performance from that in forgetting and emphasizing their trade-off. We sincerely thank the reviewer for the careful reading and constructive comment, which provided an opportunity to clarify our intended interpretation.
Response to Reviewer 3 (1/4)
We sincerely thank the reviewer for the careful review. We also appreciate the reviewer's recognition that the method gives an exact guarantee with a simple mechanism that keeps both LoRA factors trainable, and that it is evaluated on vision and language under both class- and task-incremental protocols. We answer each weakness and question below. G1–G6 and Tables R1–R6 refer to the General Response, and [1]–[5] follow the reviewer's reference list. In particular, we address the reviewer's concerns about novelty (Weakness 1) and the validation of the Fisher-coverage rule (Weakness 2), to which we devote the most detailed replies below.
Weakness 1.
We agree with the reviewer that the orthogonal projection operator itself is standard, and that constraining parameter updates to subspaces orthogonal to historical input activations is a well-established strategy for mitigating catastrophic forgetting in continual learning. However, we strongly disagree that these similarities are sufficient to dismiss the novelty of the Residual Response Nullification proposed in Section 4.1. The novelty of our approach does not lie in introducing a new projection operator, but rather in how the projection is formulated and enforced within the specific optimization structure of continually accumulated low-rank residual adapters. We believe that the reviewer's assessment overlooks several fundamental distinctions between SPACE-LoRA and the cited methods.
First, SPACE-LoRA addresses a different optimization structure from GPM [1] and Adam-NSCL [3]. Both methods constrain updates to existing network weights by projecting gradients or optimizer-generated parameter updates onto subspaces associated with previous tasks. In contrast, SPACE-LoRA considers a continual adaptation setting in which previously learned weights and adapters remain frozen, while each incoming task introduces a new additive low-rank residual branch. In this setting, catastrophic forgetting can occur even without modifying any previously learned parameters, because the newly introduced branch may produce non-zero responses to historical activations. SPACE-LoRA explicitly formulates this additive interference as the object of protection and imposes a hard constraint directly on the input factor of the new branch. Consequently, the constraint is maintained at the parameter level throughout training, rather than relying solely on restricting the directions of individual optimization updates. This distinction is particularly relevant to factorized low-rank adaptation, where independently optimizing the two factors introduces an optimization structure absent from conventional full-weight projection methods.
Second, SPACE-LoRA differs fundamentally from InfLoRA [2] in how the adaptation subspace is determined and optimized. InfLoRA constructs a low-interference input subspace before training each task and fixes the corresponding LoRA input factor throughout optimization, allowing only the output factor to be learned. As a result, the directions available for adaptation are predetermined and cannot change during training. In contrast, SPACE-LoRA keeps both LoRA factors trainable while continuously enforcing orthogonality between the input factor and the protected historical activation subspace. This allows the model to learn and adjust its adaptation directions within the entire unprotected input space, rather than being restricted to a fixed low-dimensional subspace selected before optimization. Importantly, this is not merely a difference in implementation. It changes the set of low-rank updates that the model can represent and optimize while preserving the same historical response constraint. SPACE-LoRA therefore maintains the exact null-response property on the protected subspace without sacrificing the trainability of the input factor. Table R1 of the General Response (G1) quantifies this. Under an identical protection rule, keeping the input factor trainable is worth 1.94 points of final accuracy on 5-task ImageNet-R.