acceptodds
Under review as a conference paper at ICLR 2027

SPACE-LORA: ALLOCATING ACTIVATION-SUBSPACE PROTECTION FOR CONTINUAL LEARNING

12 views

Abstract

This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning (PEFT), Catastrophic Forgetting, Subspace Protection, Empirical Fisher Information

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2W4BOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPACE-LoRA, a LoRA-based continual learning method. For each new task, it projects the LoRA input factor onto the orthogonal complement of the dominant historical activation directions. As a result, the new branch gives zero response on that subspace. The number of protected directions in each module is chosen by a Fisher-coverage rule. Experiments cover vision benchmarks with ViT-B/16 and a 15-task language benchmark with T5-large.

Critical strengths
  1. Simple mechanism with an exact guarantee. A single projection after each optimizer step ensures zero response on the protected subspace, while keeping both LoRA factors trainable. This makes the method easy to implement.

  2. Broad evaluation setting. The method is tested on vision and language tasks, under both class- and task-incremental protocols. This suggests the approach is not tied to one architecture or protocol.

Critical weaknesses
  1. Limited novelty. Projecting updates onto the orthogonal complement of historical activation subspaces closely resembles GPM [1], applied here to the LoRA input factor. InfLoRA [2] and Adam-NSCL [3] also constrain new updates with respect to old input subspaces. The main new element appears to be the Fisher-coverage rule for choosing the protection size.

  2. The Fisher-coverage rule is not fully validated.

    • The rule is compared only with uniform allocation (Table 5). There is no comparison with energy-based allocation as in GPM, which the paper argues is inferior.
    • In that ablation, the Fisher rule raises accuracy but also increases forgetting.
    • The conclusion states that Fisher sizing does not consistently beat uniform sizing, which seems inconsistent with Section 5.4.
  3. Gains are limited and comparisons are narrow.

    • SPACE-LoRA trails InfLoRA and Janus-LoRA on ImageNet-R with 10 and 20 tasks (Table 1).
    • Recent methods on the same benchmarks, such as SD-LoRA [4] and CoSO [5], are not compared.

References

[1] Gradient Projection Memory for Continual Learning. ICLR 2021.

[2] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. CVPR 2024.

[3] Training Networks in Null Space of Feature Covariance for Continual Learning. CVPR 2021.

[4] SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning. ICLR 2025.

[5] Continuous Subspace Optimization for Continual Learning. NeurIPS 2025.

Questions for authors
  1. How does the Fisher-coverage rule compare with an activation-energy threshold at a matched average protected dimension?
  2. Section 5.4 and the conclusion seem to disagree on whether the Fisher rule beats uniform allocation. Which reflects the full results?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/4)

We sincerely thank the reviewer for the careful review. We also appreciate the reviewer's recognition that the method gives an exact guarantee with a simple mechanism that keeps both LoRA factors trainable, and that it is evaluated on vision and language under both class- and task-incremental protocols. We answer each weakness and question below. G1–G6 and Tables R1–R6 refer to the General Response, and [1]–[5] follow the reviewer's reference list. In particular, we address the reviewer's concerns about novelty (Weakness 1) and the validation of the Fisher-coverage rule (Weakness 2), to which we devote the most detailed replies below.

Weakness 1.

We agree with the reviewer that the orthogonal projection operator itself is standard, and that constraining parameter updates to subspaces orthogonal to historical input activations is a well-established strategy for mitigating catastrophic forgetting in continual learning. However, we strongly disagree that these similarities are sufficient to dismiss the novelty of the Residual Response Nullification proposed in Section 4.1. The novelty of our approach does not lie in introducing a new projection operator, but rather in how the projection is formulated and enforced within the specific optimization structure of continually accumulated low-rank residual adapters. We believe that the reviewer's assessment overlooks several fundamental distinctions between SPACE-LoRA and the cited methods.

First, SPACE-LoRA addresses a different optimization structure from GPM [1] and Adam-NSCL [3]. Both methods constrain updates to existing network weights by projecting gradients or optimizer-generated parameter updates onto subspaces associated with previous tasks. In contrast, SPACE-LoRA considers a continual adaptation setting in which previously learned weights and adapters remain frozen, while each incoming task introduces a new additive low-rank residual branch. In this setting, catastrophic forgetting can occur even without modifying any previously learned parameters, because the newly introduced branch may produce non-zero responses to historical activations. SPACE-LoRA explicitly formulates this additive interference as the object of protection and imposes a hard constraint directly on the input factor of the new branch. Consequently, the constraint is maintained at the parameter level throughout training, rather than relying solely on restricting the directions of individual optimization updates. This distinction is particularly relevant to factorized low-rank adaptation, where independently optimizing the two factors introduces an optimization structure absent from conventional full-weight projection methods.

Second, SPACE-LoRA differs fundamentally from InfLoRA [2] in how the adaptation subspace is determined and optimized. InfLoRA constructs a low-interference input subspace before training each task and fixes the corresponding LoRA input factor throughout optimization, allowing only the output factor to be learned. As a result, the directions available for adaptation are predetermined and cannot change during training. In contrast, SPACE-LoRA keeps both LoRA factors trainable while continuously enforcing orthogonality between the input factor and the protected historical activation subspace. This allows the model to learn and adjust its adaptation directions within the entire unprotected input space, rather than being restricted to a fixed low-dimensional subspace selected before optimization. Importantly, this is not merely a difference in implementation. It changes the set of low-rank updates that the model can represent and optimize while preserving the same historical response constraint. SPACE-LoRA therefore maintains the exact null-response property on the protected subspace without sacrificing the trainability of the input factor. Table R1 of the General Response (G1) quantifies this. Under an identical protection rule, keeping the input factor trainable is worth 1.94 points of final accuracy on 5-task ImageNet-R.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/4)

Third, SPACE-LoRA differs from all three methods in how the protection capacity is determined. GPM primarily selects historical directions according to activation energy, while Adam-NSCL identifies an approximate null space using the singular values of historical feature covariance. InfLoRA constructs its adaptation subspace using historical gradient-subspace information together with current-task representations. In contrast, SPACE-LoRA separates the identification of historical activation directions from the determination of their protection coverage. Historical activations determine the candidate directions and their ordering, while Fisher sensitivity measures how much past-task loss sensitivity is captured by protecting those directions. A common Fisher-coverage target then determines the protection dimensionality independently for each module, while preserving sufficient input dimensions for the fixed-rank LoRA adapter. This enables the protection constraint introduced in Section 4.1 to be applied with module-specific capacity determined by historical task sensitivity, rather than solely by representation energy or covariance magnitude.

Taken together, these distinctions demonstrate that SPACE-LoRA is not simply an application of GPM-style gradient projection to a LoRA parameter. While the underlying orthogonality principle is shared with prior work, SPACE-LoRA formulates the preservation of historical behavior as a hard constraint on newly introduced low-rank residual responses, maintains this constraint while jointly learning both LoRA factors, and determines the protected subspace dimensionality through Fisher sensitivity coverage. We therefore respectfully disagree with the implication that the use of a standard orthogonal projection operator diminishes the novelty of our formulation. The contribution lies in the constrained low-rank adaptation framework and its integration with sensitivity-guided protection allocation, rather than in the projection operator considered in isolation.

AuthorsOpenReview ↗

Response to Reviewer 3 (3/4)

Weakness 2 and Questions 1, 2. Validation of the Fisher-coverage rule.

Comparison with activation-energy allocation (Question 1). We thank the reviewer for suggesting an additional comparison with activation-energy-based allocation. In response, we have conducted additional experiments using the energy-based allocation rule following GPM and included the results in General Response G2. As shown in Table R2, Fisher-guided allocation consistently achieves higher final predictive accuracy than GPM-style energy-based allocation across all evaluated configurations. Specifically, Fisher-guided allocation improves final accuracy by 0.98–2.52 percentage points across ImageNet-R and VTAB, while requiring substantially fewer protected dimensions. Under the evaluated energy thresholds, GPM-style allocation protects approximately 1.67–2.51 times as many directions as our Fisher-coverage rule, yet consistently yields lower final accuracy. These results further support the effectiveness of the proposed Fisher-coverage rule in determining module-wise protection capacity, demonstrating that it achieves better final predictive performance with a smaller protected subspace under the evaluated settings. We appreciate the reviewer's constructive suggestion, which allowed us to strengthen the experimental validation of our proposed allocation strategy.

Section 5.4 and the Conclusion (Question 2). The reviewer also pointed out an apparent inconsistency between Section 5.4 and the Conclusion in how we discuss the comparison with uniform allocation. However, we would like to clarify that this concern appears to stem from a misunderstanding of the intended interpretation of our experimental results. Specifically, the performance advantage of the proposed Fisher-coverage rule discussed in Section 5.4 explicitly refers to final predictive performance. At the same time, Section 5.4 also reports the forgetting results, which favor uniform allocation. These observations should be interpreted in the context of the stability–plasticity trade-off, where the relative advantage of each allocation strategy depends on the evaluation perspective. It was precisely with this consideration in mind that we included the corresponding statement in the Conclusion. Nevertheless, we understand that the original presentation of Section 5.4 places relatively greater emphasis on predictive accuracy, which may have led readers to interpret the results as suggesting an overall superiority of Fisher-guided allocation. To make our intended interpretation clearer, we have refined the relevant descriptions in both Section 5.4 and the Conclusion, explicitly distinguishing the comparison in final predictive performance from that in forgetting and emphasizing their trade-off. We sincerely thank the reviewer for the careful reading and constructive comment, which provided an opportunity to clarify our intended interpretation.

Reviewer_HueuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SPACE-LoRA addresses catastrophic forgetting when task-specific LoRA adapters are accumulated during continual learning. For each adapted module, it builds an ordered basis from historical activation Gram matrices and uses an input-space empirical Fisher statistic to decide how long a prefix of that basis to protect. A single Fisher-coverage target therefore yields different protection dimensions across modules. While learning a new task, the method projects the new adapter’s input factor onto the orthogonal complement of the protected subspace after every optimizer step, keeping both new LoRA factors trainable and all previously learned parameters frozen. This gives exactly zero residual response for inputs in the protected subspace, without implying zero forgetting for the full network. The claimed contributions are the direct input-factor constraint and the Fisher-guided protection sizing. Experiments use ViT-B/16 on vision continual-learning benchmarks and T5-large on sequences of language classification and understanding tasks (Sections 3–5; Appendices A–B).

Critical strengths
  1. A clear and technically valid protection mechanism. The projection directly enforces a null response on the selected subspace, independently of the output factor. The method avoids a soft orthogonality penalty, and Appendix A is careful to distinguish this exact module-level property from network-level retention. This is a useful, appropriately scoped guarantee (Section 4.1, Eqs. 7–10; Appendix A).
  2. Encouraging results across distinct evaluation settings. The experiments cover class-incremental vision, task-incremental VDD, and language sequences of up to 15 tasks. In Table 3, SPACE-LoRA raises ten-task language accuracy from InfLoRA’s 70.83% to 73.95%, and fifteen-task accuracy from 67.54% to 69.79%. These gains deserve attention even though the method does not win on every benchmark (Section 5.2; Tables 1–3).
  3. Historical information is maintained without replaying raw examples. The activation and Fisher statistics are accumulated at task endpoints and reused later without storing previous-task training examples. This gives the method a clearly specified replay-free mechanism, although the cost of maintaining these statistics still needs to be quantified (Appendix B; Algorithm 2).
  4. An informative analysis of the stability–plasticity trade-off. Table 4 reports protected dimensions alongside accuracy and forgetting. Raising coverage from 0.75 to 0.99 increases the average protected dimension from 90 to 517 and generally reduces forgetting, while accuracy peaks at 0.90. The authors rightly acknowledge that stronger protection does not necessarily improve final accuracy (Section 5.3).
Critical weaknesses
  1. The main Fisher-allocation claim is not adequately isolated. Table 5 compares Fisher-guided protection with “Uniform,” but the uniform dimensions are not given, and it is not stated whether the total protection budgets are matched. The gains of 2.12 points on ImageNet-R and 1.34 points on ten-task language learning could therefore reflect a different overall amount of protection rather than a better distribution across modules. An activation-energy-based sizing rule within the same projection framework is also an important control, since Section 2 discusses this alternative. A comparison with reported, comparable protection budgets would directly test the claimed contribution. Fisher-guided protection has higher forgetting than uniform protection in both settings, but I do not see that alone as a failure; higher final accuracy can be a useful trade-off. The issue is attributing the improvement to the proposed allocation rule (Section 5.4; Table 5).
  2. The distinctive value of the enforcement mechanism needs clearer evidence. The paper’s own related-work discussion describes historical-representation projection in GPM and Adam-NSCL, and output preservation on historical activations in JANUS-LoRA. The real distinction is therefore the direct projection of the new input factor while keeping both factors trainable, together with the protection-sizing rule, not the general idea of preventing changes along historical activation directions. That distinction could be a useful contribution, but the current comparisons change several design choices at once. A focused comparison using the same historical basis and protection dimensions, with the input factor trainable versus fixed, would clarify its practical value. I am not claiming equivalence to prior methods; the concern is that the benefit of this specific change is not separated from the other changes (Sections 2 and 4.1).
  3. The reported results contain an unresolved inconsistency. Table 2 reports VDD accuracy of 86.26% for SPACE-LoRA with coverage 0.90 and 80.88% for InfLoRA. Figure 2 on page 8, however, assigns InfLoRA a zero VDD gap and SPACE-LoRA a 0.62-point gap, which reverses the ranking in the table. The figure does not indicate a different SPACE-LoRA configuration that would explain this. The mean-gap column should also be checked. The VDD result is also unusually sensitive to the coverage target: SPACE-LoRA reports 86.26% at ρ = 0.90 but 80.41% at ρ = 0.95, a 5.85-point swing, whereas the two settings differ by less than one point on every other benchmark in Tables 1–2. Taken together, these two observations make it unclear which VDD number is reliable. This is a reporting issue that needs clarification, not evidence that the experiments are invalid. In addition, Tables 2 and 3 have no uncertainty estimates, and the paper does not state the run counts or the meaning of the error bars shown elsewhere. This particularly limits interpretation of small gains, such as the 0.20-point improvement over InfLoRA on five-task language learning (Tables 1–3; Figure 2).
  4. Important experimental details are missing from the appendix. Section 5.1 promises details of the T5 adaptations of the vision-oriented baselines, but Appendix C mainly specifies shared training settings rather than method-specific adaptations. It describes selecting a single learning rate for all methods without reporting its value or a clear selection procedure. Common data and schedules are valuable controls, but a shared learning rate with default method-specific hyperparameters does not by itself establish comparable tuning, and this matters when interpreting the language gains. Separately, reporting the time and memory spent on statistics collection would improve the practical assessment, since the method maintains dense historical matrices and requires effective-weight gradients, eigendecompositions, and repeated projections. I regard this cost reporting as a secondary limitation, not an independent reason for rejection (Section 5.1; Appendix C.2; Algorithm 2).
Questions for authors
  1. Protection allocation. How is “Uniform” defined, and what are the per-module and total protected dimensions for both variants? Can the existing ablation be extended to uniform and activation-energy-based sizing under comparable protection budgets, keeping the basis construction and projection mechanism fixed?
  2. Mechanism and interpretation. What evidence isolates the advantage of training the input factor within the permitted subspace rather than fixing it? Also, Eq. 15 is unchanged if a module’s Fisher matrix is scaled by a positive constant. Should the claimed module-wise sensitivity adaptation therefore be understood as adaptation to the distribution of sensitivity across directions, rather than to its absolute magnitude across modules?
  3. Result reconciliation and uncertainty. Which VDD results and coverage setting underlie Figure 2? Please reconcile it with Table 2, check the mean-gap column, and explain why VDD is so much more sensitive to ρ than the other benchmarks. How many runs were used per benchmark, what do the error bars represent, and how variable are the main language results?
  4. Reproducibility. Please specify the T5 adaptations of InfLoRA, SplitLoRA, and JANUS-LoRA, the selected learning rate and its selection criterion, and the method-specific hyperparameters used. Was there any evidence that the common learning rate was reasonable for the compared methods?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/3)

We sincerely thank the reviewer for the careful and constructive review. We also appreciate the reviewer's recognition that the protection mechanism is technically valid and appropriately scoped, that the results are encouraging across distinct settings, including the language gains on 10 and 15 tasks, and that the stability–plasticity analysis is informative. We answer each question below. G1–G6 and Tables R1–R6 refer to the General Response.

Weakness 1 and Question 1. Protection allocation, and isolating the Fisher-allocation claim.

What the rule claims. The Fisher-coverage rule is meant to balance two things (§1). It avoids protecting directions a module does not need, so that more of the input space stays available for new tasks, and it protects the directions where old tasks are sensitive, so that those tasks are retained. The reviewer's question separates these two parts, and we answer each.

How much to protect. We note that the rule also sets the total, since each module decides how many directions it needs, namely the smallest activation-ordered prefix that covers a fraction ρ of its own Fisher sensitivity. A fixed total would therefore remove part of what the rule decides, so we report both views below. With the GPM energy thresholds used in practice (ε = 0.95 and 0.99), the energy rule protects 1.7–2.5× as many directions as the Fisher rule and is less accurate, as shown in Table R2 of the General Response. Following the reviewer's suggestion, we add this comparison to Table 5.

Where to protect. Here the Fisher rule decides the total and the alternative receives the same total, with the basis construction and the projection fixed.

  • The uniform baseline in Table 5 uses the Fisher rule's total budget split evenly across modules. At this budget the Fisher rule is more accurate on vision and language (79.09 vs 76.97, 73.95 vs 72.61) and has slightly higher forgetting (4.82 vs 4.40, 0.98 vs −0.12). As the reviewer noted, higher forgetting on its own is not a failure, it is the trade-off the rule is designed to make. The Fisher rule protects only the directions that cover a fraction ρ of the old tasks' sensitivity in each module and leaves the rest to new tasks, so new tasks are learned to a higher accuracy, while the small sensitivity left in those directions and the higher accuracy reached on each task raise forgetting slightly. Final accuracy already includes forgetting and is higher on both benchmarks.

Dimensions. Following the reviewer's suggestion, we report the per-module and total dimensions of both rules, as shown in Table R3 of the General Response. The uniform baseline spreads the Fisher rule's total evenly over the 24 key and value projections, while the Fisher rule gives 11–49 directions to blocks 0–1 and 461–493 to blocks 10–11.

At the same budget, the Fisher allocation therefore gives higher final accuracy than uniform allocation, and the energy rule at the thresholds used in practice protects more and is less accurate.

AuthorsOpenReview ↗

Response to Reviewer 2 (2/3)

Weakness 2 and Question 2. Mechanism and interpretation, and the distinctive value of the enforcement.

Trainable vs fixed input factor. Following the reviewer's suggestion, we ran this comparison, as shown in Table R1 of the General Response. All variants use the same protection rule. Each module protects the activation-ordered prefix that covers ρ = 0.90 of its Fisher sensitivity, computed from the run's own statistics, and the protected subspace is enforced in the same way. Only the input factor A is treated differently. On ImageNet-R 5T, keeping A trainable reaches 79.09, vs 77.15 with the InfLoRA-style fixed A and 77.17 with a fixed random A. The gain comes from learning new tasks (just-learned accuracy 82.65 vs 81.21 and 80.83) at similar forgetting (4.82 vs 5.08 and 4.57).

Eq. 15. The reviewer's reading is correct. The module-wise adaptation is to how a module's past Fisher energy is distributed over its activation-ordered directions, not to its absolute magnitude. Scaling a module's Fisher matrix by a > 0 scales the numerator and the denominator of Eq. 15 equally, Cm(k;aF)=a∑j≤kum,j⊤Fum,j/(a tr F)=Cm(k;F)C_m(k; aF) = a \sum_{j \le k} u_{m,j}^\top F u_{m,j} / (a\,\mathrm{tr}\,F) = C_m(k; F), so the same ρ selects the same kmk_m.

We normalize this way on purpose. A module that reaches ρ leaves at most a fraction 1 − ρ of its own Fisher energy in the unprotected subspace, so every module meets the same relative preservation criterion, and the number of directions needed to meet it differs by module. Whatever its Fisher trace, a module whose energy is concentrated in a few directions reaches ρ with few directions. A module whose energy is spread over many directions needs more, even if its trace is the same as another module's. The spread varies widely in practice.

  • In T5-large, the decoder cross-attention query reaches 90% coverage within 1–26 directions, while encoder self-attention modules need 450–615.
  • In ViT-B/16, the top 9 activation directions of block 0 hold 77–90% of the Fisher energy, while the deepest blocks need about 450–500 directions.

Weakness 3 and Question 3. Result reconciliation and uncertainty.

VDD. We thank the reviewer for catching this inconsistency. The entry was copied by hand from the run log into Table 2, and one digit was mistyped. The run log, Figure 2 and the mean-gap column all use the correct value, SPACE-LoRA (ρ = 0.90) 80.26 vs InfLoRA 80.88, hence the 0.62 gap (forgetting 3.32, unchanged). It also explains the apparent sensitivity to ρ, since the corrected ρ = 0.90 / 0.95 gap is 0.15 (80.26 vs 80.41), in line with the other benchmarks. The text now states that InfLoRA is the best method on VDD. The VDD entries are single runs.

Runs and error bars. Tables 1 and 4 and the Fisher row of Table 5 report three runs per entry, and ± is the standard deviation across those runs. Tables 2 and 3 report single runs.

Language variability. In order to answer the author's question regarding language results, we reran the language benchmark with the settings of the paper, changing only the seed. Over three seeds, the results vary by ±0.38 points (standard deviation), similar to the 0.2–0.6 points of the vision results in Table 1.

AuthorsOpenReview ↗

Response to Reviewer 2 (3/3)

Weakness 4 and Question 4. Reproducibility and missing details.

We thank the reviewer for reading the appendix closely and for pointing out these missing details. The expanded Appendix C now gives them as follows.

Shared setup (Appendix C, expanded).

  • AdamW with a constant learning rate of 1e−3 and weight decay 0.01, batch size 8, 3 epochs per task.
  • LoRA rank 8 with scaling 16 on the query and value projections (144 in total) of all attention modules of T5-large (encoder self-attention, decoder self-attention, decoder cross-attention).
  • A new adapter per task; maximum input length 128.

Method-specific adaptations and hyperparameters (official defaults). All three methods are applied to each q/v module of T5-large separately, using that module's own inputs, with their official update code.

  • InfLoRA (λ = 0.98, λ_e = 1.0) builds a frozen A for each module from its input subspace with DualGPM and trains B.
  • SplitLoRA (α = 20) builds each module's old-task subspace from its accumulated inputs and draws a frozen A from the minor subspace.
  • JANUS-LoRA (koek_{oe} = 50, ηoe\eta_{oe} = 1e−3, margin 0.3, temperature 0.07, DML weight 1.0) estimates each module's input subspace online, and its DML loss uses the last decoder hidden state as the feature. For O-LoRA, we use λ_orth = 0.05.

Learning rate. The batch size, number of epochs and learning rate follow the training settings of O-LoRA on this benchmark, as stated in §5.1. As described in Appendix C, the learning rate was selected once on the 5-task sequence, and the selected value, 1e−3, is used for every method and every sequence length without method-specific tuning. Method-specific hyperparameters stay at their official defaults.

We consider one shared setting the fair choice. If the learning rate were tuned separately for each method, the comparison would reflect how much tuning each method received rather than the methods themselves. We agree that a different learning rate can raise an individual method's result. Because this holds for every method, we kept the learning rate the same for all of them.

Statistics cost (Table R5 of the General Response). On ImageNet-R 10T, statistics take 2.3% of training time. Training speed equals InfLoRA's. Peak memory is 12.8 GB, vs 11.6 GB for InfLoRA and 12.6 GB for Janus-LoRA, and total time is +5.2% vs InfLoRA and -0.9% vs Janus-LoRA. For T5-large, the same pass runs once per task over the 144 q/v projections (d = 1024). For a module with input dimension d and output dimension m and a task with N examples of L tokens, it costs O(N (L m d + m d²)), linear in the task's data, and the projection costs O(r d k) per optimizer step for LoRA rank r and k protected directions.

We thank the reviewer again for the careful reading, including the appendix. The comparisons and corrections prompted by the reviewer's questions have made the allocation study and the reporting clearer.

Reviewer_Q3B2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This draft focused on continual learning with parameter-efficient fine-tuning (PEFT), especially CL with LoRA adapters. In this work, the authors considered the additive interference introduced by newly learned LoRA adapters on previous task activations. Specifically, the proposed method, named SPACE-LoRA, constrains the input factor of each new LoRA adapter to be orthogonal to the historical activation subspace through a hard projection after each optimization step. To balance stability and plasticity, the authors further proposed a module-wise protection allocation mechanism, where the historical activation Gram matrix determines the protected directions and Fisher information is used to adaptively determine the number of protected directions for each module. The authors provide theoretical analysis of the zero residual response property on the protected historical activation subspace. Experiments on multiple vision and language continual learning benchmarks were conducted to evaluate the proposed method against existing LoRA-based continual learning approaches.

Critical strengths
  1. In this draft, the authors considered the additive interference introduced by newly learned LoRA adapters, which may affect previous task predictions even when previously learned adapters are frozen. The proposed hard projection directly constrains the response of new adapters on historical activation directions, providing a principled mechanism for reducing interference with previous tasks.

  2. This draft enables a relatively straightforward implementation by directly projecting the LoRA input factor after each optimization step, while keeping both LoRA factors trainable. Compared with methods that require reconstructing the full weight updates, such as Janus-LoRA, the proposed approach provides a simpler way to enforce the orthogonality constraint.

  3. In the experimental part, the authors evaluated SPACE-LoRA across multiple vision and language continual learning benchmarks, including ImageNet-R, DomainNet, VTAB, VDD, and NLP classification tasks. The experiments cover both class-incremental and task-incremental learning settings, providing a relatively comprehensive evaluation of the proposed method across different architectures and task sequences.

  4. The authors provided the implementation code to help the readers better understand the proposed method.

Critical weaknesses
  1. The practical advantages of the proposed hard projection were not sufficiently investigated. Although this manuscript directly projects the LoRA input factor while keeping both factors trainable, the current experiments do not clearly isolate the contribution of this design from other differences between the compared methods. Moreover, comparisons of computational overhead and optimization efficiency with closely related methods, such as InfLoRA and Janus-LoRA, were not provided.

  2. The ablation studies on Fisher-guided protection allocation were not sufficient. In Table 5, the authors compared the proposed strategy with uniform allocation, but the protection dimensions of the two strategies were not reported. Other protection criteria, such as activation-energy-based allocation and direct Fisher-sensitivity-based direction selection, were also not compared. Therefore, it is not clear whether the performance gain comes from the use of Fisher information or differences in the protection capacities.

See the Questions part for more details.

Questions for authors

Questions For Authors

  1. In the experimental part, the authors provided comparisons with InfLoRA, SplitLoRA, and Janus-LoRA. However, other relevant LoRA-based continual learning methods, such as CL-LoRA [1] and LoDA [2], were not included. I recommend the authors provide further comparisons with these methods on the vision class-incremental benchmarks, using the same pre-trained backbone and task splits, with comparable training budgets. Any differences in replay settings or additional information used during training should also be clarified. Moreover, the authors should discuss the differences between SPACE-LoRA and these methods in preserving previous knowledge and maintaining adaptation capacity for new tasks. This would help the readers better understand the advantages and limitations of the proposed method.
  2. About the proposed hard projection. This manuscript constrains the LoRA input factor through projection after each optimization step, while keeping both factors trainable. This is different from freezing the input factor or reconstructing the factor updates. However, the current comparisons involve differences in both the projection mechanism and the protection allocation. I wonder how much of the performance gain comes from the proposed projection mechanism itself. Could the authors provide a controlled comparison of these strategies using the same protected subspaces and protection dimensions? This would help the readers understand the role of directly projecting the input factor.
  3. About the ablation studies. In Section 5.4 and Table 5, the authors provide comparisons between Fisher-guided allocation and uniform allocation. However, how was the protection dimension selected for the uniform baseline? Was its total or average protection dimension matched to that of Fisher-guided allocation? I recommend the authors report these dimensions and provide further comparisons with activation-energy-based allocation under the same hard projection mechanism and comparable protection budgets. This would help the readers figure out whether Fisher information provides additional benefits beyond changing the amount of protection.
  4. As for the computational efficiency, the authors should provide comparisons of computational overhead with existing methods, particularly InfLoRA, SplitLoRA, and Janus-LoRA, under matched experimental settings. This would help clarify the computational overhead introduced by the proposed protection mechanism and historical statistics.

References

[1] CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning. CVPR 2025.

[2] Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning. ICML 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/3)

We sincerely thank the reviewer for the positive and constructive review. We also appreciate the reviewer's recognition that the hard projection is a principled way to control additive interference, that it is simpler to implement than reconstructing full weight updates, and that it is evaluated across a broad set of benchmarks. Several weaknesses and questions ask about the same point, so we answer them together below. G1–G6 and Tables R1–R6 refer to the General Response.

Weakness 1 and Questions 2, 4. Practical advantages of the hard projection, isolation of its contribution, and computational overhead.

Where the advantage comes from. In a LoRA branch, the interference on old inputs is the new branch's response B A x. Projecting the input factor A onto the complement of the protected subspace after every step makes this response exactly zero on that subspace for any B. The guarantee therefore holds while both A and B stay trainable.

Isolating the contribution. Following the reviewer's suggestion, we keep the protection rule fixed (the same ρ, basis construction and projection), and change only how A is treated, as shown in Table R1 of the General Response. On ImageNet-R 5T, keeping A trainable reaches 79.09, vs 77.15 with the InfLoRA-style fixed A and 77.17 with a fixed random A. The gain comes from learning new tasks better, as reflected in the higher final accuracy at similar forgetting (4.82 vs 5.08 and 4.57).

Why the alternatives that adapt A are only approximate. With the same protected basis and dimensions, the alternatives differ in how the constraint is enforced.

  • Gradient projection (GPM style). It projects A's gradient onto the complement. With plain SGD this keeps A in the complement, but with Adam the per-coordinate rescaling of the projected gradient reintroduces components along the protected directions, and this drift accumulates over training steps.
  • Orthogonality penalty. A loss term in the style of O-LoRA, here λ‖A V‖², trades the constraint against the task loss, so any finite λ leaves a nonzero residual.
  • Factor reconstruction (Janus-LoRA). It projects the full update and reconstructs the factor updates by regularized least squares, so the composite update only approximately follows the projected direction.
  • Freezing A (InfLoRA, LoDA). The constraint stays exact, but A no longer adapts. The controlled comparison above measures that cost.

SPACE-LoRA applies an idempotent projection to A itself after every step, so A V = 0 holds at floating-point precision for the whole run, while A stays trainable.

Computational overhead. The measured cost is in Table R5 of the General Response. Training speed equals InfLoRA's, and the total time is +5.2% vs InfLoRA and −0.9% vs Janus-LoRA. Peak memory is 12.8 GB, vs 11.6 GB for InfLoRA and 12.6 GB for Janus-LoRA. The table below breaks the cost down for one adapted module with input dimension d, output dimension m, LoRA rank r and k protected directions, and a task with N examples of L tokens each.

StepCostWhenMeasured share
Projection A ← A − (A V)VᵀO(r d k)every optimizer stepnegligible (training speed equals InfLoRA's)
Activation Gram, Σ x xᵀO(N L d²)once per taskcollected inside the Fisher pass
Input-space Fisher, ∑nGn⊤Gn\sum_n G_n^\top G_n with Gn=∑tδtxt⊤G_n = \sum_t \delta_t x_t^\topO(N (L m d + m d²)), plus one forward/backward pass with per-example gradientsonce per taskalmost all of the statistics time
Eigendecomposition and directional energiesO(d³)once per tasksmall (d = 768)
Stored statisticstwo d×d matrices per module, independent of the number of tasks——

The projection adds O(r d k) per step, far below the module's own forward/backward cost of O(L d m) per example, which is why training speed is unchanged. The only substantial extra term is the Fisher pass. It runs once per task, is linear in the task's data, and takes 2.3% of training time. SplitLoRA, like InfLoRA, derives its subspaces once per task from statistics of the task data, so its overhead is of the same order as InfLoRA's.

AuthorsOpenReview ↗

Response to Reviewer 1 (2/3)

Weakness 2 and Question 3. Ablation and budget of the Fisher-guided allocation, and comparison with activation-energy allocation.

Budget of the uniform baseline. The uniform baseline in Table 5 uses the same total protection budget as the Fisher rule, split evenly across modules and rounded to whole directions per module. Following the reviewer's suggestion, we report the per-module and total dimensions of both rules, as shown in Table R3 of the General Response.

What the comparison shows. At the same budget, the Fisher rule is more accurate on vision and language (79.09 vs 76.97 on ImageNet-R, 73.95 vs 72.61 on the language benchmark) with slightly higher forgetting (4.82 vs 4.40 and 0.98 vs −0.12). This follows from how the rule spends the budget. It protects each module only until a fraction ρ of the old tasks' sensitivity is covered. Where that sensitivity is concentrated in a few directions, it protects few directions and leaves the rest to new tasks, which are therefore learned better. The directions left to new tasks carry at most a fraction 1 − ρ of the old tasks' sensitivity, so using them changes old-task predictions only slightly, and a task learned to a higher accuracy also has more accuracy that can decrease afterwards. Both raise forgetting slightly. Forgetting alone can always be lowered by protecting more, and within the Fisher rule, raising ρ from 0.90 to 0.99 lowers forgetting from 4.82 to 3.50 but also lowers accuracy from 79.09 to 78.30 (Table 4). As Reviewer 2 also noted, higher forgetting on its own is not a failure, and higher final accuracy can be a useful trade-off. Final accuracy already includes the effect of forgetting, and the Fisher rule is higher on it by 2.12 points on ImageNet-R and 1.34 points on the language benchmark.

Comparison with activation-energy allocation. An activation-energy rule sizes each module by a threshold ε on activation energy, as in GPM. With the thresholds used in practice (ε = 0.95 and 0.99), it protects 1.7–2.5× as many directions as the Fisher rule and is less accurate at both thresholds on both benchmarks, by 0.98–2.52 points, as shown in Table R2 of the General Response, and we add this comparison to Table 5.

Choosing directions directly by Fisher sensitivity. We order candidate directions by activation energy and use Fisher only to decide how many to protect, deliberately.

  • The quantity that interferes with old tasks is the new adapter's response B A x on old inputs. For fixed adapter norms, its size outside the protected subspace is bounded by the old activation energy left outside it.
  • Protecting the top activation directions removes the most old-activation energy for a given k, which minimizes this bound (Appendix A).
  • Directions chosen by Fisher sensitivity alone can carry little activation energy, so protecting them would leave much of the old activation energy outside the protected subspace, where new adapters still respond to it.

The coverage rule combines the two. Activation energy selects the directions, and loss sensitivity sets how many are needed.

AuthorsOpenReview ↗

Response to Reviewer 1 (3/3)

Question 1. Comparison with CL-LoRA and LoDA.

Following the reviewer's suggestion, we ran both methods with their official code under the protocol of Tables 1 and 2, with the same backbone, class order, task splits, optimizer, schedule and number of epochs, and no exemplars, as shown in Table R6 of the General Response. Method-specific hyperparameters stay at their official defaults, and no method, including SPACE-LoRA, was tuned on these results. All three methods are rehearsal-free, and none stores raw examples. SPACE-LoRA keeps only per-module activation Gram and input-space Fisher statistics.

  • SPACE-LoRA is more accurate than CL-LoRA on the five benchmarks (79.09 vs 68.06 on ImageNet-R 5T, 86.05 vs 64.37 on VTAB), and CL-LoRA's forgetting is 10–32 points and grows with the number of tasks. CL-LoRA protects its shared blocks 0–5 only by distillation, and it predicts by comparing features with class prototypes that are computed once and not updated. When the shared blocks change, old classes' features can move away from their stored prototypes, which is consistent with forgetting that grows over tasks.
  • On ImageNet-R, where the tasks share similar input statistics, LoDA is close to SPACE-LoRA (−0.77 on 5T, +0.80 on 10T, −0.61 on 20T relative to SPACE-LoRA). On 10T, LoDA is slightly better in both just-learned accuracy (80.62 vs 80.25) and forgetting (5.99 vs 6.42).
  • On DomainNet and VTAB, whose tasks come from different domains, SPACE-LoRA is more accurate by 1.36 and 6.34 points and forgets less by 2.60 and 1.83 points. LoDA fixes A to directions computed before training and learns only B, so it adapts less to a new domain than SPACE-LoRA, which trains A in the complement of the protected subspace. LoDA also has no constraint that keeps the response on old activations at zero.

How each method preserves old knowledge and keeps capacity for new tasks.

  • CL-LoRA splits adaptation into task-shared adapters, whose down-projections are random orthogonal matrices and whose shared knowledge is preserved by knowledge distillation, and task-specific adapters with learnable block-wise weights. Preservation is soft. It is encouraged by a loss, and no subspace of old activations is guaranteed to be unaffected.
  • LoDA decomposes the input space into a general subspace shared across tasks and an isolated subspace of directions strongly activated by the current task. The down-projection of each branch is fixed to these bases, and only the up-projection is learned. Preservation comes from subspace isolation with a frozen input factor, so new-task capacity is limited to what the chosen bases allow.
  • SPACE-LoRA gives an exact, B-independent guarantee on the protected subspace of each module while keeping both factors trainable in its complement. Its new-task capacity is the complement, of dimension d−kmd - k_m, and the Fisher rule keeps the complement large in modules whose sensitivity is concentrated in a few directions. Keeping the input factor trainable, rather than fixing it as LoDA and InfLoRA do, is worth +1.9 points under the same protection rule, as shown in Table R1 of the General Response.

We thank the reviewer again. The controlled comparison, the budget and dimension report and the cost analysis that the reviewer asked for have made the contribution of each component clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.