acceptodds
Under review as a conference paper at ICLR 2027

RG-LoRA: Residual Geometry-Guided Continual Low-Rank Adaptation

40 views

Abstract

Continual learning requires a model to acquire new tasks while retaining earlier knowledge, yet sequential adaptation can cause catastrophic forgetting. This challenge persists in low-rank adaptation, where task updates may interfere with previously learned representations. We ask whether past updates can themselves provide a bounded history for guiding future adaptation. We introduce RG-LoRA, a residual geometry-guided approach to continual low-rank adaptation. RG-LoRA summarizes each learned residual into a rank-bounded history that captures the directions and strengths of past model changes. This history attenuates new updates along strongly represented directions without preventing their reuse. After each task, the learned residual is merged into the adapted model, while its geometry updates the bounded history. Across continual classification and instruction-generation benchmarks, RG-LoRA is competitive with reported baselines on T5-Large and achieves substantially higher final performance on Llama-2-7B-Chat and Llama-3.2-3B, with larger gains on longer task sequences. These results suggest that residual geometry provides a compact and effective signal for continual low-rank adaptation without requiring a separate post-task data collector.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Catastrophic Forgetting, Residual Geometry, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Wiq9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RG-LoRA, which is a residual geometry-guided continual adaptation method. RG-LoRA utilizes historical residual geometry to induce task-fixed soft attenuation around new LoRA updates along strongly represented directions, and merges the learned residual into the adapted model after learning each task. RG-LoRA requires no replay and no growing LoRAs, which can save storage cost. Experimental results show the effectiveness of RG-LoRA on T5-Large and Llama models.

Critical strengths

The paper proposes a novel method for continual learning, using historical input and output residual directions to induce task-fixed soft attenuation around a new LoRA update. This idea is novel in LoRA-based continual learning and gives insights into considering the relationship between previous knowledge and new knowledge.

Critical weaknesses
  1. The experimental results need more analysis on the components of RG-LoRA, since current results focus on the performance of RG-LoRA across models and tasks, but it does not show how each component of RG-LoRA itself contributes to the final results.

  2. The quality of the paper needs to be improved, such as the explanation of the method. For example, in section 4.2, the title is “task-fixed soft attenuation”, but it only shows a bunch of equations without a coherent story connecting all equations. I think the (b) part in Figure 1 is clear in explaining the purpose of each component, but there is no such clear explanation in Section 4.2.

Questions for authors
  1. Can authors explain why RG-LoRA defines Eq.(16) as such? Is there any derivation, reference, or evidence?

  2. Why are Eq.(20) and Eq.(22) equal? How to obtain Eq.(22) from Eq.(21) and Eq.(20)? Also, why do you use Eq.(21)? Since current explanations in Section 4.2 lack a connection between individual parts, explaining the relationship between these equations makes it better.

  3. There are not enough ablation studies to analyze the components of RG-LoRA. For example, the storage cost and computation cost of RG-LoRA since it involves two additional HinH_{in} and HoutputH_{output}, or analyzing the effect of HinH_{in} and HoutputH_{output} by removing either of them.

  4. Can authors evaluate the performance of RG-LoRA on Qwen model?

  5. Table 1 and Table 2 overlap in the results of RG-LoRA on T5-large, Llama-2-7B-Chat, and Llama-3.2-3B. It should use Table 2 to show new results on ablation studies.

4 comments

Thank authors for responding to the rebuttal and appreciating the added experimental results. The responses addressed part of my concerns. I think from the table for Q3, the input-only variant seems identical to Full RG-LoRA, and I think this is an important comparison for establishing the contributions of RG-LoRA, but the current results do not show the evidence. I would like to keep my original score.

AuthorsOpenReview ↗

Clarification on the output-side contribution

Thank you for clarifying this concern. We apologize for not presenting the additional comparison in Table 4 (p. 10) directly in our earlier reply.

The example in Appendix E.3, Eq. (80) explains the different roles of the two sides. Input attenuation acts on feature directions, whereas output attenuation acts on layer-output directions. An old and a novel input can share an output direction. Input attenuation can protect the old input while leaving the novel one unchanged, whereas output-only attenuation affects both. This motivates using output-side attenuation as complementary control, rather than as a replacement for input-side protection.

Table 2 reports final scores. To examine what the output map adds, Table 4 measures changes in old-task predictions that may not change the decoded answer. Before each merge, both maps act on the same Full-trained proposal. Input-only is scaled to match Full's total residual norm across adapted layers, controlling for global update magnitude.

The two metrics measure different aspects of preservation:

  • Reference NLL measures the negative log-probability of an old-task reference answer. Lower NLL means the model assigns more probability to that answer.
  • Predictive KL measures the change from the pre-task model's next-token distribution. This reference has the current residual disabled and all earlier merged updates retained. We compare full vocabulary distributions at identical reference-answer prefixes. Lower KL means less predictive drift, not necessarily higher accuracy.

We average over answer tokens within each example, then over examples, excluding prompt tokens. The table summarizes evaluation-only old-task probes across task boundaries. KL is reported in units of 10⁻³.

SettingInput-only (norm-matched) NLL ↓Full NLL ↓Input-only (norm-matched) KL ↓Full KL ↓
CL O10.16480.16450.6610.675
CL O40.25320.251414.2129.794
SuperNI O11.10741.106220.03316.607

Full reduces old-task predictive KL by approximately 31% on CL O4 and 17% on SuperNI O1, with slightly lower NLL on all three settings. CL O1 has slightly higher KL under Full. NLL complements KL by checking reference-answer probability as well as distributional change. Together, these measurements provide additional evidence supporting the output map in Full RG-LoRA beyond global norm reduction. The complete protocol and results are in Appendix F.4, Table 23 (p. 30).

We hope the clarification above and the evidence in Table 4 address your remaining concern about the output-side contribution. We would appreciate your reconsideration.

As the discussion period is coming to an end, we would greatly appreciate your feedback on our additional clarification.

Thank you again for your time and consideration.

Reviewer_Dt9yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose RG-LoRA, which is a method for continual learning that uses the geometry of previously learned LoRA updates to guide adaptation to new tasks while reducing interference with earlier knowledge. Instead of storing previous task adapters or enforcing strict orthogonality constraints, the approach maintains a compact, fixed-size summary of historical updates using input- and output-side Gram matrices. This information is used to softly attenuate new updates along directions that were strongly represented in previous tasks, while still allowing those directions to be reused. After each task, the learned update is merged into the backbone and the historical representation is updated. The method achieves competitive results on T5-Large and reports substantial improvements over published baselines on Llama-2-7B-Chat and Llama-3.2-3B, particularly on longer task sequences and SuperNI. Ablation studies suggest that historical information contributes to performance, with input-side attenuation accounting for the majority of the observed benefit

Critical strengths
  • RG-LoRA is well-motivated, extending existing subspace-based continual learning methods. The approach is based on using the geometry of previously learned LoRA updates to guide subsequent adaptations. Instead of enforcing strict orthogonality to previous update subspaces, the approach softly attenuates updates along historically prominent directions while allowing those directions to be reused.
  • Their representation provides bounded historical represenatations without need for retaining task-specific adapters. The approach also avoids retaining previous training data or a growing collection of task-specific adapters. A fixed-size historical state is practically attractive for continual adaptation, particularly as task sequences grow.
Critical weaknesses
  • The strongest performance claims aren't supported by controlled baselines comparisons: baseline results are taken from previously published papers rather than from implementations evaluated under a consistent experimental protocol. This makes it difficult to determine how much of the reported improvement can be attributed to RG-LoRA itself. For example, RG-LoRA uses a gradient-aligned LoRA initialisation based on the first training accumulation group, rather than the standard random LoRA initialisation. There are also differences in training protocols between RG-LoRA and ASO-LoRA, including learning rates and optimisation settings.
    Since the substantial improvements over existing methods are a central contribution of the paper, I think stronger controlled comparisons are important. Ideally would want the strongest relevant baselines to be rerun with matched training settings, model backbones, task data, LoRA initialisation, and optimisation budget.

  • The experiments don’t sufficiently isolate the contribution of their residual-geometry approach: The approach is based on the hypothesis that directions strongly represented in previous learned updates are useful proxies for directions that should be protected against future interference. The ablations demonstrate that historical attenuation is beneficial, but they do not establish that these directions are particularly important for preserving previous task performance. I would like to see more evidence supporting the hypothesis. Eg. Investigating whether perturbations along high-history-strength directions produce greater old-task degradation than perturbations along lower-strength directions could provide more direct evidence for the proposed mechanism.

  • Limited analysis of long-term retention: The use of either 4 or 15 tasks in the experiments is relatively limited for evaluating the usefulness of a historical representation for continual learning. Because the history is repeatedly compressed, it is unclear how well important directional information is preserved over substantially longer task sequences. An evaluation over longer sequences would help establish the scalability of the approach beyond its bounded storage requirements.

Questions for authors

From Table 4 it looks like the input-side operator may be the main driver of performance, with input-only attenuation nearly matching the full method and output-only attenuation performing a lot worse. Why is the input-side representation so much more useful? Do the authors expect this behaviour to generalise across architectures and tasks? Would a simpler input-only method perform equally well across more settings?

Reviewer_g3g5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether learned LoRA updates can provide the historical information needed for subsequent continual adaptation. RG-LoRA extracts input- and output-side Gram matrices from each executed residual and consolidates them into bounded spectral histories. These histories define inverse-square-root transformations that attenuate future updates along previously represented directions while leaving those directions available for reuse. After each task, the executed residual is merged into the backbone and the history is updated. Experiments cover T5-Large and two Llama backbones on classification and instruction-generation streams. The strongest reported gains occur on the Llama long-sequence and SuperNI settings. The paper also provides three-seed final results, selected retention measurements, and sensitivity analyses.

Critical strengths
  1. Computing geometry from the executed residual avoids dependence on a particular factorization of the same LoRA update. The compact QR/SVD construction and trace-preserving compression make the bounded-state claim concrete, rather than simply asserting that previous information can be compressed.

  2. Three-seed results and per-seed scores are reported, rather than only the best run. Performance is competitive on T5-Large and substantially higher than the cited Llama baselines on longer streams. The history-budget and adapter-rank sweeps provide additional context.

Critical weaknesses
  1. Comparing newly trained models against published results (Table 1) confounds the method's effectiveness with discrepancies in training setups. RG-LoRA utilizes gradient-aligned initialization and specific hyperparameters (e.g., a SuperNI learning rate of 2e-5 vs. Merge Before Forget's 5e-5) that differ from the baselines. This discrepancy is highlighted in Table 4: the no-history variant achieves 38.00 on SuperNI O1, far exceeding the published SeqLoRA result of 18.4. While RG-LoRA improves further to 52.03, the initial gap shows the training setup itself is doing heavy lifting.

  2. Table 4 proves attenuation maps help, but not why. RG-LoRA alters both directional scaling and optimization dynamics (via the identity-Jacobian surrogate). Because the attenuation maps are invertible and preserve rank-r updates, their benefit stems from altering the optimization path, not just restricting the hypothesis space. It remains unclear if specific historical directions are essential for retention.

Questions for authors
  1. How much of the reported advantage remains when the strongest baseline is trained with RG-LoRA’s exact data, initialization, optimizer, and evaluation settings?

  2. How does a spectrum-matched history with randomized directions (or isotropic attenuation) perform compared to the learned history?

  3. What happens to performance if Equations 20–22 use standard differentiation instead of the identity-Jacobian surrogate?

2 comments

Thanks for your response. You have addressed most of my concerns.

Thank you for your time and constructive suggestions, which have helped us strengthen this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.