- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RG-LoRA, which is a residual geometry-guided continual adaptation method. RG-LoRA utilizes historical residual geometry to induce task-fixed soft attenuation around new LoRA updates along strongly represented directions, and merges the learned residual into the adapted model after learning each task. RG-LoRA requires no replay and no growing LoRAs, which can save storage cost. Experimental results show the effectiveness of RG-LoRA on T5-Large and Llama models.
Critical strengths
The paper proposes a novel method for continual learning, using historical input and output residual directions to induce task-fixed soft attenuation around a new LoRA update. This idea is novel in LoRA-based continual learning and gives insights into considering the relationship between previous knowledge and new knowledge.
Critical weaknesses
-
The experimental results need more analysis on the components of RG-LoRA, since current results focus on the performance of RG-LoRA across models and tasks, but it does not show how each component of RG-LoRA itself contributes to the final results.
-
The quality of the paper needs to be improved, such as the explanation of the method. For example, in section 4.2, the title is “task-fixed soft attenuation”, but it only shows a bunch of equations without a coherent story connecting all equations. I think the (b) part in Figure 1 is clear in explaining the purpose of each component, but there is no such clear explanation in Section 4.2.
Questions for authors
-
Can authors explain why RG-LoRA defines Eq.(16) as such? Is there any derivation, reference, or evidence?
-
Why are Eq.(20) and Eq.(22) equal? How to obtain Eq.(22) from Eq.(21) and Eq.(20)? Also, why do you use Eq.(21)? Since current explanations in Section 4.2 lack a connection between individual parts, explaining the relationship between these equations makes it better.
-
There are not enough ablation studies to analyze the components of RG-LoRA. For example, the storage cost and computation cost of RG-LoRA since it involves two additional and , or analyzing the effect of and by removing either of them.
-
Can authors evaluate the performance of RG-LoRA on Qwen model?
-
Table 1 and Table 2 overlap in the results of RG-LoRA on T5-large, Llama-2-7B-Chat, and Llama-3.2-3B. It should use Table 2 to show new results on ablation studies.
4 comments
Clarification on the output-side contribution
Thank you for clarifying this concern. We apologize for not presenting the additional comparison in Table 4 (p. 10) directly in our earlier reply.
The example in Appendix E.3, Eq. (80) explains the different roles of the two sides. Input attenuation acts on feature directions, whereas output attenuation acts on layer-output directions. An old and a novel input can share an output direction. Input attenuation can protect the old input while leaving the novel one unchanged, whereas output-only attenuation affects both. This motivates using output-side attenuation as complementary control, rather than as a replacement for input-side protection.
Table 2 reports final scores. To examine what the output map adds, Table 4 measures changes in old-task predictions that may not change the decoded answer. Before each merge, both maps act on the same Full-trained proposal. Input-only is scaled to match Full's total residual norm across adapted layers, controlling for global update magnitude.
The two metrics measure different aspects of preservation:
- Reference NLL measures the negative log-probability of an old-task reference answer. Lower NLL means the model assigns more probability to that answer.
- Predictive KL measures the change from the pre-task model's next-token distribution. This reference has the current residual disabled and all earlier merged updates retained. We compare full vocabulary distributions at identical reference-answer prefixes. Lower KL means less predictive drift, not necessarily higher accuracy.
We average over answer tokens within each example, then over examples, excluding prompt tokens. The table summarizes evaluation-only old-task probes across task boundaries. KL is reported in units of 10⁻³.
| Setting | Input-only (norm-matched) NLL ↓ | Full NLL ↓ | Input-only (norm-matched) KL ↓ | Full KL ↓ |
|---|---|---|---|---|
| CL O1 | 0.1648 | 0.1645 | 0.661 | 0.675 |
| CL O4 | 0.2532 | 0.2514 | 14.212 | 9.794 |
| SuperNI O1 | 1.1074 | 1.1062 | 20.033 | 16.607 |
Full reduces old-task predictive KL by approximately 31% on CL O4 and 17% on SuperNI O1, with slightly lower NLL on all three settings. CL O1 has slightly higher KL under Full. NLL complements KL by checking reference-answer probability as well as distributional change. Together, these measurements provide additional evidence supporting the output map in Full RG-LoRA beyond global norm reduction. The complete protocol and results are in Appendix F.4, Table 23 (p. 30).
We hope the clarification above and the evidence in Table 4 address your remaining concern about the output-side contribution. We would appreciate your reconsideration.
As the discussion period is coming to an end, we would greatly appreciate your feedback on our additional clarification.
Thank you again for your time and consideration.
Thank authors for responding to the rebuttal and appreciating the added experimental results. The responses addressed part of my concerns. I think from the table for Q3, the input-only variant seems identical to Full RG-LoRA, and I think this is an important comparison for establishing the contributions of RG-LoRA, but the current results do not show the evidence. I would like to keep my original score.