- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes ChainLoRA, a replay-free continual learning framework for large language models that formulates catastrophic forgetting from a task-vector geometry perspective. The method decomposes each LoRA update into a row-orthonormal carrier and corresponding magnitude coefficients, with the goal of separating transferable shared knowledge from task-specific variations. During continual training, ChainLoRA uses chain-updated initialization, a one-sided geometric regularization term, and effective-rank regularization to construct the task geometry while only relying on the most recent carrier. After training, an Adaptive SVD merging procedure estimates a shared carrier, aligns it with the latest task through Procrustes rotation, and combines the shared component with weighted task-specific residuals into a single adapter. The paper also provides theoretical analyses relating task-vector overlap to inter-task interference and motivating the proposed geometric regularization and alignment. Experiments on several continual learning benchmarks and multiple Llama models show competitive replay-free performance, particularly on longer task streams, together with ablations supporting the contribution of the main components.
Critical strengths
- The paper targets a very interesting and meaningful research question on how to effectively conduct continual PEFT while avoiding forgetting.
- The experiments are comprehensive.
Critical weaknesses
The main weakness I am concerned about is that this paper is hard to follow, in terms of its writing and motivation. In the writing, there are some notations or abbreviations that are used before being defined, leading to confusion. For the motivation, Sec.3 and 5 heavily depend on the heuristics without a clear explanation, in my opinion. This makes it hard to justify the reliability of the method and its challenges. I may misunderstand this paper, and am glad to raise my score if my concerns are addressed.
Questions for authors
- In Fig.1, CL is used here without being defined first.
- In Theorem 1, why does lead to task disentanglement? If we use as the merged parameters, does not imply there is no conflict in a forward pass.
- In Definition 1, what is the superscript ?
- SO is not defined in Eq.2.
- When computing , do we need to store of all the historical tasks? This will increase the deployment and maintenance costs.
- I am unclear about the purpose of doing Eq.2. Why do we need to align the common subspace with the latest one? How about the earlier ones?
- Eq.3 does not make much sense to me. Why does represent the task-specific knowledge? As they are an orthonormal basis, I don't quite understand the meaning of subtracting them.
- LoRA's right matrix is usually initialized as a Gaussian. Any specific reason for Kaiming initialization?
- Line 301: Lora -> LoRA
- What are the values of and used in experiments?
- For , if we expect some directions to vanish, why not use rather than ?