acceptodds
Under review as a conference paper at ICLR 2027

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

10 views

Abstract

Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present ChainLoRA, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Learning, Orthogonal Subspace, Chain Update

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qF7vOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ChainLoRA, a replay-free continual learning framework for large language models that formulates catastrophic forgetting from a task-vector geometry perspective. The method decomposes each LoRA update into a row-orthonormal carrier and corresponding magnitude coefficients, with the goal of separating transferable shared knowledge from task-specific variations. During continual training, ChainLoRA uses chain-updated initialization, a one-sided geometric regularization term, and effective-rank regularization to construct the task geometry while only relying on the most recent carrier. After training, an Adaptive SVD merging procedure estimates a shared carrier, aligns it with the latest task through Procrustes rotation, and combines the shared component with weighted task-specific residuals into a single adapter. The paper also provides theoretical analyses relating task-vector overlap to inter-task interference and motivating the proposed geometric regularization and alignment. Experiments on several continual learning benchmarks and multiple Llama models show competitive replay-free performance, particularly on longer task streams, together with ablations supporting the contribution of the main components.

Critical strengths
  • The paper targets a very interesting and meaningful research question on how to effectively conduct continual PEFT while avoiding forgetting.
  • The experiments are comprehensive.
Critical weaknesses

The main weakness I am concerned about is that this paper is hard to follow, in terms of its writing and motivation. In the writing, there are some notations or abbreviations that are used before being defined, leading to confusion. For the motivation, Sec.3 and 5 heavily depend on the heuristics without a clear explanation, in my opinion. This makes it hard to justify the reliability of the method and its challenges. I may misunderstand this paper, and am glad to raise my score if my concerns are addressed.

Questions for authors
  • In Fig.1, CL is used here without being defined first.
  • In Theorem 1, why does <ΔWi,ΔWj>F=0<\Delta W_i, \Delta W_j>_F=0 lead to task disentanglement? If we use ∑iΔWi\sum_i \Delta W_i as the merged parameters, <ΔWi,ΔWj>F=0<\Delta W_i, \Delta W_j>_F=0 does not imply there is no conflict in a forward pass.
  • In Definition 1, what is the superscript dd?
  • SO is not defined in Eq.2.
  • When computing P~0\tilde{P}_0, do we need to store PiP_i of all the historical tasks? This will increase the deployment and maintenance costs.
  • I am unclear about the purpose of doing Eq.2. Why do we need to align the common subspace with the latest one? How about the earlier ones?
  • Eq.3 does not make much sense to me. Why does Q−PQ-P represent the task-specific knowledge? As they are an orthonormal basis, I don't quite understand the meaning of subtracting them.
  • LoRA's right matrix AA is usually initialized as a Gaussian. Any specific reason for Kaiming initialization?
  • Line 301: Lora -> LoRA
  • What are the values of λ1\lambda_1 and λ2\lambda_2 used in experiments?
  • For LrankiL^i_{rank}, if we expect some directions to vanish, why not use ∣∣⋅∣∣2|| \cdot ||_2 rather than ∣∣⋅∣∣22||\cdot||_2^2?
Reviewer_1xX9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ChainLoRA, which uses chain-updated task-vector geometry as the proxy to use parameter-merging for continual learning. ChainLoRA separates LoRA weights into shared space and task-specific space for model merging to reduce task interference. It uses a regularization loss term to reduce the overlap between decomposed task vectors and initializes new LoRA by last carrier. Experiments show the effectiveness of ChainLoRA across models and benchmarks.

Critical strengths

The motivation of this paper is reasonable since inter-task interference and limited adaptation capacity are real issues in continual learning.

Critical weaknesses
  1. The proposed method, ChainLoRA, keeps all task vectors for merging them together, rather than using sequential merging methods, making it hard to be flexible under realistic settings when the number of sequential tasks is very large.

  2. Another concern is the novelty. The idea of making task vectors orthogonal to reduce task interference has already been used in model merging [1], and separating model weights into common space and task-specific space for model merging has also been used in [2]. The proposed ChainLoRA looks like a mixture of existing techniques, and the paper should clarify its own novelty.

    [1] “Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging”, NeurIPS 2025.

    [2] “No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces”, ICML 2025.

Questions for authors

Major:

  1. In the abstract, the authors mentioned "continual merging" in line 013 for introducing "We present ChainLoRA, a replay-free continual merging framework built on chain-updated task-vector geometry.", but in the algorithm, ChainLoRA stores all task vectors after all tasks (see step 14 in line 259), so it's concurrent merging to merge all task vectors at one time, rather than continual merging to merge task vectors one by one. Can authors explain this inconsistency?

  2. In Theorem 1, it mentions making PiPj⊤=0P_iP_j^{\top}=0 can achieve ⟨Δi,Δj⟩F=0\langle \Delta_i, \Delta_j\rangle_{\mathrm{F}}=0, but in Eq.(6), it tries to use loss term to minimize Pi−Pi−1P_i-P_{i-1}, but if this local change is zero, it means Pi=Pi−1P_i=P_{i-1} rather than PiPi−1⊤=0P_iP_{i-1}^{\top}=0. Can authors explain this?

  3. The motivation for designing each part of ChainLoRA is not clear. For example, why uses Procrutes to transform P~0\tilde{P}_0 to PTP_{T} and why not keep updating P0P_0 when learning a new task?

  4. The paper does not analyze the additional computation cost induced by ChainLoRA, since it introduces QR, SVD, Procrustes, and addtional loss terms, which need additional computation.

  5. For Theorem 1, why not using Wi(d)W_i^{(d)} and Wj(d)W_j^{(d)} as the condition for ∣⟨Δi,Δj⟩F∣|\langle \Delta_i, \Delta_j\rangle_{\mathrm{F}}|?

  6. In Algorithm 1, why use PtP_t to intialize At+1A_{t+1} and use 00 to initialize Bt+1B_{t+1}? Why not use Wt(d)W_{t}^{(d)} to initialize Bt+1B_{t+1}?

Minor:

  1. Can authors please remove the repeated “(3)” in line 123?

  2. It should be “O-LoRA” rather than “OLoRA” in lines 079 and 086.

Reviewer_2CAyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the merging of LoRA task vectors for continual learning in LLMs. It decomposes the update into a row-orthogonal carrier and a coefficient matrix, initializes the new task with the previous task's carrier, and trains with one-sided geometric regularization and rank regularization. Historical updates are then merged using SVD, Procrustes alignment and residual weighting. The paper attempts to analyze task interference through carrier overlap and coefficient coupling, and evaluates on Standard, Large, SuperNI, multiple model scales and task orders, supplemented by module and merging ablations.

Critical strengths
  1. Treating carrier geometry and coefficient coupling separately has analytical value. The Frobenius inner-product upper bound of Theorem 1 holds, and under row orthogonality and correct Procrustes alignment, Theorem 2 in Appendix C also has a reasonable mathematical basis. This provides a perspective for understanding the merging of low-rank updates that is worth further study, but it is not equivalent to having proven that historical task loss is controlled.
  2. The experimental scope and the disclosure of failure cases provide useful information. The paper covers Standard, Large, SuperNI, multiple models and task orders, and provides component ablations; it also discloses that SST-2 collapses to zero at the end of SuperNI and that the 13B model degrades. The Large/SuperNI results of 75.5/33.8 reported in Table 1 would constitute positive evidence if verified against consistent run records. As things stand, the current numerical conflicts keep these potential strengths from being sufficient to support acceptance.
Critical weaknesses
  1. The accuracy and BWT reported under the same experimental identifier are in direct contradiction, so the core performance claims cannot be confirmed. Table 5 (PDF p. 9) is labeled Standard CL O1; its 7B final four-task accuracies of 95.6, 60.0, 71.2 and 90.0 average to 79.2, whereas Tables 1 and 4 report 7B O1 as 80.5. Using the BWT definition on p. 7, the old-task BWT obtained from Table 5 is about −2.93, which agrees with Table 5's −2.9, but Table 6 (p. 17) reports −0.6; a similar contradiction appears for 3B, with about −2.33 versus −0.1. In addition, Table 6 labels a lower BWT as better, which is the opposite of the definition "final accuracy minus the accuracy when the task was learned". These discrepancies exceed rounding error and must be re-checked consistently with run IDs, checkpoints and per-stage matrices; until then, the performance and the degree of forgetting of the method cannot be judged reliably. In particular, if Table 5 turns out to be the correct record (7B O1 = 79.2, BWT = -2.9), the method falls below MagMax (80.1) on that order; and since the Standard-order average of 80.1 reported in Table 1 is already not higher than MagMax's 80.3, the claims of "best" performance would have to be withdrawn. This is the main basis for the current score.
  2. The rank regularization in the main text and in the appendix is not the same training objective, which affects the method definition and the mechanistic explanation. Section 5.1 (p. 6) uses the sum of squared column norms of the coefficient matrix W(d)W^{(d)}, i.e. a squared Frobenius magnitude regularizer, whereas Eq. (14) in Appendix D.1 applies a smoothed non-squared group penalty to the columns of the raw BB. Since W(d)=BR^⊤W^{(d)} = B \hat{R}^\top, the two act on different variables and do not share the same sparsity meaning. In particular, A→cA,  B→B/cA \to cA,\; B \to B/c leaves BABA unchanged yet reduces the BB penalty in the appendix, unless an additional scale constraint is imposed. Since the geometric loss, the LoRA scaling of the common component and the key hyperparameters also lack a unified statement, the existing ablations do not map uniquely onto a reproducible algorithm. This issue concerns the actual training and the core claim about effective rank, and cannot be treated merely as a writing slip.
Questions for authors
  1. What is the true final average accuracy of 7B O1, 80.5 or 79.2, and is BWT −0.6 or about −2.9? Why is the 3B BWT also inconsistent? Please state which tables come from the same run, the same task order and the same checkpoint, and correct the reported direction of what counts as better BWT.
  2. Which rank regularizer and geometric loss does the actual code use? Please give the exact formulas, regularization weights, rank, target modules and LoRA scaling, and state whether a constraint exists that rules out the A/B rescaling that evades the raw-BB penalty. Do the discrepancies between the main text and the appendix change any of the reported results?
  3. Can you provide the per-stage accuracy matrices of the runs above, the result aggregation scripts, and ablations consistent with the actual training objective, and regenerate the main tables and the BWT tables?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.