acceptodds
Under review as a conference paper at ICLR 2027

McLoRA: Multiplicative Composition for LoRA-Based Continual Learning

20 views

Abstract

Low-Rank Adaptation (LoRA) suffers from catastrophic forgetting in continual learning. Existing methods for addressing this problem still retain the standard additive composition, in which the LoRA update does not receive direct guidance from the backbone output. This paper proposes McLoRA (Multiplicative Composition LoRA), which replaces the standard additive composition with element-wise multiplicative composition between the low-rank branch and the frozen backbone output. This design makes the learning gradient of each output dimension in the low-rank branch explicitly correlated with the output magnitude of the corresponding backbone dimension, thereby encouraging updates to focus on dimensions that respond more strongly to the current task while suppressing updates on weakly activated dimensions. Empirically, McLoRA produces more concentrated updates with less cross-task interference. Experiments on multiple large language models and sequential learning benchmarks show that, under the same parameter budget and computational complexity as LoRA, McLoRA significantly alleviates catastrophic forgetting when a single shared LoRA module is trained sequentially, while maintaining comparable single-task performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Low-Rank Adaptation, Multiplicative Composition, Continual Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fnedOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

McLoRA replaces LoRA’s additive correction, Wx + BAx, with the elementwise product (Wx) ⊙ (1 + BAx), making the low-rank adapter gradients explicitly depend on the frozen backbone output. The authors argue that this directs updates toward dimensions that are more active for the current task and thereby reduces interference with earlier tasks. They provide a gradient derivation, activation interventions, and analyses of update concentration and cross-task overlap. Experiments across several backbones and sequential-learning streams generally show higher final performance and less forgetting than standard LoRA. The paper also studies a bounded variant, extensions to other LoRA methods, and single-task performance.

Critical strengths
  • Simple and well-specified method. McLoRA changes the composition rule without increasing LoRA’s trainable-parameter budget. The gradient derivation clearly identifies the effect of the backbone output on adapter updates.

  • Substantial empirical results. In Table 2, McLoRA improves final average performance in 27 of 28 backbone–sequence settings and reduces forgetting in 27 of 28. Longer task streams and settings without explicit task boundaries extend the evaluation.

  • Useful analyses beyond final scores. The activation interventions and update statistics help characterize the method’s behavior. Experiments with the bounded variant and other LoRA methods provide additional context.

Critical weaknesses
  • The central anti-forgetting motivation remains insufficiently justified. The paper provides evidence that dimensions with large backbone outputs influence current-task predictions, but does not establish why directing more updates to those dimensions should preserve previous tasks. Successive tasks may rely on the same strongly activated dimensions, in which case concentrating updates there could still create interference. The reported overlap statistics are supportive observations, but do not resolve when this update-allocation principle should work or fail.

  • The specific choice of multiplication is not adequately isolated. Appendix C derives the multiplicative form after imposing the desired Wx factor in the gradient; it does not independently justify that factor as a continual-learning principle. The motivation emphasizes activation magnitude, whereas the method uses the signed, unnormalized Wx. Multiplication also changes the forward function. Without controlled alternatives, it remains unclear which of these changes accounts for the performance gains.

  • Comparative and robustness evidence is incomplete. The main shared-adapter experiments do not provide matched comparisons with the specialized continual-learning methods discussed in the related work. The S1–S7 experiments use task-specific classification heads, although the longer shared-head streams partly address this limitation. The result tables also lack run-to-run variability, making close comparisons difficult to assess.

Questions for authors
  • How does McLoRA perform when successive tasks rely on substantially overlapping high-activation dimensions? Can the authors measure whether new-task updates affect dimensions important to earlier tasks and directly relate this to changes in old-task loss?

  • Under matched parameter counts and update scales, how do normalized or magnitude-based activation gates, shuffled activation signals, and activation-weighted additive updates compare with the proposed multiplication?

  • Can the authors report variability across random seeds and compare against specialized shared-adapter continual-learning methods under the same task stream and classification-head protocol?

6 comments

Response 1 (Weakness 1; Question 1): The anti-forgetting mechanism, the overlapping case, and old-task loss

We thank the reviewer for this comment. We agree that concentrating updates on the high-activation dimensions of the current task does not by itself guarantee the protection of old tasks. We provide a conditional account of the mechanism and conduct experiments for both parts of the question.

Mechanism: (1) Figure 1 and Appendix A of the paper show that dimensions with larger backbone output magnitudes tend to exhibit higher sensitivity to the current task. (2) The high-activation dimensions of different tasks partially overlap but are not identical. (3) McLoRA modulates according to the backbone output. Consider one gradient-descent step on B with a new-task input xx (learning rate η\eta); the first-order change of the old-task loss on an old-task input x′x' is:

  • LoRA: ΔLold≈−η⟨Ax,Ax′⟩∑iδinew(x)δiold(x′)\Delta L_{\mathrm{old}}\approx-\eta\langle Ax,Ax'\rangle\sum_i\delta_i^{\mathrm{new}}(x)\delta_i^{\mathrm{old}}(x')
  • McLoRA: ΔLold≈−η⟨Ax,Ax′⟩∑iδinew(x)δiold(x′)ui(x)ui(x′)\Delta L_{\mathrm{old}}\approx-\eta\langle Ax,Ax'\rangle\sum_i\delta_i^{\mathrm{new}}(x)\delta_i^{\mathrm{old}}(x')u_i(x)u_i(x')

where δ=∂L/∂h\delta=\partial L/\partial h and u=Wxu=Wx is the backbone output (same notation as Appendix C). Of the additional factors, ui(x)u_i(x) comes from the new-task training side, i.e., the learning signal is scaled by the backbone output of the new task; ui(x′)u_i(x') comes from the old-task evaluation side, i.e., the effect of the adapter change on the old-task output is scaled by the backbone output of the old task. (4) Therefore, as long as either the new or the old task is weakly activated in a dimension, the interference in that dimension is reduced accordingly; when both are strongly activated, it is not suppressed, and forgetting still occurs if the update directions conflict. (5) This predicts that the more the high-activation dimensions overlap, the smaller McLoRA's advantage. The experiments below test this prediction.

The overlapping case: On 24 "old task → new task" directions formed from 7 tasks (RoBERTa-large), we compute the overlap OO between the new and old tasks from the top 20% dimensions in backbone output of each task on the frozen backbone (Jaccard, averaged over modules; chance level ≈ 0.11), and divide the directions into three groups by OO:

Table R10 | Forgetting grouped by the overlap of high-activation dimensions (LoRA / McLoRA)

Overlap groupRange of OOOld-task accuracy dropOld-task loss increase
Relatively low0.410–0.4524.19 / 1.040.150 / 0.030
Middle0.481–0.5375.03 / 3.890.110 / 0.050
Relatively high0.565–0.6142.44 / 2.100.071 / 0.065
All 24 directions0.410–0.6143.89 / 2.340.110 / 0.048

McLoRA reduces forgetting overall, but its advantage shrinks as the overlap increases, and the two are close in the highest-overlap group, consistent with the above analysis.

Effect on dimensions important to old tasks and on old-task loss: On the same training runs, we compute the first-order increase of the old-task loss, E[gold⊤Δhold]\mathbb{E}[g_{\mathrm{old}}^\top\Delta h_{\mathrm{old}}], from the actual output change before and after training on the new task; the first-order prediction accounts for about 85% of the measured loss increase. We divide the dimensions into four groups according to whether they are highly activated for the new and the old task:

Table R11 | Dimension-wise decomposition of the first-order increase of the old-task loss

Dimension groupLoRAMcLoRAReduction
High activation for both tasks0.02360.020911%
High activation for the old task only0.00380.000393%
High activation for the new task only0.00410.000880%
High activation for neither task0.06240.019269%
Total0.09400.041256%

The reduction in interference occurs mainly in dimensions where at least one side is weakly activated; in dimensions highly activated for both tasks, it is reduced by only 11%, consistent with the above analysis. We also directly examine the dimensions important to the old task: selecting the old-task-important dimensions by the sensitivity of the old-task loss to each dimension's output (the same set for both methods), the increase in old-task loss caused by McLoRA on these dimensions is 26% smaller than that caused by LoRA; on the part of these dimensions not needed by the new task, it is 63% smaller.

Revision: The revised version will rephrase the mechanism in the above conditional form and include the above analyses.

Response 2 (Weakness 2; Question 2): Design rationale and controlled alternatives

We thank the reviewer for this comment. We address the points one by one:

  • Basis of the Wx factor. Figure 1 and Appendix A show that dimensions with larger backbone output magnitudes tend to exhibit higher sensitivity to the current task; we therefore want the backbone output to enter the gradient and guide the allocation of updates across dimensions. Appendix C proves that, among element-wise compositions, multiplication is the simplest mechanism that makes the gradient contain this factor. Why this factor benefits continual learning is explained by the first-order forgetting analysis in Response 1: it multiplies the interference between the new and old tasks in each dimension by the product of their backbone outputs, so the interference is suppressed as long as one side is weakly activated.
  • Signed and unnormalized. Keeping the sign is required by multiplicative modulation itself: if ∣Wx∣\lvert Wx\rvert were used instead, positive and negative responses would be scaled in opposite directions by the same BAx; in Table R12 below, the magnitude gate is also clearly worse than McLoRA. Per-token RMS normalization only changes the overall scale of the gate at each token, not the relative allocation across dimensions within the same token; in Table R12, the RMS-normalized gate is close to McLoRA.

Controlled alternatives: McLoRA can be written as h=Wx+Wx⊙BAxh=Wx+Wx\odot BAx. All variants use the same forward form h=Wx+g(x)⊙BAxh=Wx+g(x)\odot BAx and only replace the gate g(x)g(x); they have the same number of parameters as McLoRA, and the update scale is matched per layer. The experiments follow the protocol of Table 2 of the paper (S2, RoBERTa-large).

Table R12 | Controlled alternatives

MethodGate g(x)g(x)Final Avg ↑Forgetting ↓
LoRA173.122.4
McLoRAWxWx81.811.8
Shuffled activationWxWx with randomly permuted dimensions73.621.8
Magnitude gate∣Wx∣\lvert Wx\rvert76.718.8
Static activation weightingEx∣Wx∣\mathbb{E}_x\lvert Wx\rvert computed at the start of each task73.421.9
RMS-normalized gateWxWx with per-token RMS normalization81.612.1

Conclusion: (1) Shuffled activation is close to LoRA: when the distribution of gate values is kept and only the correspondence of dimensions is shuffled, the gain almost disappears, indicating that the gain relies on modulating each dimension by its own backbone output. (2) Static activation weighting is also close to LoRA: a fixed per-dimension weighting cannot replace an input-dependent gate. (3) The magnitude gate is clearly worse than McLoRA, indicating that the sign matters. (4) The RMS-normalized gate is close to McLoRA, indicating that the gain comes from the relative allocation across dimensions within the same token, rather than from the overall scale of the gate.

Revision: The revised version will add the first-order forgetting analysis of Response 1 to the main text as the continual-learning basis of this factor, and include the controlled alternatives in Table R12.

Response 3 (Weakness 3; Question 3): Specialized continual-learning baselines, classification heads, and multiple random seeds

We thank the reviewer for the suggestion. The LoRA-based continual-learning methods discussed in Section 2 of the paper do not match our NLP setting with a fixed budget and a single shared LoRA: O-LoRA, N-LoRA, AM-LoRA, and LB-CL introduce new LoRA modules or low-rank parameters for each task (these four have been compared under the multi-adapter protocol in Table 6 of the paper); BranchLoRA targets multimodal large models and adds a new router for each task; SD-LoRA, CL-LoRA, and C-LoRA target class-incremental learning in vision; Share stores a set of coefficients for each task. Therefore, we chose EWC and LwF, which can be applied directly to the same shared adapter without adding trainable parameters or replaying data, compared them under the same task stream and classification-head protocol as the main experiments, and report the mean ± standard deviation over 3 random seeds (S2, RoBERTa-large; λ and α selected on the validation set).

Table R13 | Specialized continual-learning baselines and multi-seed results

MethodFinal Avg ↑Forgetting ↓
LoRA76.7 ± 3.618.2 ± 4.2
McLoRA83.2 ± 1.79.3 ± 2.8
LoRA + EWC84.1 ± 2.47.3 ± 2.2
LoRA + LwF83.8 ± 2.97.7 ± 2.7
McLoRA + EWC89.4 ± 1.41.1 ± 0.3
McLoRA + LwF88.3 ± 2.12.3 ± 0.7

Conclusion: (1) The Final Avg of McLoRA is on average 6.5 points higher than that of LoRA, and its forgetting is on average 8.9 points lower, with smaller variation across seeds (standard deviation 1.7 vs. 3.6). (2) Used alone, McLoRA is comparable to LoRA + EWC and LoRA + LwF, without requiring an additional regularization term, Fisher estimation, or a teacher model; the two can be combined, and McLoRA + EWC performs best. (3) Regarding classification heads: S1–S7 follow the setting of related work and use task-specific classification heads; the long sequences LS1–LS4 and their boundary-free variants all use the shared language-modeling head, and McLoRA outperforms LoRA in all these settings (Table 3 of the paper).

Revision: The revised version will include the above multi-seed results and baseline comparisons.

Reviewer_JjXTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel multiplicative composition method, McLoRA, for LoRA-based continual learning. It utilizes the frozen backbone output to element-wise multiply with the low-rank branch output, instead of standard additive composition. McLoRA makes the gradient in low-rank branch correlated with the output magnitude of backbone, to produce more concentrated updates with less interference across tasks. Experimental results across different base models and benchmarks show that McLoRA can mitigate forgetting while maintaining performance in single-task fine-tuning.

Critical strengths

The paper proposes a new paradigm which multiplies low-rank branch output with frozen backbone output to replace standard additive composition in LoRA for continual learning. The idea of designing a new paradigm for LoRA in continual learning is interesting to me, since previous works mostly follow the standard additive composition.

Critical weaknesses
  1. It’s not clear why McLoRA can reduce forgetting in continual learning based on the current analysis in the paper. There is no forgetting error analysis in theory or principle, for example, analyzing the forgetting error reduced by using McLoRA. Table 1 shows more concentrated gradient updates and less overlap across tasks in major updates, but the experiments only use two tasks. It’s not solid to give such a strong conclusion that McLoRA mitigates forgetting in the paper.

  2. For the design of McLoRA, it’s not clear why using WxWx rather than WW, and there is no such analysis. The motivation for the design of McLoRA is not clear.

Questions for authors
  1. Can authors explain why there is 1\boldsymbol{1} in Eq.(1)? What’s the purpose of 1\boldsymbol{1} here?

  2. Can authors explain why using W0xW_{0}x rather than W0W_{0} in McLoRA?

  3. Can authors compare McLoRA with LoRMA [1]? Since LoRMA also uses similar multiplying LoRA with backbone, it would be better to explain what benefits are achieved by McLoRA over LoRMA.

    [1] “LoRMA: Low-Rank Multiplicative Adaptation for LLMs", Findings of the Association for Computational Linguistics: ACL 2025.

  4. Section 3.2 only shows the results on the gradient on BB, but what are the results on AA? Since LoRA components have assymetric property [2], we cannot simply extend the results from BB to AA.

    [2] “Asymmetry in Low-Rank Adapters of Foundation Models”, ICML2024.

  5. It would be better to show more results on more tasks in Table 1 and Figure 3, since only two task results are very weak for obtaining a strong conclusion.

7 comments

Response 1 (Weakness 1, part 1): Why McLoRA reduces forgetting

We thank the reviewer for this comment. We add a first-order analysis starting from the old-task loss and verify it experimentally.

Mechanism: Consider one gradient-descent step on B with a new-task input xx (learning rate η\eta). The first-order change of the old-task loss on an old-task input x′x' (i.e., the forgetting error) is:

  • LoRA: ΔLold≈−η⟨Ax,Ax′⟩∑iδinew(x)δiold(x′)\Delta L_{\mathrm{old}}\approx-\eta\langle Ax,Ax'\rangle\sum_i\delta_i^{\mathrm{new}}(x)\delta_i^{\mathrm{old}}(x')
  • McLoRA: ΔLold≈−η⟨Ax,Ax′⟩∑iδinew(x)δiold(x′)ui(x)ui(x′)\Delta L_{\mathrm{old}}\approx-\eta\langle Ax,Ax'\rangle\sum_i\delta_i^{\mathrm{new}}(x)\delta_i^{\mathrm{old}}(x')u_i(x)u_i(x')

where δ=∂L/∂h\delta=\partial L/\partial h and u=Wxu=Wx is the backbone output (same notation as Appendix C). McLoRA multiplies the interference term in each dimension by the product of the backbone outputs of the new and old tasks in that dimension, so the smaller the backbone output, the more the interference in that dimension is suppressed: as long as one side is weakly activated, the interference is reduced accordingly; when both sides are strongly activated, it is not suppressed. We verify this in actual training below.

Experiment: On 24 "old task → new task" directions formed from 7 tasks (RoBERTa-large), we compute the first-order increase of the old-task loss, E[gold⊤Δhold]\mathbb{E}[g_{\mathrm{old}}^\top\Delta h_{\mathrm{old}}], from the actual output change before and after training on the new task, and decompose it into four groups of dimensions (defined by the top 20% dimensions in backbone output for the new and the old task, respectively):

Table R4 | Dimension-wise decomposition of the first-order increase of the old-task loss

Dimension groupLoRAMcLoRAReduction
High activation for both tasks0.02360.020911%
High activation for the old task only0.00380.000393%
High activation for the new task only0.00410.000880%
High activation for neither task0.06240.019269%
Total0.09400.041256%

Conclusion: McLoRA reduces the forgetting error (the first-order increase of the old-task loss) by 56%, and the first-order prediction accounts for about 85% of the measured loss increase; the reduction occurs mainly in dimensions where at least one side is weakly activated, consistent with the above analysis; in dimensions where both sides are strongly activated, the improvement is limited. Correspondingly, McLoRA's advantage shrinks as the overlap of high-activation dimensions between the new and old tasks increases: dividing the directions into three groups by overlap, the improvements in accuracy drop are 3.15, 1.14, and 0.34 percentage points, respectively; averaged over the 24 directions, the accuracy drop decreases from 3.89 to 2.34.

Revision: In the revised version, we will accordingly rephrase the mechanism in the above conditional form.

Response 2 (Weakness 1, part 2; Question 5): Results on more tasks

We thank the reviewer for the suggestion. Appendix E of the paper already extends Figure 3(a) to more settings (Section 3.2 refers to this appendix): Figure 5 shows the results of RoBERTa-large on QQP and QNLI, and Figure 6 shows the results of LLaMA-2-7B on MNLI and SST-2. We further provide quantitative results for three settings with all metrics of the original Table 1 (Tables R5–R7), which, together with RoBERTa-large / MNLI–SST-2 in the main text, form a "two backbones × two task pairs" comparison. The setup is the same as in the mechanism experiments of the paper.

Table R5 | RoBERTa-large, QQP / QNLI

MethodTaskGini ↑Norm. Entropy ↓Top-1% Energy ↑Top-5% Energy ↑k80k_{80} ↓rlowSr_{\mathrm{lowS}} ↓Support Overlap ↓Support Jaccard ↓Energy on old_unique ↓
LoRAQQP0.33240.96526.08%17.97%617.117.00%508.50.6825.86%
LoRAQNLI0.30590.96905.87%17.16%639.317.35%508.50.6826.07%
McLoRAQQP0.60300.849323.71%43.13%382.99.33%281.90.4774.03%
McLoRAQNLI0.51940.892118.42%36.20%476.410.15%281.90.4775.87%

Table R6 | LLaMA-2-7B, MNLI / SST-2

MethodTaskGini ↑Norm. Entropy ↓Top-1% Energy ↑Top-5% Energy ↑k80k_{80} ↓rlowSr_{\mathrm{lowS}} ↓Support Overlap ↓Support Jaccard ↓Energy on old_unique ↓
LoRAMNLI0.28060.97615.26%15.45%2575.216.85%1907.30.6139.25%
LoRASST-20.32610.96866.25%17.65%2435.216.82%1907.30.6137.89%
McLoRAMNLI0.46080.931012.49%28.20%1984.512.67%1319.50.5267.49%
McLoRASST-20.50490.918414.29%31.09%1816.312.18%1319.50.5265.72%

Table R7 | LLaMA-2-7B, QQP / QNLI

MethodTaskGini ↑Norm. Entropy ↓Top-1% Energy ↑Top-5% Energy ↑k80k_{80} ↓rlowSr_{\mathrm{lowS}} ↓Support Overlap ↓Support Jaccard ↓Energy on old_unique ↓
LoRAQQP0.32690.96876.22%17.74%2431.016.50%1956.00.6607.21%
LoRAQNLI0.30220.97305.59%16.43%2509.416.54%1956.00.6607.62%
McLoRAQQP0.50780.916914.48%31.62%1805.111.57%1322.70.5555.54%
McLoRAQNLI0.48440.925813.06%29.67%1900.712.38%1322.70.5556.21%

Conclusion: All three settings are consistent with the original Table 1: McLoRA's gradients are more concentrated (higher Gini, smaller k80k_{80}), a lower fraction falls on weakly activated dimensions (lower rlowSr_{\mathrm{lowS}}), the overlap of the major update supports across tasks is smaller (lower Support Overlap and Jaccard), and the energy falling into the critical dimensions unique to the old task is also lower.

Revision: The revised version will add Tables R5–R7 to Appendix E and provide the S–D heatmaps for the new settings.

Response 3 (Weakness 2; Question 2): Why Wx rather than W

This design comes directly from the starting point of the paper. As stated in Section 1, dimensions with larger backbone output magnitudes tend to exhibit higher sensitivity to the current task (Figure 1, Appendix A). We therefore want the update of the ii-th dimension to be guided by its own response to the current input, i.e., the gradient of the ii-th row of the low-rank branch should contain the factor (Wx)i(Wx)_i. This per-dimension factor arises only when the output WxWx is modulated: in McLoRA, this gradient is δi(Wx)i(Ax)⊤\delta_i(Wx)_i(Ax)^\top; if the weights were modulated instead, e.g., h=[W⊙(1+BA)]xh=[W\odot(1+BA)]x, the gradient would be scaled by the individual weight entries WijW_{ij}, which are the same for all inputs and tasks and do not reflect the actual response (Wx)i=∑jWijxj(Wx)_i=\sum_j W_{ij}x_j, since its terms may cancel each other. Appendix C gives the rigorous statement: among forms that use element-wise composition, introduce no additional parameters, and preserve the backbone output when the low-rank signal is zero, Wx⊙(1+BAx)Wx\odot(1+BAx) is the simplest form whose gradient for the ii-th dimension contains exactly (Wx)i(Wx)_i.

Revision: We will move this argument from Appendix C to Section 3.1, so that the design motivation is directly visible in the main text.

Response 4 (Question 1): The role of the constant 1 in Eq. (1)

The 1 here is an all-ones vector, not a trainable parameter. Since B is zero-initialized, g(x)=1g(x)=1 at initialization and the layer output is h=Wxh=Wx, identical to the output without the adapter, so the model starts training from the pretrained behavior; without the 1, the layer output would become zero under zero initialization. Section 3.1 states the zero initialization of B after Eq. (1), and lines 176–178 state that McLoRA belongs to the class of compositions that "preserve the backbone output when the low-rank update is zero"; the constant 1 is exactly the term that makes this condition hold.

Revision: The revised version will state this directly after Eq. (1).

Response 5 (Question 3): Comparison with LoRMA

We thank the reviewer for the suggestion. The difference between the two lies in where the multiplication acts. LoRMA performs multiplicative adaptation in weight space. Taking LoRMA₊ as an example, h=(I+BA)Wxh=(I+BA)Wx, and the gradient of the ii-th row of B is δi(AWx)⊤\delta_i(AWx)^\top: WxWx, as the input to the low-rank branch, is mixed across dimensions by A, and the differences among the row gradients come only from δi\delta_i, which is the same as in LoRA in this respect. The gradient of McLoRA is δi(Wx)i(Ax)⊤\delta_i(Wx)_i(Ax)^\top, which is modulated by each dimension's own backbone output (Section 2 and Appendix Table 8 of the paper).

Experiment: On S2 with RoBERTa-large, the three methods use the same rank and the same number of trainable parameters, following the hyperparameters in Table 13; the implementation of LoRMA₊ follows the official code.

Table R8 | Comparison with LoRMA (S2, RoBERTa-large)

MethodFinal Avg ↑Forgetting ↓
LoRA73.122.4
LoRMA₊51.348.6
McLoRA81.811.8

Note: LoRA and McLoRA are taken from Table 2 of the paper; LoRMA₊ is run under the same protocol.

Conclusion: LoRMA₊ learns new tasks normally, but its forgetting is far higher than that of both. Although it is also a multiplicative composition, multiplication in weight space does not mitigate forgetting; McLoRA's advantage comes from its output-space, per-dimension modulation design, rather than from multiplicative composition itself.

Revision: The revised version will add the comparison in Table R8 to the appendix.

Response 6 (Question 4): Results on A

Section 3.2 uses B as an example because each row of B corresponds to an output dimension, whereas the rows of A correspond to rank components and have no direct output-dimension index. The work cited by the reviewer (Zhu et al., 2024) also points out that B maps features to outputs and has a larger impact on adaptation. To directly address A, we map the gradient of A to the output dimensions through B and compute the same metrics as for B (RoBERTa-large; 4 task orders: MNLI↔SST-2 and QQP↔QNLI). Gini measures concentration; Spearman(U, S) measures whether the allocation favors dimensions with large backbone output magnitude S; rlowSr_{\mathrm{lowS}} is the fraction of energy falling on the 20% of dimensions with the lowest S, which is about 20% under a uniform distribution.

Table R9 | Allocation of the gradients of A and B over output dimensions (each cell: LoRA / McLoRA)

Measured quantityGiniSpearman(U, S)rlowSr_{\mathrm{lowS}}
Gradient of B0.593 / 0.8570.160 / 0.38913.5% / 4.1%
Gradient of A (mapped through B)0.573 / 0.588−0.020 / 0.10620.5% / 19.1%

Conclusion: After being mapped through B, the gradient of A shows only weak alignment, much weaker than that of B. That is, McLoRA allocates updates across output dimensions mainly through B. This is consistent with the findings of Zhu et al., and also with the form of the gradient of A, B⊤(δ⊙Wx)x⊤B^\top(\delta\odot Wx)x^\top, in which WxWx is aggregated across dimensions by B⊤B^\top. The overall effect of A and B together still favors dimensions with large backbone outputs: during training, the Spearman correlation between the actual change of BA and the backbone output magnitude is 0.212 (−0.039 for LoRA), and the energy falling on the 20% of dimensions with the lowest backbone output is 15.1% (20.5% for LoRA).

Revision: The revised version will add the statistics of A (Table R9) to the appendix.

Reviewer_mEFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

McLoRA replaces LoRA’s additive update with input-dependent multiplicative modulation of backbone outputs. The paper evaluates this change across several models and continual-learning benchmarks, including long task sequences and settings without explicit task boundaries.

Critical strengths
  • The method is simple and maintains a fixed adapter budget. Table 2 shows better final accuracy and lower forgetting in 27 of 28 settings, making the empirical improvement convincing.
  • The gains extend to long task streams, boundary-free learning, and AdaLoRA/DoRA variants. These experiments support the usefulness of the proposed modification.
Critical weaknesses
  • The mechanism analysis measures raw gradients of B\mathbf{B}, rather than actual AdamW updates or the combined effect of updating A\mathbf{A} and B\mathbf{B}. Moreover, Appendix F defines old-task-critical dimensions differently for the two methods. These diagnostics do not yet establish that McLoRA causes less interference on a common set of important dimensions.

  • The main single-adapter experiments lack a dedicated continual-learning baseline under a comparable parameter budget. Results also lack variability across random seeds, making smaller improvements, such as the 0.5-point advantage in Table 6, difficult to assess.

Questions for authors
  • Does the concentration pattern persist when measuring actual AdamW parameter updates, including both A\mathbf{A} and B\mathbf{B}?
  • Does reduced interference remain evident when both methods use the same definition of old-task-critical dimensions?
  • Can the authors provide a matched-budget continual-learning comparison and multi-seed results for representative settings?
5 comments

Response 1 (Weakness 1, part 1; Question 1): Actual AdamW updates and the joint effect of A and B

We thank the reviewer for this comment. We agree that Section 3.2 is based on the raw gradients of B, and that under AdamW the gradient is not equivalent to the actual update. We therefore recorded the actual AdamW updates at every training step, and computed statistics for the actual update of B as well as for the joint update of A and B, Δ(BA)\Delta(BA) (which includes the changes of both and the cross term). The results are shown in Table R1.

Setup: RoBERTa-large; the training scale is the same as in the analysis of Section 3.2 (2,000 training examples and 1,000 steps per task), and the tasks are extended to 4 task orders: MNLI↔SST-2 and QQP↔QNLI. To compare with the actual updates, the gradient row directly uses the raw δ\delta, without the normalization that fixes the norm of δ\delta in the paper; no weight decay is used in training, so that the actual update is obtained entirely from the gradient through the Adam transformation and can be compared with the gradient directly. The statistics follow Appendix F: Gini measures concentration; Spearman(U, S) measures whether updates favor dimensions with large backbone output magnitude S; rlowSr_{\mathrm{lowS}} is the fraction of energy falling on the 20% of dimensions with the lowest S, which is about 20% under a uniform distribution.

Table R1 | Dimension-wise allocation statistics under actual AdamW updates (each cell: LoRA / McLoRA)

Measured quantityGini ↑Spearman(U, S) ↑rlowSr_{\mathrm{lowS}} ↓
Raw gradient of B0.593 / 0.8570.160 / 0.38913.5% / 4.1%
Actual AdamW update of B0.190 / 0.246−0.054 / 0.25020.5% / 12.3%
Joint update of A and B, Δ(BA)\Delta(BA)0.307 / 0.343−0.039 / 0.21220.5% / 15.1%

Conclusion: Under the actual updates, the gap in concentration narrows, which is consistent with the approximate invariance of Adam to per-coordinate rescaling of the gradient (Kingma & Ba, 2015). However, the location of the concentration is preserved: the actual updates of McLoRA still favor highly activated dimensions (Spearman 0.250, rlowSr_{\mathrm{lowS}} 12.3%), whereas LoRA is at chance level (−0.054, 20.5%); the same holds for the joint update Δ(BA)\Delta(BA).

Explanation: Adam can cancel the overall scaling of the gradient, but not the inconsistency of gradient directions. In McLoRA, the gradient of the ii-th row of B contains the factor (Wx)i(Wx)_i, which is close to 0 on weakly activated dimensions and changes sign across tokens, making the gradient direction unstable; Adam's effective step size becomes smaller accordingly. Our checks are consistent with this: the sign of the backbone output is less stable on weakly activated dimensions (the Spearman correlation between sign stability and S is 0.55); in the lowest-S bin, the directional consistency of McLoRA gradients within 10 steps is 0.25, close to chance level, compared with 0.40 for LoRA; the correlation between McLoRA's Adam effective step ratio and S (0.24) is comparable to the alignment of its actual updates (0.25), whereas for LoRA it is close to 0.

Revision: In the revised version, we will add statistics based on the actual updates to Section 3.2 and Table 1, and include the above checks in the appendix.

Response 2 (Weakness 1, part 2; Question 2): A unified definition of old-task-critical dimensions

We thank the reviewer for this comment. Appendix F defines the critical dimensions for each method using the signal that drives its own updates (E∣δ∣\mathbb{E}\lvert\delta\rvert for LoRA and E∣δ⊙Wx∣\mathbb{E}\lvert\delta\odot Wx\rvert for McLoRA). We agree that this does not allow the two methods to be compared on the same set of dimensions, and we therefore recomputed the statistics using a definition and a mask that are identical for both methods.

Common definition: On the single-task reference models of both methods, we measure the importance of each dimension by the gradient magnitude of the old-task loss with respect to the module output, E∣∂L/∂h∣\mathbb{E}\lvert\partial L/\partial h\rvert, normalize it within each layer, and average it over the two methods. The smallest set of dimensions covering 80% of the cumulative importance is taken as the old-task-critical dimensions, and removing the critical dimensions of the new task gives old_unique. Both methods use the same mask, and the statistics use the same training runs as in Response 1.

Table R2 | Fraction of new-task update energy falling into the common old_unique mask (each cell: LoRA / McLoRA; uniform-allocation baseline ≈ 13.9%)

Measured quantityEnergy fraction
Actual AdamW update of B13.7% / 9.6%
Joint update of A and B, Δ(BA)\Delta(BA)13.9% / 10.0%

Conclusion: Under the common mask, the energy of LoRA's new-task updates falling into old_unique is on par with the uniform baseline, whereas that of McLoRA is about 0.7 times the baseline, i.e., McLoRA's updates still fall less into the critical dimensions unique to the old task.

Revision: In the revised Appendix F, the common definition will be the primary one, and the original definition will be kept as a supplement.

Response 3 (Weakness 2; Question 3): Budget-matched continual-learning baselines and multi-seed results

We thank the reviewer for the suggestion. Under the setting of a single shared LoRA, with no additional trainable parameters and no data replay, there is no strictly budget-matched LoRA-based continual-learning method in the existing literature:

  • O-LoRA, N-LoRA, AM-LoRA, and LB-CL introduce new LoRA modules or low-rank parameters for each task, rather than continually training the same shared adapter. These four methods have already been compared with Inc-McLoRA under the multi-adapter protocol in Table 6 of the paper.
  • BranchLoRA targets continual instruction tuning of multimodal large models and adds a new router for each task.
  • SD-LoRA, CL-LoRA, and C-LoRA target class-incremental learning in vision; among them, SD-LoRA and CL-LoRA also introduce new directions or task-specific adapters for each task.
  • Share stores a set of coefficients for each task, so its parameter count grows with the number of tasks.

Therefore, we chose EWC and LwF, classical continual-learning methods that can be applied directly to the same shared adapter without adding trainable parameters or replaying data, as baselines, and also examined their combination with McLoRA. All results are reported as mean ± standard deviation over 3 random seeds.

Setup: RoBERTa-large, sequence S2, following the hyperparameters in Table 13; λ for EWC is selected from {10, 100, 1000} and α for LwF from {0.1, 1, 10} on the validation set.

Table R3 | Budget-matched continual-learning baselines and multi-seed results (S2, RoBERTa-large)

MethodFinal Avg ↑Forgetting ↓
LoRA76.7 ± 3.618.2 ± 4.2
McLoRA83.2 ± 1.79.3 ± 2.8
LoRA + EWC84.1 ± 2.47.3 ± 2.2
LoRA + LwF83.8 ± 2.97.7 ± 2.7
McLoRA + EWC89.4 ± 1.41.1 ± 0.3
McLoRA + LwF88.3 ± 2.12.3 ± 0.7

Conclusion: (1) The Final Avg of McLoRA is on average 6.5 points higher than that of LoRA, and its forgetting is on average 8.9 points lower, with smaller variation across seeds (standard deviation 1.7 vs. 3.6). (2) When combined with EWC or LwF, McLoRA achieves better performance than LoRA combined with the corresponding method. (3) Even when used alone, McLoRA achieves results competitive with LoRA + EWC and LoRA + LwF, without requiring an additional regularization term, Fisher estimation, or a teacher model.

Revision: The revised version will include the above multi-seed results and baseline comparisons.

Dear Reviewer,

Thank you again for your time and constructive comments. We have tried to address each of your concerns in our responses above, including additional experiments and analyses. We would be grateful to know whether our responses have resolved your concerns, and we look forward to further discussion if any questions remain.

Best regards, The Authors

Comments on the paper

2 comments

General Response to the AC and All Reviewers

We sincerely thank the AC and the three reviewers for their time and valuable comments. We have responded to all review comments point by point and provided the corresponding additional experiments and analyses, but we have not yet received further feedback from the reviewers. We would very much like to know whether our responses have addressed the reviewers' concerns, and we look forward to further discussion with the reviewers during the discussion period. For ease of reference, we briefly summarize below the reviewers' assessments and our additions.

1. Strengths recognized by the reviewers

  • A new composition paradigm (Reviewer JjXT): The paper replaces the standard additive composition in LoRA with element-wise multiplication between the frozen backbone output and the low-rank branch output, which is a new paradigm. The reviewer specifically noted that "the idea of designing a new paradigm for LoRA in continual learning is interesting to me, since previous works mostly follow the standard additive composition."
  • A simple method with convincing empirical improvement (Reviewer mEFT): The method is simple and maintains a fixed adapter budget. In Table 2, McLoRA achieves better final accuracy and lower forgetting in 27 of 28 settings, "making the empirical improvement convincing." The gains also extend to long task streams, boundary-free learning, and AdaLoRA/DoRA variants.
  • A clear method, substantial experiments, and useful analyses (Reviewer fned): The method is simple and well-specified, does not increase the trainable-parameter budget, and its gradient derivation clearly identifies the effect of the backbone output on adapter updates. The empirical results are substantial. Analyses beyond final scores, such as activation interventions and update statistics, help characterize the method's behavior.

2. Main concerns and our additions

The reviewers' concerns mainly fall into the following three aspects, all of which we have addressed:

  1. Mechanism analysis (all three reviewers)
    • We added a first-order analysis of the forgetting error: McLoRA multiplies the interference between the new and old tasks in each dimension by the product of their backbone outputs, so the interference is suppressed as long as one side is weakly activated. We verified this on 24 "old task → new task" directions, where the forgetting error is reduced by 56%. We also identified the condition under which the mechanism holds: the more the high-activation dimensions of the new and old tasks overlap, the smaller the advantage.
    • We recomputed the statistics based on the actual AdamW updates and the joint update of A and B: the updates still favor highly activated dimensions.
    • Under a definition of old-task-critical dimensions shared by both methods, McLoRA's updates fall less into the critical dimensions unique to old tasks.
    • We provided all metrics of the original Table 1 on more backbones and tasks, as well as the results on A.
  2. Design rationale (Reviewers JjXT and fned): We explained why Wx rather than W is used, and the role of the constant 1 in Eq. (1). We added controlled alternatives (shuffled activation, magnitude gate, static activation weighting, and RMS-normalized gate), which show that the gain comes from modulating each dimension by its own backbone output. In addition, we added a comparison with LoRMA.
  3. Baselines and robustness (Reviewers mEFT and fned): We added two budget-matched continual-learning baselines, EWC and LwF, as well as results over 3 random seeds. Used alone, McLoRA is already comparable to these two methods, and it can be combined with them.

We are happy to provide further clarification on any other questions.

Update after the reviewers' replies

We thank Reviewers mEFT, JjXT, and fned for their replies.

  • Reviewer mEFT confirmed that the unified mask, the EWC/LwF baselines, and the multi-seed results resolve those concerns. On the remaining question about the mechanism under AdamW, we have replied in that thread: dimensions with larger backbone output magnitudes still receive stronger actual updates, and the effects under SGD and AdamW arise from the same backbone-output factor Wx in the gradient. We will revise Section 3 to clearly distinguish gradients from actual updates.
  • Reviewer JjXT noted that the mechanism remains unclear and that the experiments are not comprehensive enough. We would appreciate it if the reviewer could point out the specific remaining issues, so that we can address them during the discussion period. Our mechanism analysis and additional experiments are summarized under "Main concerns and our additions" above. The paper also evaluates McLoRA on 4 backbones × 7 task sequences (Table 2), where it improves final performance and reduces forgetting in 27 of 28 settings, as well as on long task sequences of up to 38 tasks, settings without explicit task boundaries, AdaLoRA/DoRA variants, and single-task fine-tuning.
  • Reviewer fned appreciated the conditional explanation and asked for explicit conditions under which the multiplicative formulation reduces old-task loss. In that thread, we derived such a condition: at the same new-task progress, McLoRA yields smaller first-order forgetting if and only if a ratio R of backbone-output-weighted averages is below 1. The reviewer's case of shared strongly activated dimensions corresponds to R ≈ 1 (no first-order advantage). Following the reviewer's suggestion, we will reorganize Section 3 around this condition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.