acceptodds
Under review as a conference paper at ICLR 2027

Static Gradient Attribution Underperforms a Density-Matched Random Mask Within LoRA’s B-Matrix

Abstract

Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA (): the gradient with respect to is identically zero (), so gradient masks select strictly from the matrix whereas unrestricted random masks draw from both and . Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability–plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero. The 95% interval extends to percentage points in the gradient mask's favour, so the pre-specified 4-point non-inferiority criterion is not met. On loss-based forgetting, Random-B consistently outperforms the density-matched gradient mask across seven seeds; on accuracy, the evidence remains inconclusive at . The attribution pass is therefore not justified by the loss-based forgetting evidence, while its accuracy consequences require a better-powered evaluation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Lifelong Learning, Parameter-Efficient Fine-Tuning (PEFT), LoRA, Gradient Attribution, Selective Masking, Catastrophic Forgetting, Large Language Model, Small Device

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CQK4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on selecting a fixed subset of LoRA parameters for continual learning. It asks whether gradient attribution is worth its cost compared with random selection. Specifically, standard LoRA initialization gives A no gradient at the start. Gradient-based selection therefore picks only from B, while an unrestricted random mask draws from both A and B. This changes both the selection rule and the candidate pool. To isolate the selection rule, the authors use the same 10% budget within B for both methods. Experiments conducted on Qwen2.5-3B across five sequential tasks and seven seeds, and on Phi-2 with four seeds, show that random selection consistently yields lower loss-based backward transfer.

Critical strengths
  1. The paper provides a controlled comparison by restricting gradient-based and random selection to the same parameter pool and 10% update budget, allowing the selection strategies to be compared under matched conditions.
Critical weaknesses

1.The practical value of the study remains questionable. The paper fixes most LoRA parameters from initialization and uses the same mask across all tasks. While this may reduce forgetting, it may also limit the model's ability to learn new tasks, raising concerns about its usefulness in practical continual learning.

2.The evaluation does not adequately assess overall continual learning performance. Both loss-based and accuracy-based Backward Transfer (BWT) measure changes in previous-task performance, so a model that learns poorly may still exhibit low forgetting. The authors should consider reporting common metrics such as final average performance and average incremental performance [1] to better assess the balance between learning and retention.

[1]Towards Lifelong Learning of Large Language Models: A Survey. ACM Computing Surveys, 2025.

Questions for authors
  1. What practical contribution does the proposed random masking strategy make to continual learning? Since the same randomly selected parameters remain fixed across all tasks, how does this design support learning new tasks while retaining previously acquired knowledge?

  2. Could the authors compare random and gradient-based parameter selection starting from the same checkpoint after the first task? Applying the two selection strategies to subsequent tasks would help clarify whether random masking remains advantageous once the model has acquired task-specific knowledge.

  3. The experimental results raise concerns about robustness. Table 19 shows that the advantage of Random-B is largely concentrated on the Medical task, suggesting that the findings may be sensitive to specific task characteristics.

  4. Although Random-B consistently outperforms Grad-B in loss-based BWT, this advantage is not confirmed by the accuracy-based results. As shown in Figure 4, the mean Accuracy-BWT difference is only 0.94 percentage points, with a 95% confidence interval that includes zero. This raises concerns about whether the reported reduction in loss-based forgetting translates into meaningful improvements in task performance.

2 comments

We thank the reviewer for raising the distinction between reduced forgetting and successful continual learning. We agree that the value of a static mask must be judged by both retention and acquisition, and that our evidence does not establish a generally superior continual-learning algorithm. We clarify the limits of the evidence.

Q1. Practical value and ability to learn new tasks.

Our contribution is a controlled negative result and evaluation protocol, not a new adaptive CL algorithm. With standard LoRA initialized at B = 0, initialization-time attribution assigns zero gradient to A. Unrestricted A+B random sampling therefore confounds the selection rule with the pool. By holding both the B-only pool and the 1,651,508-coordinate budget fixed, we test whether paying for gradient ranking improves retention relative to uniform selection. In this setting Random-B lowers loss-based BWT from 0.449 to 0.294 across seven paired Qwen2.5-3B seeds, while avoiding a separate 64-sample attribution pass.

We acknowledge the stability–plasticity cost of fixing one sparse mask across all tasks. Indeed, Grad-B obtains lower post-acquisition (diagonal) loss in all 35 seed × task comparisons (Appendix R, Table 20). Random-B therefore does not learn each new task better, and lower BWT alone would be insufficient evidence of practical superiority. However, the effect is not explained solely by a weaker starting point: averaging final held-out loss over the four prior tasks, Random-B is lower in six of seven seeds (Grad-B minus Random-B = +0.0813, paired p = 0.023; Appendix R, Table 21). This endpoint check does not establish superior task accuracy or the full plasticity–retention trade-off. We will emphasize that distinction.

Q2. Shared checkpoint after the first task.

We agree this is an important complementary experiment, but it answers a different question. Our mask is chosen before task 1 to isolate the zero-initialization confound. After task 1, B will generally be nonzero and ∂L/∂A need no longer vanish; the motivating confound is no longer guaranteed. A clean extension would first train an identical task-1 checkpoint, then construct equal-budget gradient and random masks from that checkpoint, train both on tasks 2–5 with identical data and seeds, and compare acquisition, final performance, and forgetting. We have not run this experiment, and do not claim that our initialization-time ordering extends to post-task adaptation. We will identify it explicitly as future work.

Q3. Medical-task concentration and robustness.

We agree: Medical contributes approximately 79.9% of the aggregate Qwen2.5-3B gap (Appendix Q, Table 19). The effect is not uniform across domains. Still, the per-task mean Random-B minus Grad-B BWT is negative on Code (−0.046), Medical (−0.496), and Math (−0.076); Legal is approximately neutral (−0.003). The density-matched Phi-2 replication preserves the aggregate direction in four of four seeds (mean difference −0.1152), although Medical still accounts for 67.2% of that gap. The medical concentration is a genuine limitation; the single-seed 7B probe also slightly favors gradient (0.441 vs. 0.447). We do not establish robustness to balanced task mixtures or multiple task orders; the existing reversed-order probe is only single-seed and shows that task order can substantially alter absolute BWT. We will further foreground this limitation.

Q4. Loss-based versus accuracy-based evidence.

We agree that lower loss-BWT does not demonstrate better task success. On Medical and Legal, accuracy-BWT differs by only +0.94 percentage points (Grad-B minus Random-B), with a 95% CI of [−2.91, +4.80] pp. This is inconclusive and fails our pre-specified 4-pp non-inferiority criterion. Nor have we reported the standard final average accuracy or average incremental accuracy requested by the reviewer. The stored task-by-checkpoint loss matrices permit analogous loss-based summaries, but these are not substitutes for task-success metrics. Reliable task-level evaluations for Code and Math, alongside final average and incremental accuracy, are required before making a deployment recommendation. We will revise the discussion to make this evidence gap explicit rather than implying that the loss result transfers to accuracy.

In summary, the supported claim is not that fixed random masking is the best practical CL strategy. It is that, for this particular static, initialization-time LoRA-B selection problem, gradient attribution does not outperform an equally sparse random baseline on loss-based retention, despite its calibration cost. Post-task checkpoint and broader performance evaluations remain important tests of external validity. We will reinforce the title’s qualifiers—static, within-B, and loss-based—throughout the abstract and discussion.

Reviewer_CQK4OpenReview ↗

Official Comment

Thank you for the response. My two main concerns remain unresolved.

  1. The authors explain why they select the mask at initialization. My concern is about the practical value of keeping the same mask fixed across all tasks. As the authors report, Random-B achieves higher loss than Grad-B immediately after learning each new task in all 35 paired comparisons. Reducing forgetting has limited practical value for continual learning if it comes at the cost of weaker learning on new tasks.
  2. As the authors acknowledge, lower loss-based forgetting does not imply better task performance. The response provides no additional results on final average performance or average incremental performance, so it remains difficult to assess the effectiveness of the method in continual learning. For these reasons, I will keep my original rating.
Reviewer_e9SHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sparse-mask continual fine-tuning of LoRA adapters. It notes that under standard zero-initialised LoRA (B = 0), ∂L/∂A = 0 at initialisation, so a gradient-attribution mask selects only from lora_B, whereas an unrestricted random baseline draws from both A and B. To isolate the selection rule, both arms are restricted to lora_B at the same density: Grad-B keeps the top 10% of B coordinates by mean absolute gradient on 64 Alpaca calibration samples, and Random-B selects 10% uniformly at random. The mask is fixed before the first task. On Qwen2.5-3B (4-bit, rank 16) over a five-task sequence (Code, Medical, Legal, Math, Creative, with 500 training and 100 evaluation samples per task), Random-B has lower loss-based BWT in all seven seeds (0.294 vs. 0.449, paired t(6) = 5.80). About 80% of the gap comes from the Medical task. Grad-B achieves lower diagonal loss on every task, and Random-B achieves lower mean final loss on the prior tasks (+0.081, p = 0.023). Accuracy-BWT on Medical and Legal is inconclusive (+0.94 pp, 95% CI [−2.91, +4.80]). A four-seed Phi-2 replication shows the same direction.

Critical strengths
  1. Matching both the parameter pool and the number of trained coordinates is the right control for comparing selection rules, and the paired within-session analysis is well suited to the observed session drift.

  2. The paper reports per-seed results, leave-one-out analyses, a pre-specified non-inferiority margin that is not met, and a diagonal-loss check showing that Grad-B learns each task better (35/35 cells) yet ends with higher prior-task loss.

  3. The scope of the conclusions is stated carefully, including the fact that the accuracy evidence is inconclusive.

Critical weaknesses
  1. Inconsistent reporting of the same contrasts between Sect. D, Appendix F and Table 2.

  2. Sect. 6.1 states that no single-seed probe reverses the conclusion, yet at 7B the gradient mask has slightly lower BWT than the random mask (0.441 vs. 0.447, Table 13).

Questions for authors
  1. Why do Table 2 and Fig. 2 report different values for Random-B − Grad-Full, and which sessions does each come from?

  2. Please use a larger evaluation set (for example the full MedQA and CaseHOLD test sets). Is the accuracy-BWT difference still indistinguishable from zero?

  3. The gradient attribution is a weak baseline. Based on the baseline, the conclusion is limited. It will afftect the paper's claim. Especially, the title’s claim that gradient attribution underperforms random selection applies only to this specific static design.

The rating is based on 1) inconsistent numbers weakening confidence in the main contrast. 2) The experiment is poor.

1 comment

We thank the reviewer for identifying two reporting issues and for acknowledging the value of the pool- and density-matched design. We will clarify the numerical provenance and scope of our claims.

Q1. Why do Table 2 and Fig. 2 differ?

The values come from different, independently executed sessions. Both report Random-B minus Grad-Full (a pool-matched but density-unmatched contrast), but Fig. 2 uses the earlier within-B session summarized in Appendix D, Table 6: Random-B 0.338, Grad-Full 0.409, mean difference −0.072. Table 2's unmatched column instead comes from the subsequent D1 session, in which Random-B was also compared against density-matched Grad-B: mean Random-B minus Grad-Full −0.144. The primary D1 contrast is different: Random-B minus Grad-B, with the same 10% B budget, yielding −0.155 (0.294 vs. 0.449; 7/7 paired seeds).

Appendix F, Table 9 makes the cross-session variation concrete: for seed 42, Random-B is 0.326 in Run A versus 0.278 in Run C (D1), and Grad-Full is 0.491 versus 0.498. Its cause is unresolved, and we never pair measurements across sessions. All significance tests are computed only from method contrasts within the same session and seed. The D1 matched contrast also reproduces in mean on a separate A800 environment (+0.152 in Grad-B minus Random-B units, vs. +0.155 on V100; Appendix N), although individual seeds vary. We will label sessions and matched/unmatched contrasts explicitly and consolidate per-seed results.

Q2. Accuracy on larger evaluation sets.

We agree that 100 evaluation examples per task are insufficient for a definitive task-success claim. We have not evaluated the full MedQA and CaseHOLD held-out sets and therefore cannot state whether the accuracy-BWT difference remains nonsignificant. The current seven-seed estimate is Grad-B minus Random-B = +0.94 percentage points, 95% CI [−2.91, +4.80], which spans zero and exceeds our pre-specified 4-point non-inferiority margin. We explicitly do not claim accuracy equivalence or superiority.

A larger held-out evaluation would reduce item-sampling uncertainty, but the present interval also reflects substantial variation across training seeds (SD 4.17 pp); more evaluation items alone may not resolve that variation. The necessary follow-up is to evaluate identical held-out items at each task checkpoint, report final accuracy and accuracy-BWT, and use seed-level paired intervals. Until such measurements exist, our conclusion remains restricted to loss-based forgetting.

Q3. Baseline strength, experimental scope, and the 7B exception.

We agree that our gradient mask is a specific static, initialization-time gradient-magnitude baseline, not a representative evaluation of all attribution-guided continual-learning methods. In particular, Attribution-Guided CL scores a separately trained model and uses continuous gating, so the zero-init confound does not apply to it. Our aim is not to reject that method or establish a new SOTA continual-learning algorithm. The methodological question is narrower: when the same B coordinates are eligible and precisely 1,651,508 parameters are trained in each arm, does this extra gradient-based ranking improve retention over uniform selection? In our main test, it does not. The controlled result remains informative precisely because it removes a confound that can otherwise invalidate interpretation of a gradient-versus-random comparison.

The reviewer is also correct about Table 13. At Qwen2.5-7B, Grad has lower BWT than Random (0.441 vs. 0.447, a 0.006 reversal). Our sentence in Sec. 6.1 stating that no single-seed probe reverses the conclusion is incorrect and will be removed. This is a single-seed scale observation, not a multi-seed, density-matched confirmation, and neither establishes an advantage at 7B nor supports scale-wide generalization. We will explicitly mark it as an exception and further restrict the text to the tested Qwen2.5-3B D1 sequence and four-seed Phi-2 replication.

We appreciate that these issues weakened confidence. The distinction we ask the reviewer to retain is between correctable reporting/overgeneralization errors and the primary controlled comparison: D1 fixes pool, density, task stream and within-session pairing; Random-B has lower loss-BWT in all seven Qwen2.5-3B seeds, with the same direction in four Phi-2 seeds. We will revise the presentation and claims accordingly, while leaving accuracy and broader task-order/scale generalization as open questions.

Reviewer_gWXROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a comparison confound caused by standard zero-initialized LoRA: initialization-time gradients vanish for AA, so gradient selection and unrestricted random selection use different candidate pools. It then matches both the candidate pool and selection density within BB. Random masks exhibit less loss-based forgetting in the evaluated setting.

The title, methodology and conclusion are well aligned, and the authors appropriately restrict the central claim.

Critical strengths
  1. The strongest aspect is the experimental control. Both matched arms select the same 1,651,508 parameters, and the primary effect has a consistent direction across seven seeds.

  2. The endpoint-loss analysis helps show that the finding is not merely an artifact of backward-transfer normalization.

  3. The Phi-2 replication, calibration-domain check and execution-order checks add credibility.

  4. The authors report the inconclusive accuracy result rather than extrapolating from loss to task success. This restraint improves the paper.

Critical weaknesses

The practical reach of the finding remains uncertain. The closest attribution-guided continual-learning method scores a separately trained model, so the zero-initialization confound does not apply to it. The paper needs a clearer account of which existing selection protocols are actually affected. The main result also relies on a small, fixed five-task sequence, with the medical task accounting for roughly 80% of the aggregate gap. The accuracy comparison does not meet the stated non-inferiority criterion, while code and math lack adequate task-success evaluation. These limitations do not invalidate the controlled result. They do, however, leave its decision value for continual-learning practitioners unclear.

Questions for authors
  1. Which concrete existing methods would change their recommended procedure after adopting this matched comparision?

  2. Does the result persist across several task orders and more balanced task collections?

  3. At matched acquisition quality, does random selection still improve final retention and task accuracy?

1 comment

We thank the reviewer for recognizing the controlled design, seven-seed consistency, and careful treatment of the accuracy results. We agree that the main question is how far this controlled negative result can inform practical continual learning. Our intended claim is deliberately narrower than "random masking is generally better than attribution": under standard zero-initialized LoRA and a fixed, initialization-time mask, gradient selection and random selection must use the same eligible parameters and update budget before their retention can be compared.

Q1. Which existing methods would change their procedure?

We agree that Attribution-Guided CL (Liu et al., 2026) is not subject to our specific confound: it scores a separately trained model, where B is nonzero, and uses continuous gating rather than our static binary mask. Similarly, we do not claim that EWC/SI-style penalties, non-zero-initialized adapters (e.g., PiSSA), or other CL methods should change their training algorithms.

The concrete recommendation concerns evaluation protocols for one-shot gradient-based masking applied at standard LoRA initialization. Since ∂L/∂A = 0 when B = 0, comparing its gradient-selected coordinates against random coordinates drawn from A∪B changes both the selection rule and the candidate pool. Such a study should include a within-B, density-matched random baseline.

Our D1 test activates exactly 1,651,508 B coordinates in each arm (10% of B), obtaining loss-BWT 0.294 for Random-B versus 0.449 for Grad-B (seven of seven paired seeds; mean difference −0.155). Thus, our actionable contribution is a necessary experimental control and a strong inexpensive baseline for this particular selection protocol—not a claim that a named prior CL algorithm has been invalidated. Within our tested setup, omitting the separate 64-sample attribution pass also avoids calibration data and computation equivalent to 12.8% of one 500-step task budget.

Q2. Task-order and task-composition robustness.

We agree that the five-task sequence limits external validity. Medical contributes 79.9% of the Qwen2.5-3B aggregate difference; the remaining Code and Math contributions favor Random-B, while Legal is nearly neutral (Fig. 3; Appendix Q).

The density-matched Phi-2 replication shows the same direction in all four seeds (mean Random−Grad BWT = −0.1152), with Medical still contributing 67.2% (Appendix S). This supports reproducibility across architectures, not invariance to task composition.

Importantly, our single-seed reversed-order check changes overall BWT from approximately 0.42 to 0.11 (Appendix K), demonstrating that order can matter more than the method contrast; it cannot establish multi-order robustness.

We will make this boundary more prominent. A decisive generalization test would use several pre-registered permutations, balanced domain collections, and task-level results stratified by whether Medical occurs early or late. We do not claim that these experiments have been completed.

Q3. Matched acquisition quality and task success.

We agree this is the key unresolved practical question. Gradient-B reaches lower immediate post-task loss in all 35 seed × task comparisons (Appendix R), so Random-B's lower BWT cannot be explained by superior initial fitting.

Moreover, Random-B has lower final held-out loss averaged over prior tasks in six of seven seeds (Grad−Random = +0.0813; paired p = 0.023), which helps rule out a purely BWT-baseline artifact.

However, these are not acquisition-matched checkpoints, and we cannot infer that random selection remains better after equating initial learning quality. The next decisive analysis is a retention–acquisition trade-off curve at matched per-task validation loss/accuracy, followed by final task-success comparisons.

Our present accuracy-BWT result on Medical and Legal is inconclusive (Grad−Random = +0.94 pp; 95% CI [−2.91, +4.80] pp) and fails the pre-specified 4-pp non-inferiority test. Code and Math additionally require reliable functional/answer-level evaluation. We therefore do not claim random masking improves task accuracy or quality at matched acquisition.

In short, the paper provides a controlled counterexample to the assumption that initialization-time gradient importance automatically improves loss-based retention when the pool and density are fixed. Its practical conclusion is a better comparison standard and a viable low-cost baseline in the evaluated regime, not a universal prescription for continual learning. We appreciate the reviewer's suggestions and will sharpen this distinction in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.