acceptodds
Withdrawn from ICLR 2027

Preserve More, Freeze Less: Loss-Aware Subspaces for Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Continually fine-tuning a pretrained language model requires acquiring new capabilities (plasticity) while retaining both its initial capabilities and those learned along the way (stability). Projection-based methods constrain parameter updates to protect directions associated with prior knowledge. Existing approaches select directions without measuring their importance for prior behavior, protect directions too coarsely, reducing plasticity, or store reference gradients, which does not scale to large models. We formulate an explicit preservation objective and derive a tractable approximation that identifies pairs of input and output directions in each linear transformation and estimates their importance from limited reference data. Building on this formulation, we develop SCLL, a projection-based continual learning method that protects only the most important pairs, improving the balance between preservation and adaptation. We evaluate SCLL on two continual learning benchmarks while protecting five initial capabilities. On TRACE, whose tasks differ in language, domain, and output format, SCLL improves average accuracy over the strongest projection baseline by 4 points while matching its retention and freezing fewer update directions; on Long CL, whose tasks require few new directions, it matches the strongest baseline. Protecting the selected initial capabilities also retains performance on related datasets not used for protection.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Update Projection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N3sXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a projection-based method for continual learning in language models. It uses loss statistics from reference data to estimate the importance of input–output direction pairs in each weight matrix, then removes components along protected directions from optimizer updates when learning new tasks. The paper presents the definition of these directions as a flexible design choice, but this flexibility makes its main contribution less clear. Moreover, it does not directly validate whether the importance scores accurately identify the directions that should be protected, and it provides insufficient detail on key parameter settings and how the protected set changes across tasks.

Critical strengths

1、Finer-grained protection. SCLL selects input–output direction pairs rather than protecting all output directions associated with a given input direction. This provides a reasonable approach to retaining prior capabilities while learning new tasks.

2、Evaluation across different settings. The paper evaluates new-task learning and retention of prior capabilities on the TRACE and Long CL task streams. It also examines performance on related datasets that were not used to construct the protection.

Critical weaknesses

1、Insufficient validation of the importance scores. The paper makes a plausible attempt to address limitations of existing methods, but provides limited empirical evidence for its central mechanism. In particular, it proposes an approximate importance score without directly validating it. It remains unclear whether selecting higher-scoring directions leads to better training outcomes under the same protection budget.

2、Underreported settings and limited analysis of the protected set. Key method parameters are not clearly specified or ablated. For example, the main text does not report the importance-coverage threshold ρ\rho. The appendix indicates that SCLL and the strong baseline NSCL use different values of ρ\rho, but neither choice is accompanied by a corresponding ablation study. In Long CL, the proportion of directions selected for protection decreases more noticeably, yet the paper does not analyze which matrices or input–output direction pairs enter or leave the protected set.

3、Limited model diversity and scale. The experiments are restricted to Qwen3.5 models in the 0.8B–2B range. It remains unclear whether the findings extend to other model families, such as Llama, or to larger models, such as a 7B-parameter model.

4、Unclear effect of the factor-aggregation approximation. The paper aggregates the AA and BB matrices by averaging them across tasks. However, the mean of Kronecker products is generally not equal to the Kronecker product of the means. How large is the resulting estimation error? Can the final importance scores faithfully represent the expected squared change after a parameter update?

Questions for authors

1、How closely does the ranking induced by λpσq\lambda_p\sigma_q agree with the actual squared change in reference loss when the corresponding direction pair is perturbed? When the number or proportion of protected directions is held fixed, does selecting pairs by this score yield a better trade-off between learning new tasks and retaining prior capabilities than alternative selection rules?

2、What evidence supports the scalability of SCLL to larger models or other model families? Beyond the asymptotic storage complexity, could the authors report measured runtimes and peak memory usage for reference-statistics collection, eigendecomposition, and projection during training, and explain how these costs scale with model width?

3、How were the values of ρ\rho chosen for SCLL and NSCL, respectively? How sensitive are the results to these choices?

Reviewer_t8NqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SCLL for continual fine-tuning of language models. It uses a first-order preservation objective and Kronecker-factored statistics to identify loss-sensitive input–output pairs, then projects optimizer updates away from the selected pairs. Experiments on TRACE and Long CL evaluate task acquisition, forgetting, and retention of initial capabilities, with a Permuted MNIST study examining pair-level protection.

Critical strengths
  1. The motivation for pair-level protection is somekind sound. Because output sensitivity σ_q varies across outputs sharing the same input direction, protecting the entire input direction (as in NSCL) unnecessarily restricts harmless directions; deriving pair importance λ_p·σ_q from the loss objective gives a principled reason for finer-grained protection, with NSCL recovered as the all-outputs-protected special case.
Critical weaknesses
  1. Limited model scale and weak statistical evidence in the main results. The headline TRACE gain rests on a 0.8B model with a single task order and a single seed; the 2B result appears only in the appendix, but I can't find the appendix, there is no 7B-class or larger evaluation. Claims of practical relevance for large LLMs are therefore not yet established.
  2. Layerwise second-moment matrices, eigendecompositions, and per-step two-sided projections introduce extra cost. Training time and peak memory are not quantified in the main text.
Questions for authors
  1. Can the authors evaluate SCLL against NSCL on a 7B-class or larger model?
  2. What are the wall-clock and peak-memory costs of factor construction, eigendecomposition, and per-step projection?
Reviewer_gPDoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCLL, a projection-based method for continual learning in language models. The key idea is to protect individual input-output direction pairs instead of entire input directions. The importance of each pair is estimated from input activations and output-side loss sensitivity, using a Kronecker-factored approximation of a preservation objective. During new-task training, updates along the selected directions are removed. Experiments on TRACE and Long CL show improvements over existing projection methods, particularly on TRACE.

Critical strengths

The paper addresses an important limitation of projection-based continual learning: protecting too many directions can unnecessarily restrict learning new tasks. Protecting individual input-output pairs is a reasonable way to improve the trade-off between stability and plasticity.

The method has a clear mathematical formulation. Rather than selecting directions solely from weight spectra or input activations, it attempts to measure their importance to the reference loss.

The TRACE results are promising. SCLL improves average accuracy over NSCL by 4.01 points while achieving similar retention and freezing fewer update directions. The evaluation on related datasets also provides some evidence that the protection generalizes beyond the reference datasets.

Critical weaknesses

The methodological novelty is not sufficiently clear. The Kronecker factorization and the resulting product of eigenvalues follow naturally from the K-FAC approximation. The main contribution appears to be applying this structure to select individual input-output pairs for projection. While this is a reasonable design, the paper should better distinguish its contribution from existing gradient projection and Kronecker-factored methods.

The contribution of loss-sensitivity weighting is not adequately isolated. SCLL uses sensitivity-weighted input statistics and output-sensitivity directions in the language-model experiments. However, the Permuted MNIST experiment uses unweighted factors, primarily demonstrating the benefit of pair-level protection. It remains unclear how much of the main improvement comes from finer-grained pair selection and how much comes from sensitivity weighting. A more complete ablation on the language-model benchmarks is needed.

The comparison with NSCL does not fully establish a better stability-plasticity trade-off. SCLL freezes 52.11% of update directions on TRACE, compared with 66.46% for NSCL. This supports the claim that SCLL preserves more update freedom, but it does not establish whether pair-level selection is better under the same protection budget. Comparing both methods across matched freezing ratios would provide stronger evidence.

The preservation objective relies on several approximations. The method uses a first-order approximation of reference-loss changes, a block-diagonal approximation across parameter matrices, and a Kronecker-factored approximation within each matrix. These assumptions make the method tractable, but their effects on the estimated importance of input-output pairs are not sufficiently investigated. In particular, preserving the reference loss does not necessarily preserve the model's full predictive behavior.

The empirical evaluation is limited. The main experiments use Qwen3.5-0.8B, and TRACE is evaluated with a single task order and training seed. The gains on Long CL are also small: average accuracy improves from 81.12 to 81.44. More seeds, stronger statistical analysis, and experiments on larger models would help establish the robustness and scalability of the method.

The computational and storage costs need further clarification. Although SCLL avoids storing full parameter-sized gradients for every protected direction, it maintains covariance-like factor matrices and performs eigendecompositions for individual weight matrices. The paper should quantify the additional memory, computation, and runtime compared with NSCL, particularly for larger models.

Questions for authors

Can the authors better clarify the novelty of SCLL relative to K-FAC-based importance estimation and existing gradient projection methods? Which part of the proposed formulation cannot be obtained through a straightforward application of these existing techniques?

Can the authors provide a controlled ablation on TRACE that separately evaluates input-only protection, unweighted input-output pair protection, and the full sensitivity-weighted SCLL? This would help identify the source of the reported improvement.

How does SCLL compare with NSCL when both methods protect the same proportion of update directions? Can the authors report stability-plasticity trade-off curves across different protection thresholds?

How sensitive are the selected pairs to the reference dataset size, the energy threshold, and the Kronecker approximation? Does the estimated importance reliably predict the actual change in reference loss after an update?

The preservation objective is defined through changes in reference loss. Can the authors provide evidence that minimizing this proxy also preserves predictive distributions or performance on capabilities beyond the reference datasets?

What are the additional runtime and memory costs of SCLL compared with NSCL, and how do these costs scale with model size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.