acceptodds
Under review as a conference paper at ICLR 2027

Stability–Plasticity Balance via Singular-Vector Selection in LLM Continual Learning

16 views

Abstract

Domain-specific continual adaptation of LLMs risks catastrophic forgetting, creating a fundamental tension between acquiring new capabilities and preserving those learned during pretraining. PEFT mitigates this problem by restricting the number of trainable parameters, but existing methods lack a principled unit for deciding where plasticity should be allocated and stability should be preserved. We identify the singular-vector channel as a natural unit for managing this trade-off. Each channel represents an input–output transformation, which can be updated to acquire new knowledge or fixed to preserve pretrained capabilities. Based on this perspective, we introduce SVC, a parameter-efficient continual-learning method that selectively updates Singular-Vector Channels. Before fine-tuning, SVC uses domain-specific data to estimate each channel’s adaptation benefit and a fixed public general-domain corpus only as a history activation proxy for estimating forgetting cost. It then adaptively selects trainable channels based on these scores via knee-based cost screening, Pareto-front filtering, and Otsu thresholding. Experimental results across four LLM families and eight downstream tasks show that SVC better preserves pretrained capabilities while achieving strong downstream performance relative to existing PEFT baselines. Further analysis of channel scoring and selection demonstrates that selective plasticity at the singular-vector-channel level enables effective continual LLM adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Large Language Models, Catastrophic Forgetting, Parameter-Efficient Fine-Tuning, Stability–Plasticity Balance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rCogOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the balance between learning new tasks and preserving existing capabilities in continual adaptation of large language models. The proposed method, SVC, estimates the adaptation benefit and potential forgetting cost of singular-vector channels using current-task gradients and activation statistics from a public reference corpus. It then selects channels through a three-stage filtering procedure, updating the selected singular vectors alongside all singular values. The method does not store previous-task examples or accumulate task-specific modules. Experiments across four models, eight continual-learning tasks, and three task orders show competitive downstream performance and improved retention of previously acquired capabilities.

Critical strengths
  1. Relevant problem and clear motivation. Selectively allocating updates to balance adaptation and knowledge preservation is a practically meaningful direction. Considering both potential learning benefits and forgetting risks provides an interpretable rationale for selecting which parts of the model to update.

  2. Broad experimental coverage and competitive results. Evaluation across multiple models and task orders provides useful evidence of the method’s effectiveness. The experiments assess both continual-task performance and capability retention.

Critical weaknesses
  1. The additional value of the complete selection procedure is not sufficiently established. SVC combines several selection stages, but the current evidence does not clearly demonstrate their necessity relative to simpler alternatives. On Llama3-8B under Order 1, the low-cost selection variant in Table 2 matches SVC’s channel count and achieves better OP and BWT, while obtaining similar capability-retention results. Moreover, the stage-removal ablations in Figure 3 also change the number of selected channels. Consequently, they do not fully distinguish the benefit of the selection strategy from the effect of changing the update budget. Clearer evidence separating these factors would strengthen the paper’s central methodological contribution.

  2. SVF, which directly motivates the proposed method, is evaluated in the independent-task motivation experiment but lacks a complete continual-learning comparison. The related spectral adaptation method SVFT [1] is also absent from the experimental comparisons. Additionally, the cited PIECE method [2] studies task-relevant parameter selection for continual adaptation and capability preservation, making it another relevant baseline. Comparisons with these methods would help establish the benefits of SVC over closely related approaches.

  3. SVC and the main PEFT baselines also differ in their update scope and parameter allocation, yet the actual trainable parameter counts of SVC are not reported. Appendix D partially addresses the difference in module coverage, but does not establish comparability under matched parameter budgets. In my opinion, reference-corpus processing, matrix decomposition, and channel scoring introduce additional computation, without corresponding runtime or peak-memory measurements. These omissions limit the assessment of the method’s practical efficiency.

[1] SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors. [2] Parameter Importance-Driven Continual Learning for Foundation Models.

Questions for authors
  1. How should the strong performance of the simple low-cost selection variant in Table 2 be interpreted? Which existing results best establish the additional value of the complete selection procedure?

  2. Could the authors clarify the omission of full continual-learning comparisons with SVF, SVFT, and PIECE, and indicate whether comparable results are already available?

  3. What are the actual trainable parameter counts across models and tasks, and how do channel counts change in the Figure 3 ablations?

  4. Are capability-evaluation results available before training and after individual tasks? These would help distinguish preservation of existing capabilities from transfer benefits acquired during the task sequence.

8 comments
AuthorsOpenReview ↗

Response to W1, Q1 (Part 1/3)

We thank the reviewer for raising this question. Low C in Table 2 uses the number of channels adaptively determined by full SVC for each task and each layer, and then selects the same number of lowest-cost channels. Its strong performance supports the effectiveness of cost scoring, but does not establish that lowest-cost selection can independently replace the full procedure. SVC must determine both which channels to select and how many to select, and learning benefit estimation provides an important basis for this adaptive allocation of channel counts.

To further distinguish the effects of channel selection under a given budget from those of autonomous budget allocation, we conducted the following three sets of additional experiments.

First, we examined whether cost alone can support adaptive selection of both channels and their number. Using Llama3-8B under task order 1, we retained SVC’s knee-point filtering and Otsu thresholding steps, removed the Pareto filtering based on the learning benefit–forgetting cost objectives, and used only cost information during Otsu thresholding. This variant independently determines the number of channels instead of adopting the counts determined by full SVC. All other training and evaluation settings remained unchanged.

order1
HumanEvalGSM8KOPBWT
Llama3-8B
min_cost001.86-12.86
SVC (ours)30.4951.5549.62-0.94

Under this selection rule, cost alone cannot replace the full adaptive SVC procedure.

AuthorsOpenReview ↗

Response to W1, Q1 (Part 2/3)

Second, we compared selection strategies across four models under the same channel budgets. For each task and each layer, min_cost_fix_size uses the channel count determined by full SVC and replaces only the channel selection strategy with a ranking that selects the lowest-cost channels. All experiments used task order 1, with all other training and evaluation settings held constant.

order1
HumanEvalGSM8KOPBWT
Llama3-8B
min_cost_fix_size29.2751.451.12-0.20
SVC (ours)30.4951.5549.62-0.94
Mistral-7B
min_cost_fix_size27.4439.5848.04-0.39
SVC (ours)27.4436.5448.37-0.37
Gemma2-9b
min_cost_fix_size53.6677.8650.94-0.47
SVC (ours)56.175.5150.25-1.37
Qwen3.5-9b
min_cost_fix_size50.6169.7555.00-1.15
SVC (ours)43.2972.2557.050.02

The selection strategy still affects the results when the update budget is held constant. Low C’s advantages in OP and BWT are not consistent across models.

AuthorsOpenReview ↗

Response to W1, Q1 (Part 3/3)

Finally, we examined whether fixed channel counts can replace adaptive allocation across tasks. Using Llama3-8B under task order 1, we tested two fixed budgets corresponding to the minimum and maximum channel counts determined by full SVC across tasks. Both variants always selected the lowest-cost channels. AP denotes the average performance across tasks, with each task evaluated immediately after it is learned.

order1
HumanEvalGSM8KOPBWTAP
Llama3-8B
min_cost_fix_size_task_min29.2751.1849.32-0.5549.80
min_cost_fix_size_task_max21.9548.9849.43-1.2450.52
SVC (ours)30.4951.5549.62-0.9450.44

Full SVC achieves higher HumanEval, GSM8K, and OP scores than both fixed-count variants. Fixing the channel count at a lower value reduces forgetting but limits new-task learning. Fixing it at a higher value increases AP, but mathematical and programming capabilities and BWT all decline. SVC achieves an AP close to that obtained with the fixed maximum budget, together with higher OP and better capability preservation, supporting adaptive allocation of channel counts across tasks.

These results show that SVC’s advantages arise not only from selecting where to apply updates, but also from adaptively determining how many channels to train. Joint evaluation of learning benefit and forgetting cost is key to this allocation.

Reviewer_hPapOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets continual domain adaptation of LLMs under PEFT, where the tension is between plasticity (acquiring the new domain) and stability (preserving pretrained capability). It proposes to treat the singular-vector channel (u_i, v_i) — the paired input/output directions associated with one singular value of a weight matrix — as the unit at which this trade-off is decided. SVC (i) estimates a per-channel adaptation benefit B and forgetting cost C before fine-tuning, using the average current-task gradient projected onto the channel's direction (Eqs. 5–8) and a second-moment matrix Σ_old estimated from an unlabeled public corpus; (ii) selects trainable channels by a three-stage heuristic — knee-based cost screening (Kneedle), Pareto-front filtering in the log benefit–cost plane, and Otsu thresholding of a log-efficiency score (Eqs. 9–11); and (iii) during fine-tuning trains all singular values plus the singular vectors of the selected channels only, merging the increments into the weights afterwards (Eqs. 12–13). Evaluation: four LLM families (Llama3‑8B, Mistral‑7B, Gemma2‑9B, Qwen3.5‑9B) × three task orders × the eight TRACE datasets, with HumanEval/GSM8K for capability retention and OP/BWT for sequential learning, against 11 baselines spanning regularization, replay, PEFT and SVD-based families.

Critical strengths
  1. The unit of analysis is well chosen and clearly argued. Framing the trade-off as which directions rather than how many parameters is a genuine reframing, and it is placed precisely against SVF (all directions fixed), PiSSA/MiLoRA (fixed selection by singular-value magnitude) and SVFT (sparse coefficients on fixed bases). The paper's Figure 1(a–b) makes the motivating gap concrete.
  2. Honest reporting in places. The paper states that the cost-matched comparisons have small effect sizes ("strict cost matching limits the magnitude of the differences", L1336), and reports that joint selection of singular values hurts on all four metrics (L1382) rather than hiding it.
Critical weaknesses
  1. The stated setting and the baseline set disagree about available data. This affects the interpretation of the headline comparison, because four baselines (GEM, Replay, Replay-online, OSFT) use exactly the earlier-task data the setting declares unavailable.
  2. Incomplete baselines and unreconciled OSFT results. The paper omits relevant continual-learning baselines such as EWC-LoRA (ICLR 2026), and does not explain why OSFT performs substantially differently from its original paper.
  3. The stage ablation is confounded by the channel count, so the paper's central claim that the composition of the selection procedure matters is not cleanly identified. The extreme Figure 3 numbers for "w/o Pareto" (HumanEval 0.0, GSM8K 0.0, OP 2.4) also invite the question of whether K explodes in that condition.
  4. Single-seed results throughout. No error bars, no repeated runs, and no significance analysis for margins as small as 0.6 points. This is the main threat to the "10/11 best-or-tied" claim.
  5. Mechanism evidence is limited to one model/task-order/dataset condition (Llama3‑8B, Order 1), while the method's three-stage selection is the paper's main novelty.
  6. The reference-corpus choice is not neutralised. Using a corpus that is one third Python code to estimate the "general-domain" second moments, while one of two retention benchmarks is code, leaves open that part of the retention gain is corpus-specific. There is no corpus ablation.
Questions for authors

refer to weaknesses.

7 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for raising this question. The setting without access to data from previous tasks represents the application constraint that SVC is designed to address. GEM, Replay, and Replay-online serve as reference baselines with access to such data, allowing us to compare SVC with methods that rely on historical data. The results show that SVC achieves better performance in sequential task learning and preservation of original capabilities in mathematics and programming, without requiring actual training data from previous tasks.

In our original submission, OSFT did not use data from previous tasks. Instead, it used the same fixed external corpus as SVC to estimate layer importance. We have additionally included a comparison that follows the original OSFT approach, using actual data from previous tasks to estimate importance. The results under task order 1 are reported below:

order1
HumanEvalGSM8KOPBWT
Llama3-8B
OSFT 14.637.9640.54-7.13
SVC (ours)30.4951.5549.62-0.94
Mistral-7B
OSFT 02.2744.97-3.61
SVC (ours)27.4436.5448.37-0.37
Gemma2-9b
OSFT 40.2449.1350.16-2.31
SVC (ours)56.175.5150.25-1.37
Qwen3.5-9b
OSFT 21.9510.2451.02-2.94
SVC (ours)43.2972.2557.050.02

We will explicitly specify each method’s data access conditions in the descriptions of the experimental setting and baselines. We will distinguish methods that satisfy the constraint of no access to historical data from reference baselines that can access such data, clarifying the scope of the comparisons.

AuthorsOpenReview ↗

Response to W2

We thank the reviewer for raising this question. Although EWC-LoRA does not require retaining historical samples throughout training, it uses task data to estimate Fisher information at the end of each task and carries the accumulated statistics forward to subsequent tasks. This differs from the setting used for the other PEFT methods, which do not require actual historical data, so EWC-LoRA was not included as a baseline in the original manuscript. We have now added this comparison, following the general setup described in the original paper, with rank set to 8, the EWC regularization coefficient set to 10⁷, and gamma set to 1. In preliminary experiments, EWC-LoRA exhibited strong stability but limited task adaptation. To provide greater scope for updates, we extended EWC-LoRA’s target modules to all linear layers in this comparison, matching SVC’s module coverage.

For OSFT, the results in our original submission were obtained using a modified data source for importance estimation. We used a common external dataset to estimate layer importance, accommodating the setting without access to actual training samples from previous tasks. This differs from the original protocol, which uses samples from the immediately preceding task. In our experiments with external data, we observed similar layer importance scores across tasks, resulting in similar rank allocations across layers. This may reduce the adaptability of subspace allocation to differences between tasks. When the updatable subspaces for different tasks overlap substantially, interference between updates may further increase. We consider this a possible explanation for OSFT’s poorer performance in this setting. To directly examine whether our comparative conclusions hold, we used the authors’ released OSFT implementation and followed the original protocol by using actual data from previous tasks to estimate layer importance.

All additional comparisons were conducted under task order 1. The results are reported below.

order1
HumanEvalGSM8KOPBWT
Llama3-8B
OSFT 14.637.9640.54-7.13
EWC_LoRA29.8852.6939.00-1.79
SVC (ours)30.4951.5549.62-0.94
Mistral-7B
OSFT 02.2744.97-3.61
EWC_LoRA18.937.5337.36-0.92
SVC (ours)27.4436.5448.37-0.37
Gemma2-9b
OSFT 40.2449.1350.16-2.31
EWC_LoRA59.1569.2244.63-0.23
SVC (ours)56.175.5150.25-1.37
Qwen3.5-9b
OSFT 21.9510.2451.02-2.94
EWC_LoRA41.4666.4947.000.35
SVC (ours)43.2972.2557.050.02

SVC achieves the highest OP across all four models while maintaining competitive mathematical and programming capabilities. EWC-LoRA shows advantages on some capability preservation metrics and in BWT, but its OP is lower than SVC’s on all four models. This indicates that its strong stability is accompanied by limitations in domain-specific task adaptation. In comparison, SVC achieves a better balance between task learning and preservation of original capabilities.

AuthorsOpenReview ↗

Response to W3 (Part 1/2)

We thank the reviewer for raising this question. Removing Pareto filtering does indeed substantially increase the number of selected channels. Adaptively determining the channel count for each task is an important part of SVC’s selection procedure. Figure 3 therefore examines the overall effects of each stage on both update locations and channel counts, rather than isolating each stage’s contribution under a fixed channel count.

The three stages filter out different types of detrimental updates and jointly determine the final channel count.

  • Knee-point filtering excludes channels with high forgetting costs, restricting the candidate set.
  • Pareto filtering jointly considers learning benefit and forgetting cost, excluding channels with less favorable benefit–cost trade-offs without requiring manually specified weighting coefficients. Whenever the candidate set is non-empty, a Pareto front necessarily exists. This stage therefore always retains candidate channels and cannot fail because “no Pareto front can be found.”
  • Otsu thresholding further excludes candidates with low benefit–cost efficiency and adaptively determines the final update budget. We have added the channel counts for each ablation setting on Llama3-8B under task order 1. This model has 724,992 candidate channels.
order1w/o Kneew/o Paretow/o Otsu
C-STANCE507863902904199369
FOMC488659472088258821
MeetingBank466760412211237760
ScienceQA470457402039777977
NumGLUE-cm506861961990188606
NumGLUE-ds507865772173769075
20Minuten460258002141757764
Py150469756251918117836

Removing any stage increases the number of selected channels. These results indicate that Pareto filtering influences not only update locations but also plays an important role in limiting update budget expansion.

Reviewer_zM62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SVC, a continual learning method that selectively updates singular-vector channels to balance learning and forgetting, with a focus on modern LLMs. It estimates potential learning potentials from task gradients and forgetting cost from output drift on model activations. Experiments on four models, eight TRACE tasks, and three task orders show strong results compared to competitive methods. Ablations also support the scoring and selection design made in the method.

Critical strengths
  1. SVC delivers strong downstream performance and code/math capability preservation.
  2. SVC enables finer control on learning-forgetting trade-off by allowing per-channel trade-off tuning and updating "already-learned" parameters.
Critical weaknesses

The comparison with OSFT is unclear. Published OSFT (ICLR version) also uses SVD-factor reparameterization and gradient hooks, while the appendix mainly describes full-weight gradient projection and changes the data used to estimate layer importance. Given the poor OSFT baseline results, these differences make it hard to assess SVC's advantage over OSFT.

Questions for authors
  1. Cont'd of above, does the OSFT baseline use the factor reparameterization and gradient hooks from the published version or the original authors' released implementation? If not, was equivalence verified (i.e. to see if the same published results can be obtained from your implementation)?
  2. Why does low-cost-only selection outperform full SVC on OP and BWT in table 2? Does full SVC consistently improve over this variant on other models or task orders?
  3. What are the actual trainable parameter counts for each method? If we fix the number of channels per layer, do the benefit score and three selection stages still provide consistent gains?
6 comments
AuthorsOpenReview ↗

Response to W1, Q1 (Part 1/2)

We thank the reviewer for highlighting the need to clarify our OSFT implementation and validate the comparison. Our original OSFT baseline implemented the SVD factor reparameterization and corresponding gradient hooks described in the original paper within our unified training and evaluation framework. We did not directly use the authors’ complete released codebase. This integration allowed us to standardize data preprocessing, training procedures, and evaluation interfaces across methods.

To address the reviewer’s concerns, we conducted two additional evaluations: (1) reproduction under the LLaMA-2-7B-Chat settings reported in the original paper, and (2) comparison using the authors’ released code and original importance estimation protocol. We present the reproduction results here and the second evaluation in Part 2.

Reproduction under the original LLaMA-2-7B-Chat settings.

OPBWTGSM8KC-STANCEFOMCMeetingBankPy150ScienceQANumGLUE-cmNumGLUE-ds20Minuten
Results reported in the original paper48.4-7.17.74258494867285144
Our reproduced results43.2-6.35.538.455227.4349.7868.420.9948.6239.54

The reproduction differences vary across tasks. MeetingBank exhibits the largest gap and accounts for most of the OP difference, whereas Py150, ScienceQA, and BWT exceed the originally reported values. Differences in the training framework, dependency versions, and inference and evaluation configurations may contribute to these discrepancies. Although the original paper reports training settings and states that it follows the TRACE task configurations, it does not fully specify all decoding parameters. We therefore cannot attribute the score differences to a single factor. The original paper also reports a decline in GSM8K performance after continual learning, consistent with the degradation in mathematical reasoning observed in our experiments.

Clarification of importance estimation in our original submission.

Our original comparison used a common external dataset to estimate layer importance for OSFT, reflecting a setting without access to actual training samples from previous tasks. This differs from the original OSFT protocol, which estimates importance using samples from the immediately preceding task. We acknowledge that this adaptation should have been stated explicitly.

With the external dataset, we observed similar layer importance scores across tasks, leading to similar rank allocations across layers. This may limit how well subspace allocation adapts to task differences. If the resulting updatable subspaces overlap substantially across tasks, interference between updates may also increase. We consider this a possible explanation for OSFT’s poorer performance in this setting, rather than an established causal conclusion.

To assess whether SVC’s advantage persists under the original OSFT protocol, we additionally evaluated the authors’ released implementation using actual training data from previous tasks for importance estimation, as detailed in Part 2.

AuthorsOpenReview ↗

Response to W1, Q1 (Part 2/2)

Comparison using the OSFT authors’ released implementation and original importance estimation protocol.

To address implementation fidelity and comparison fairness directly, we re-evaluated OSFT on all four models using the authors’ released code. Following the original importance estimation protocol, we used actual training samples from the immediately preceding task to estimate layer importance. The results under task order 1 are shown below.

order1
HumanEvalGSM8KOPBWT
Llama3-8B
OSFT 14.637.9640.54-7.13
SVC (ours)30.4951.5549.62-0.94
Mistral-7B
OSFT 02.2744.97-3.61
SVC (ours)27.4436.5448.37-0.37
Gemma2-9b
OSFT 40.2449.1350.16-2.31
SVC (ours)56.175.5150.25-1.37
Qwen3.5-9b
OSFT 21.9510.2451.02-2.94
SVC (ours)43.2972.2557.050.02

Using the authors’ implementation and original importance estimation protocol improved OSFT’s OP and BWT relative to the setting in our original submission. It also better preserved Gemma2-9B’s initial capabilities. These findings underscore the importance of explicitly distinguishing the external-data adaptation from the original OSFT protocol.

SVC nevertheless achieves higher HumanEval, GSM8K, OP, and BWT scores across all four models in this additional comparison. The results support SVC’s ability to preserve mathematical reasoning and programming capabilities while maintaining performance on sequentially learned tasks. SVC achieves these results without accessing actual training samples from previous tasks to estimate importance.

Taken together, these additional experiments show that SVC’s advantage persists across all four evaluated models under task order 1 when OSFT is evaluated using the authors’ released code and original importance estimation protocol.

AuthorsOpenReview ↗

Response to Q2 (Part 1/3)

We thank the reviewer for raising this question. Low C in Table 2 uses the channel count adaptively determined by full SVC for each task and each layer, and then selects that number of lowest-cost channels. It therefore evaluates channel selection under SVC-determined budgets; it does not independently determine those budgets.

SVC addresses two related decisions: which channels to select and how many to select. Learning benefit estimation contributes to this joint selection and allocation procedure. To clarify its role, we conducted three additional sets of experiments examining: (1) adaptive selection using cost alone, (2) selection strategies under matched budgets across models, and (3) fixed versus adaptive channel counts.

Adaptive selection using cost alone.

Using Llama3-8B under task order 1, we retained SVC’s knee-point filtering and Otsu thresholding steps, removed Pareto filtering based on the learning benefit–forgetting cost objectives, and used only cost information during Otsu thresholding. This variant, min_cost, independently determines the channel count instead of adopting the count determined by full SVC. All other training and evaluation settings remained unchanged.

order1
HumanEvalGSM8KOPBWT
Llama3-8B
min_cost001.86-12.86
SVC (ours)30.4951.5549.62-0.94

The cost-only variant performs substantially worse than full SVC across all four metrics. Under this selection rule, cost alone does not provide an effective replacement for SVC’s joint adaptive procedure. This experiment evaluates the specified cost-only variant; it does not establish that every possible cost-only allocation strategy would fail.

The strong performance of Low C in Table 2 should therefore be interpreted alongside its reliance on channel budgets determined by full SVC. Parts 2 and 3 further examine selection under matched budgets and whether fixed budgets can replace adaptive allocation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.