- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the balance between learning new tasks and preserving existing capabilities in continual adaptation of large language models. The proposed method, SVC, estimates the adaptation benefit and potential forgetting cost of singular-vector channels using current-task gradients and activation statistics from a public reference corpus. It then selects channels through a three-stage filtering procedure, updating the selected singular vectors alongside all singular values. The method does not store previous-task examples or accumulate task-specific modules. Experiments across four models, eight continual-learning tasks, and three task orders show competitive downstream performance and improved retention of previously acquired capabilities.
Critical strengths
-
Relevant problem and clear motivation. Selectively allocating updates to balance adaptation and knowledge preservation is a practically meaningful direction. Considering both potential learning benefits and forgetting risks provides an interpretable rationale for selecting which parts of the model to update.
-
Broad experimental coverage and competitive results. Evaluation across multiple models and task orders provides useful evidence of the method’s effectiveness. The experiments assess both continual-task performance and capability retention.
Critical weaknesses
-
The additional value of the complete selection procedure is not sufficiently established. SVC combines several selection stages, but the current evidence does not clearly demonstrate their necessity relative to simpler alternatives. On Llama3-8B under Order 1, the low-cost selection variant in Table 2 matches SVC’s channel count and achieves better OP and BWT, while obtaining similar capability-retention results. Moreover, the stage-removal ablations in Figure 3 also change the number of selected channels. Consequently, they do not fully distinguish the benefit of the selection strategy from the effect of changing the update budget. Clearer evidence separating these factors would strengthen the paper’s central methodological contribution.
-
SVF, which directly motivates the proposed method, is evaluated in the independent-task motivation experiment but lacks a complete continual-learning comparison. The related spectral adaptation method SVFT [1] is also absent from the experimental comparisons. Additionally, the cited PIECE method [2] studies task-relevant parameter selection for continual adaptation and capability preservation, making it another relevant baseline. Comparisons with these methods would help establish the benefits of SVC over closely related approaches.
-
SVC and the main PEFT baselines also differ in their update scope and parameter allocation, yet the actual trainable parameter counts of SVC are not reported. Appendix D partially addresses the difference in module coverage, but does not establish comparability under matched parameter budgets. In my opinion, reference-corpus processing, matrix decomposition, and channel scoring introduce additional computation, without corresponding runtime or peak-memory measurements. These omissions limit the assessment of the method’s practical efficiency.
[1] SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors. [2] Parameter Importance-Driven Continual Learning for Foundation Models.
Questions for authors
-
How should the strong performance of the simple low-cost selection variant in Table 2 be interpreted? Which existing results best establish the additional value of the complete selection procedure?
-
Could the authors clarify the omission of full continual-learning comparisons with SVF, SVFT, and PIECE, and indicate whether comparable results are already available?
-
What are the actual trainable parameter counts across models and tasks, and how do channel counts change in the Figure 3 ablations?
-
Are capability-evaluation results available before training and after individual tasks? These would help distinguish preservation of existing capabilities from transfer benefits acquired during the task sequence.
8 comments
Response to W1, Q1 (Part 2/3)
Second, we compared selection strategies across four models under the same channel budgets. For each task and each layer, min_cost_fix_size uses the channel count determined by full SVC and replaces only the channel selection strategy with a ranking that selects the lowest-cost channels. All experiments used task order 1, with all other training and evaluation settings held constant.
| order1 | ||||
|---|---|---|---|---|
| HumanEval | GSM8K | OP | BWT | |
| Llama3-8B | ||||
| min_cost_fix_size | 29.27 | 51.4 | 51.12 | -0.20 |
| SVC (ours) | 30.49 | 51.55 | 49.62 | -0.94 |
| Mistral-7B | ||||
| min_cost_fix_size | 27.44 | 39.58 | 48.04 | -0.39 |
| SVC (ours) | 27.44 | 36.54 | 48.37 | -0.37 |
| Gemma2-9b | ||||
| min_cost_fix_size | 53.66 | 77.86 | 50.94 | -0.47 |
| SVC (ours) | 56.1 | 75.51 | 50.25 | -1.37 |
| Qwen3.5-9b | ||||
| min_cost_fix_size | 50.61 | 69.75 | 55.00 | -1.15 |
| SVC (ours) | 43.29 | 72.25 | 57.05 | 0.02 |
The selection strategy still affects the results when the update budget is held constant. Low C’s advantages in OP and BWT are not consistent across models.
Response to W1, Q1 (Part 3/3)
Finally, we examined whether fixed channel counts can replace adaptive allocation across tasks. Using Llama3-8B under task order 1, we tested two fixed budgets corresponding to the minimum and maximum channel counts determined by full SVC across tasks. Both variants always selected the lowest-cost channels. AP denotes the average performance across tasks, with each task evaluated immediately after it is learned.
| order1 | |||||
|---|---|---|---|---|---|
| HumanEval | GSM8K | OP | BWT | AP | |
| Llama3-8B | |||||
| min_cost_fix_size_task_min | 29.27 | 51.18 | 49.32 | -0.55 | 49.80 |
| min_cost_fix_size_task_max | 21.95 | 48.98 | 49.43 | -1.24 | 50.52 |
| SVC (ours) | 30.49 | 51.55 | 49.62 | -0.94 | 50.44 |
Full SVC achieves higher HumanEval, GSM8K, and OP scores than both fixed-count variants. Fixing the channel count at a lower value reduces forgetting but limits new-task learning. Fixing it at a higher value increases AP, but mathematical and programming capabilities and BWT all decline. SVC achieves an AP close to that obtained with the fixed maximum budget, together with higher OP and better capability preservation, supporting adaptive allocation of channel counts across tasks.
These results show that SVC’s advantages arise not only from selecting where to apply updates, but also from adaptively determining how many channels to train. Joint evaluation of learning benefit and forgetting cost is key to this allocation.
Response to W1, Q1 (Part 1/3)
We thank the reviewer for raising this question. Low C in Table 2 uses the number of channels adaptively determined by full SVC for each task and each layer, and then selects the same number of lowest-cost channels. Its strong performance supports the effectiveness of cost scoring, but does not establish that lowest-cost selection can independently replace the full procedure. SVC must determine both which channels to select and how many to select, and learning benefit estimation provides an important basis for this adaptive allocation of channel counts.
To further distinguish the effects of channel selection under a given budget from those of autonomous budget allocation, we conducted the following three sets of additional experiments.
First, we examined whether cost alone can support adaptive selection of both channels and their number. Using Llama3-8B under task order 1, we retained SVC’s knee-point filtering and Otsu thresholding steps, removed the Pareto filtering based on the learning benefit–forgetting cost objectives, and used only cost information during Otsu thresholding. This variant independently determines the number of channels instead of adopting the counts determined by full SVC. All other training and evaluation settings remained unchanged.
Under this selection rule, cost alone cannot replace the full adaptive SVC procedure.