acceptodds
Under review as a conference paper at ICLR 2027

Shallow-Wide Fine-Tuning for Continual Learning with Pre-Trained Models

Abstract

Full fine-tuning of Pre-Trained Models (PTMs) on continually arriving downstream tasks often leads to catastrophic forgetting. Existing PTM-based Continual Learning (CL) methods typically freeze the backbone and attach auxiliary modules to preserve previously learned knowledge. However, frozen representations limit plasticity under large distribution shifts, and auxiliary modules incur additional inference overhead. To address these limitations, we propose Shallow-Wide Fine-Tuning (**SW-Tune**), which enables stable backbone updates without any auxiliary components. Inspired by the empirical insight that shallow-and-wide architectures exhibit inherently greater resilience to forgetting than deep-and-narrow ones, SW-Tune introduces a principled optimization framework that modulates the gradient subspace of a deep PTM to emulate shallow-and-wide learning dynamics without physical architectural changes. Specifically, *periodic block freezing* restricts trainable parameters to an equidistant, sparse subset of blocks, functionally reducing the effective trainable depth. Simultaneously, *gradient spectral flattening* regularizes the singular-value spectrum of layer-wise gradients, compelling parameter updates to span a broader orthogonal subspace, thereby expanding the effective capacity width. Extensive experiments demonstrate that SW-Tune outperforms existing PTM-based CL methods while introducing no additional inference overhead. Code is available at https://anonymous.4open.science/r/SW-Tune

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Incremental Learning, Pre-Trained Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iyr3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SW-Tune for exemplar-free class-incremental learning with pre-trained models. The method fine-tunes the backbone directly through two operations. Periodic Block Freezing (PBF) leaves a small set of evenly spaced Transformer blocks trainable and freezes the rest. Gradient Spectral Flattening (GSF) applies a power transformation to the singular values of linear-layer gradients, spreading update energy more evenly across the existing singular directions while preserving the Frobenius norm. The authors interpret PBF as reducing “effective depth” and GSF as increasing “effective width,” and combine both with an existing classifier alignment procedure.

SW-Tune improves performance on three visual benchmarks with distribution shifts, without adding auxiliary modules to the backbone at inference. However, the experiments offer stronger support for the method’s empirical effectiveness than for its proposed mechanism or its technical contribution relative to prior work.

Critical strengths

The paper addresses the practical challenge of adapting pre-trained models to new tasks in continual learning without introducing additional inference overhead. The proposed method is simple and achieves consistent improvements over existing approaches on three challenging benchmarks with substantial distribution shifts.

Critical weaknesses

The proposed method achieves promising results, but several aspects require further clarification. In particular, the connection to shallow-wide architectures and the mechanism by which GSF mitigates cross-task interference are not sufficiently established. The following concerns should be addressed.

  1. Insufficient justification for the shallow-wide interpretation. The central motivation is that shallow and wide networks are more resistant to catastrophic forgetting. However, PBF only reduces the number of trainable blocks without changing the actual network depth, while GSF modifies the gradient spectrum without increasing its exact rank. It remains unclear whether these operations reproduce the optimization properties of genuinely shallow and wide networks. Further theoretical or experimental evidence is needed to establish this connection.

  2. Insufficient evidence for the proposed mechanism of mitigating cross-task interference. The paper assumes that gradients are concentrated in a few dominant singular directions and that this concentration increases interference between tasks. However, the authors do not directly analyze the singular-value distributions of the original gradients or establish a relationship between gradient concentration and catastrophic forgetting. Moreover, the analysis in Figure 6 compares the complete SW-Tune framework with full fine-tuning, making it difficult to isolate the contribution of GSF. The absolute cosine similarity between task-level parameter updates also does not directly measure harmful gradient interference. More targeted experiments are needed to support the proposed mechanism.

  3. Insufficient discussion and comparison with null-space continual learning methods. Existing methods, such as Adam-NSCL and Gradient Projection Memory (GPM), mitigate cross-task interference by constraining parameter updates using subspaces derived from previous tasks. In contrast, GSF redistributes the current gradient across its existing singular directions without explicitly considering historical task information. It is therefore unclear why spectral flattening alone should reduce interference with previously learned knowledge. The paper should discuss the relationship between GSF and these methods, clarify their differences and potential advantages, and provide comparisons under consistent experimental settings.

Questions for authors

please refer to the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iyr3 (1/2)

We appreciate the reviewer's recognition of our method's simplicity and consistent improvements without additional inference overhead. To further address the reviewer's concerns, we have:

  • clarified the meaning of effective depth and width and provided additional architectural comparisons (for Weakness 1).
  • examined the relationship between spectral concentration and forgetting, and extended the mechanism analyses to all four component settings (for Weakness 2).
  • discussed the relationship to null-space continual learning methods and compared their adapted update rules under a common training setting (for Weakness 3).

Meanwhile, we have revised the paper according to the comments, and all the revisions made in response to your comments are highlighted in green in the PDF.

W1. Insufficient justification for the shallow-and-wide interpretation of PBF and GSF.

We thank the reviewer for this valuable comment regarding the evidence connecting our method to genuinely shallow and wide networks. We clarify that although our method does not explicitly change network width or depth, it can still obtain the stability benefits observed in shallow and wide networks. Specifically, we focus on two properties: representation stability and the spectral diversity of updates. To further examine this connection, we compare Swin-T with Swin-S at the same initial width, and Swin-S with Swin-L at the same total number of blocks. As shown below, CKA measures representation stability and energy-based effective rank measures the spectral diversity of updates; both are averaged over the last block of each stage. Compared with Swin-S, the shallower Swin-T exhibits higher CKA (0.9063 vs. 0.8391), whereas the wider Swin-L exhibits higher energy-based effective rank (43.61 vs. 36.40). In particular, when comparing Swin-S + SW-Tune with Swin-S, we observe analogous trends, suggesting that our method shares similarities with shallow and wide networks in terms of representation stability and the spectral diversity of updates.

ModelTotal blocksInitial widthCKA↑energy-based effective rank↑
Swin-S24960.839136.40
Swin-T12960.9063 (+0.0672)42.01
Swin-L241920.899943.61 (+7.21)
Swin-S + SW-Tune24960.8923 (+0.0532)49.87 (+13.47)

W2. Insufficient direct evidence linking gradient concentration, GSF, and harmful cross-task interference.

We thank the reviewer for the valuable comment on analyzing the proposed mechanism for mitigating cross-task interference. To address these points, we conducted the following additional analyses. In particular, we agree that the absolute cosine similarity between task-level parameter updates does not directly measure harmful gradient interference. Inspired by the protection of old-input feature subspaces in GPM [1] and Adam-NSCL [2], we replaced it with a measure of how much of the actual parameter-update energy falls into a subspace representing important old-task features. Specifically, for each matrix, the old-feature subspace is spanned by the leading eigenvectors of the old-task input second-moment matrix, with old tasks equally weighted and the basis truncated at 99% of the input-moment energy.

First, we keep PBF fixed and vary the exponent pp to examine the relationship between spectral concentration and forgetting on ImageNet-A. A smaller pp yields stronger flattening, while p=1p=1 leaves the gradient unchanged. As pp decreases, the energy-based effective rank increases from 57.65 to 197.29, while forgetting decreases from 19.82% to 8.96%. This shows that less concentrated updates are accompanied by less forgetting.

MethodEnergy-based effective rank↑Forgetting (%)↓
p=1 (without GSF)57.6519.82
p=0.7144.3216.08
p=0.5175.249.51
p=0.3197.298.96

Second, we extend the representation and spectral analyses to isolate the contribution of PBF and GSF. As shown in the table below, PBF accounts for much of the improvement in CKA, whereas GSF accounts for much of the improvement in energy-based effective rank and old-subspace update energy fraction. These comparisons distinguish the roles of the two components: PBF mainly improves representation stability, while GSF benefits the spectral diversity of updates. In particular, GSF lowers the old-subspace update energy fraction. This suggests that GSF directs a relatively larger share of update energy outside the important old-task features subspace, thereby reducing cross-task interference. We have added the above discussion in the revised manuscript.

MethodCKA B3↑CKA B7↑CKA B11↑Energy-based effective rank↑Old-subspace update energy fraction (%) ↓
Full FT0.98940.90150.836793.53695.06
GSF only0.98660.89500.8435188.483671.73
PBF only0.99970.93220.909957.650995.24
SW-Tune (PBF+GSF)0.99990.96020.9442149.978172.58
AuthorsOpenReview ↗

Response to Reviewer iyr3 (2/2)

W3. Relationship to and comparison with null-space methods such as Adam-NSCL and GPM.

We appreciate this suggestion and have expanded the related work discussion accordingly. If any important references have been omitted from the added discussion, please let us know and we will add them accordingly. In particular, GPM [1] and Adam-NSCL [2] use historical task information to restrict updates: GPM maintains bases of old activation subspaces, while Adam-NSCL uses historical feature statistics to identify approximate null-space directions. GSF differs in that it only transforms the current gradient and redistributes its energy, without maintaining a historical projection subspace. The old-feature-subspace projection experiment described above suggests that this redistribution can nevertheless reduce update overlap with old representations.

To compare the resulting balance between stability and plasticity, we evaluated them under a common setting. Specifically, all methods use PBF and SGD with momentum 0.9 on ImageNet-A. For the historical-subspace methods, the full SGD candidate update, including momentum, is projected. We therefore denote these implementations as GPM-SGD-adapted and NSCL-SGD-adapted.

The results below show that GSF and both projection methods reduce forgetting compared with vanilla SGD. The projection methods achieve lower AF, while GSF retains higher AOA (75.58 vs. 71.25 and 70.91). As a result, GSF achieves better overall LA and AIA, indicating that it reduces forgetting while preserving more capacity to learn new tasks. In addition, GSF requires no persistent historical projection state. The adapted GPM and NSCL implementations maintain approximately 73.00 MiB and 384.75 MiB of such state, respectively.

Update rule, with PBFLA (higher is better)AIA (higher is better)AF (lower is better)AOA (higher is better)Historical projection state (MiB)
Vanilla SGD60.7072.3219.8278.540
GSF-SGD65.1774.5511.5675.580
GPM-SGD-adapted64.8573.857.1271.2573.00
NSCL-SGD-adapted64.5473.617.0770.91384.75

[1] Saha, Gobinda, Isha Garg, and Kaushik Roy. "Gradient projection memory for continual learning." ICLR 2021. [2] Wang, Shipeng, et al. "Training networks in null space of feature covariance for continual learning." CVPR 2021.

Thank you for the additional experiments and responses.

Some of my concerns have addressed, I will maintain my original score.

We are grateful for your valuable comments and your recognition of our work. Your insightful suggestions have helped significantly improve the quality of our manuscript. Thank you very much for your time and effort in reviewing our work.

Reviewer_PyhAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SW-Tune a framework for class-incremental learning using pre-trained models. Instead of adding adapter layers to a freezed backbone, SW-Tune mimics the configuration of a “shallow-and-wide” architecture by means of (1) a Periodic Block Freezing and (2) a Gradient Spectral Flattening. Experiments on three benchmarks demonstrate that SW-Tune outperforms existing approaches while introducing zero inference overhead.

Critical strengths
  • It’s interesting to see how the authors translated insights from network geometry into an optimization-level intervention.
  • The proposed approach has no computational overhead at inference time.
  • The paper is well structured and well written. The analysis and the comprehensive evaluation demonstrate the benefits of the proposed approach.
Critical weaknesses
  • The choice of K=3 is justified via an ablation study on a single model architecture and it lacks theoretical grounding. It is not clear if this choice scales universally to deeper or different architectures and to text-based transformers.
  • While the inference phase has very low overhead, the training phase takes longer time and more memory compared to other methods.
  • The evaluation is restricted to a single domain (image classification) and “similar” tasks. Although the authors clarified in A.2 the benchmark selection, more different tasks such as medical images or satellite images could give insights on the generalizability of the proposed approach.
Questions for authors
  • Is K a parameter that strictly depends on the model architecture used? How does the depth ratio (K/L) scales with deeper models? Does K also depends on the nature of tasks, e.g., fine-grained or coarse-grained differences between tasks?
  • Is GSF effective also when applied across all layers or only when restricted to the K trainable blocks?
  • How sensitive is the annealing schedule of p in GSF if the total number of tasks is unknown a priori?
  • For baselines the authors choose the default hyperparameter configurations while aligning batch size and number of epochs to SW-Tune. Is this fair considering that the original hyperparameters were optimized for different choices of batch size and epoch number?
  • Are the results in the tables statistically significant?
  • Can the proposed approach be extended to different modalities such as text?
5 comments
AuthorsOpenReview ↗

Response to Reviewer PyhA (1/3)

We appreciate the reviewer's recognition of our research perspective, inference efficiency, and presentation. To further address the reviewer's concerns, we have:

  • evaluated the same trainable-block ratio on a deeper ViT to examine the transferability of our choice of KK (for Weakness 1, Question 1).
  • provided further analysis of training time, memory consumption, and the inference savings (for Weakness 2).
  • included experiments on multimodal continual question answering involving medical and satellite images (for Weakness 3, Question 6).
  • clarified the application of GSF with and without PBF (for Question 2).
  • evaluated a fixed-pp alternative for unknown-length task streams (for Question 3).
  • examined the effect of adjusting the strongest baselines' learning rates to the shared batch size (for Question 4).
  • provided per-order results and paired significance tests (for Question 5).

Meanwhile, we have revised the paper according to the comments, and all the revisions made in response to your comments are highlighted in blue or purple in the PDF.

W1 & Q1. Transferability of the choice of KK

We thank the reviewer for raising this point. We acknowledge that K=3K=3 is an empirical choice for the 12-block ViT used in our main experiments, and we agree that its transferability to deeper or different architectures deserves clarification. A natural way to transfer this choice to deeper models is to retain the same proportion of trainable blocks, i.e., K/L=0.25K/L=0.25. To examine this, we have evaluated SW-Tune on the 24-block ViT-L/16-IN1K with K=6K=6, keeping the ratio K/L=0.25K/L=0.25 unchanged. The results on ImageNet-A are shown below. SW-Tune outperforms the strongest baseline by 4.74 pp in LA and 4.13 pp in AIA, suggesting that the same depth ratio works well on a deeper backbone. We also use K/L=0.25K/L=0.25 in the multimodal experiments reported in our response to W3/Q6. While the optimal value of KK may still depend on the nature of the tasks and model architecture, K/L=0.25K/L=0.25 serves as a practical default without task-specific tuning. We have clarified this point and included the related results in the revised manuscript.

MethodsLAAIA
L2P58.1366.43
DualPrompt57.1465.59
CODA-Prompt55.0466.01
InfLoRA65.3772.44
MOS65.9673.56
SSIAT67.4175.69
TUNA67.1575.82
SW-Tune (ours)72.1579.95
Improvements to the strongest baseline+4.74+4.13

W2. Higher training cost despite low inference overhead.

We thank the reviewer for this comment. As acknowledged in the Limitations section, SW-Tune requires more training time than some methods. Following a related suggestion from Reviewer 7bS1, we revisited the training-time measurements and found that the timing scopes in the original appendix were not fully aligned across methods—some entries included incidental debugging overhead. We have therefore recomputed all training times under a unified protocol and updated the appendix accordingly. This issue does not affect the memory-consumption or inference measurements reported in the paper.

On ViT-B, SW-Tune indeed takes more training time than some baselines. However, when scaled to the larger ViT-L backbone, SW-Tune already shows nearly the lowest training time among the baselines, indicating that the additional training cost does not hinder its scalability. For memory consumption, SW-Tune uses 15.2 GB on ViT-B, the same as SSIAT and TUNA. This is lower than L2P (16.0 GB), CODA-Prompt (19.5 GB), SLCA (20.9 GB), and SLCA++ (17.7 GB), while MOS, DualPrompt, and InfLoRA use less memory (12.0, 13.2, and 13.6 GB). Thus, its memory consumption is comparable to several strong baselines.

MethodsViT-B (RTX 3090)ViT-L (RTX PRO 6000)
L2P6521356
DualPrompt6151275
CODA-P6751240
InfLoRA10551580
SSIAT7501208
MOS8421529
TUNA10211591
SLCA9231533
SLCA++9281546
SW-Tune8871226

For ViT-B, we further provide estimates of how many inference images are needed to offset the extra training time. The results show that the extra training time is offset after processing at most 53,334 images.

MethodTrain Time (s)Inference Latency (ms)Inference images to offset extra training time
SW-Tune8876.99-
L2P65211.8248,655
DualPrompt61512.0953,334
CODA-P67513.7631,315
InfLoRA105512.930
SSIAT75012.0826,916
MOS842363.34127
TUNA1021132.530

Overall, we would like to emphasize that although SW-Tune incurs additional training cost relative to some methods, this cost is justified by its better performance and higher inference efficiency. We have explicitly discussed this performance–cost trade-off of SW-Tune in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer PyhA (2/3)

W3 & Q6. Generalization on medical/satellite tasks & beyond image modality.

We thank the reviewer for this helpful suggestion. To evaluate our method beyond the image-classification setting, we conducted additional experiments on multimodal continual question answering using LLaVA-v1.5-7B while retaining the same K/L=0.25K/L=0.25 setting. We adopt a four-task subset of the benchmark established in MLLM-CL [1], where the model sequentially learns remote sensing (satellite), medical, autonomous driving, and science tasks. The results below are measured after learning all four tasks without replay data.

We would like to emphasize that MR-LoRA, the method introduced together with this benchmark, relies on an inference-time LLM-based routing mechanism. It is therefore orthogonal to our method, which focuses on the training process. As shown below, SW-Tune performs better than all baselines that focus on the training process on each of the four tasks. These results suggest that SW-Tune can also benefit tasks involving medical and satellite images, as well as multimodal question answering with text inputs/outputs. We have included these experiments and a discussion of related text-based/multimodal continual-learning work in the revised manuscript.

Methodsremote sensingmedicalautonomous drivingscience
MR-LoRA81.0665.6854.1756.11
LoRA-FT75.1245.5638.4649.44
O-LoRA76.6044.8740.5750.58
MoELoRA76.8743.7932.8148.67
CL-MoE71.0248.0437.7050.28
HiDe75.4344.9138.3346.44
SEFE77.8347.7043.0149.04
DISCO76.1145.7649.2649.23
SW-Tune (ours)79.8658.8753.3554.27

[1] Zhao, Hongbo, et al. "Mllm-cl: Continual learning for multimodal large language models." 2025.

Q2. Effectiveness of GSF across all layers versus only the trainable blocks.

We thank the reviewer for this question. We would like to clarify that, since the frozen blocks receive no gradients, applying GSF to all backbone layers is equivalent to applying it only to the trainable blocks.

Q3. Sensitivity of the annealing schedule when the total number of tasks is unknown.

We thank the reviewer for this thoughtful comment and for advocating a more realistic continual-learning setting. When TT is unknown, one can simply use a fixed exponent, such as p=0.5p=0.5 (except p=1p=1 for the first task). We compare this fixed setting with the original schedule below. The results show that the fixed schedule performs similarly to the original schedule (or even slightly better), and both significantly outperform the baseline without GSF. This suggests that the improvement mainly comes from applying GSF rather than from the particular annealing schedule. We have clarified this distinction and included the fixed-pp results to explain how SW-Tune can be used when TT is unknown in this revision.

MethodImageNet-R-LAImageNet-R-AIAImageNet-A-LAImageNet-A-AIAObjectNet-LAObjectNet-AIA
Without GSF (p=1p=1; PBF only)79.53±0.4084.40±0.3461.99±0.7670.45±1.3759.39±0.7070.47±1.06
Original schedule (p:1→0p:1\rightarrow0)81.25±0.2585.25±0.3665.24±0.1572.00±1.5165.23±0.4173.01±1.11
Fixed p=0.5p=0.5 (except p=1p=1 for the first task)81.36±0.2085.27±0.1966.20±0.3772.73±1.5965.74±0.5374.16±1.21
AuthorsOpenReview ↗

Response to Reviewer PyhA (3/3)

Q4. Fairness of default baseline hyperparameters under modified batch sizes and training epochs.

We thank the reviewer for raising this concern. We emphasize that we adopted the same batch size and number of epochs to ensure a fair comparison, particularly by keeping the training budget roughly comparable. We acknowledge that changing these settings can affect the appropriate choices of other hyperparameters. We therefore additionally compare SSIAT and TUNA with adjusted hyperparameters on ObjectNet, where the original performance gap is smallest, to examine whether this significantly affects our comparison conclusions. We emphasize that the default epoch numbers of these two methods are already aligned; we only changed the batch size. Referring to existing studies [1], batch size mainly affects the choice of learning rate, so we scale the learning rate in proportion to the batch-size change from the source configuration. We also evaluate SW-Tune with half of its default learning rate. The results are presented below. TUNA improves by 1.63 pp after the adjustment, but its LA of 63.99 remains below the 65.23 obtained by SW-Tune with the default learning rate. SW-Tune also improves to 66.15 with an adjusted learning rate, showing that its default setting is not finely optimized either. These results suggest that the advantage of SW-Tune is maintained after adjusting the batch-size-related hyperparameters.

MethodsLA with default lrLA with adjusted lr
SSIAT64.63±0.2864.48±0.27
TUNA62.36±0.3963.99±0.33
SW-Tune65.23±0.4166.15±0.41

[1] Goyal, Priya, et al. "Accurate, large minibatch sgd: Training imagenet in 1 hour." 2017.

Q5. Statistical significance of the reported results.

Thank you for the suggestion. Since task order can affect performance, we compare SW-Tune with the strongest baseline in the original Table 1 under five identical class orders, using two-sided paired t-tests on LA. The results show that SW-Tune improves over the corresponding baseline under every one of the five task orders on all three datasets. The paired t-tests give p<0.05p<0.05 on all three datasets, which supports the statistical significance.

ImageNet-R

Methods1s2s3s4s5
SSIAT78.9578.6878.8378.3379.00
SW-Tune(Ours)81.4381.0381.3781.5380.87
Paired difference+2.48+2.35+2.54+3.20+1.87

ImageNet-A

Methods1s2s3s4s5
TUNA60.5761.5559.7161.2960.37
SW-Tune(Ours)65.1765.1165.5065.3165.11
Paired difference+4.60+3.56+5.79+4.02+4.74

ObjectNet

Methods1s2s3s4s5
SSIAT64.7764.2364.9764.3864.82
SW-Tune(Ours)65.9665.0065.3464.7765.09
Paired difference+1.19+0.77+0.37+0.39+0.27

Paired-test summary

DatasetBaselineMean paired improvement (pp)Paired t(4)t(4)pp
ImageNet-RSSIAT+2.48811.660.000309
ImageNet-ATUNA+4.54212.060.000271
ObjectNetSSIAT+0.5983.500.024831
Reviewer_7bS1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SW-Tune, an exemplar-free class-incremental learning method that fine-tunes a pre-trained ViT backbone without introducing auxiliary modules, so that inference uses only the ViT backbone. It combines Periodic Block Freezing (PBF), which updates three evenly distributed Transformer blocks in a 12-block ViT, with Gradient Spectral Flattening (GSF), which flattens and norm-rescales the singular-value spectrum of gradients in those blocks. The authors interpret these operations as reducing effective trainable depth and broadening the directions receiving update energy. Across ImageNet-R, ImageNet-A, and ObjectNet class-incremental benchmarks, SW-Tune reports higher accuracy on the reported benchmarks than the evaluated prompt-, adapter-, and other auxiliary-module-based baselines; a separate comparison on ImageNet-A also favors it over SLCA and SLCA++. Ablations suggest benefits from combining PBF and GSF and from distributing trainable blocks across the backbone. On ImageNet-A, the reported parameter and representation analyses show higher cumulative-update effective rank, lower similarity between the first and last task updates, and higher CKA similarity across checkpoints than full fine-tuning.

Critical strengths
  1. The method is simple yet effective
  2. The experiments are reasonably comprehensive. Three benchmarks × two backbones × five class orders with standard deviations; ablations cover components, layer layouts, and values of K; it reports an AOA/AF decomposition rather than only LA; it includes a direct comparison with SLCA/SLCA++ under the same paradigm; and training overhead is provided in the appendix.
Critical weaknesses
  1. The annealing schedule for pp requires knowing the total number of tasks �0��0� in advance, since pt=1−t−1T−1p_t = 1 - \frac{t-1}{T-1} depends explicitly on TT. A basic premise of continual learning is that the length of the task stream is unknown, so this is conceptually problematic.
  2. Several key baselines are missing. In particular, several of the strongest PTM-based CIL methods from the past two years are absent: RanPAC, EASE, InfLoRA, APER/ADAM. More importantly, the authors cite Lu et al. (2026), ``Adapt before Continual Learning,'' which likewise emphasizes plasticity, yet do not compare against it.
  3. Training cost is nearly twice that of L2P, and the method requires one Gram-matrix eigendecomposition per linear layer per optimization step. There is no analysis of how this scales to larger models.
Questions for authors
  1. As I stated earlier, can you compare with more baselines?
  2. Figure 4 already compares the full method with single-component methods. Could you please report these variants in a table with their final LA/AIA under the same five class orders (mean, standard deviation, and paired differences), and state whether all ablations use the same head alignment? Please also report the performance of full SW-Tune without head alignment to separate the contributions of the training strategy and the post-processing.
  3. Table 4 shows that the training time of SW-Tune is 1.85 times that of L2P. Can you provide a layer-wise cost breakdown and an estimate of the inference volume needed to offset the additional training cost? If training with a larger model, can the inference benefits offset the training cost?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 7bS1 (1/2)

We appreciate the reviewer's recognition of the simplicity and effectiveness of our method, as well as the comprehensive evaluation. To further address the reviewer's concerns, we have:

  • evaluated a fixed-pp setting that does not require knowing the task-stream length (for Weakness 1).
  • included comparisons with additional PTM-based CIL methods (for Weakness 2, Question 1).
  • examined the training cost on ViT-B and ViT-L, provided a runtime breakdown, and estimated the inference volume needed to offset the extra training time (for Weakness 3, Question 3).
  • reported component ablations under the same five class orders and additional comparisons without head alignment (for Question 2).

Meanwhile, we have revised the paper according to the comments, and all the revisions made in response to your comments are highlighted in red or purple in the PDF.

W1. Dependence of the annealing schedule for pp on the known task-stream length TT.

We thank the reviewer for this thoughtful comment and for advocating a more realistic continual-learning setting. We agree that the original annealing schedule uses the total number of tasks TT, which can be inconvenient when the task-stream length is unknown. We would like to clarify, however, that this is a relatively minor practical design choice in the schedule, and the core mechanism of our methods (PBF/GSF) does not rely on this information. When TT is unknown, one can simply use a fixed exponent, such as p=0.5p=0.5 (except p=1p=1 for the first task). We compare this fixed setting with the original schedule below. The results show that the fixed schedule performs similarly to the original schedule (or even slightly better), and both significantly outperform the baseline without GSF. This suggests that the improvement mainly comes from applying GSF rather than from the particular annealing schedule. We have clarified this distinction and included the fixed-pp results to explain how SW-Tune can be used when TT is unknown in this revision.

MethodImageNet-R-LAImageNet-R-AIAImageNet-A-LAImageNet-A-AIAObjectNet-LAObjectNet-AIA
Without GSF (fixed p=1p=1; PBF only)79.53±0.4084.40±0.3461.99±0.7670.45±1.3759.39±0.7070.47±1.06
Original schedule (p:1→0p:1\rightarrow0)81.25±0.2585.25±0.3665.24±0.1572.00±1.5165.23±0.4173.01±1.11
Fixed p=0.5p=0.5 (except p=1p=1 for the first task)81.36±0.2085.27±0.1966.20±0.3772.73±1.5965.74±0.5374.16±1.21

W2 & Q1. Comparison with more baselines.

Thank you for suggesting these relevant and key comparisons. While our main text already includes recent methods such as MOS (AAAI 2025) and TUNA (ICCV 2025), we have added evaluations on all the methods you mentioned to better demonstrate the advantages of our approach. In particular, Adapt before Continual Learning (ACL) introduces an adaptation stage before the core learning process. It is a plug-and-play component and is therefore orthogonal to our method, which focuses on the core continual-learning stage. Nevertheless, we provide as a reference its combination with SSIAT, the strongest baseline in our original comparison. The results show that SW-Tune achieves the highest performance on all three datasets. This further supports the effectiveness of SW-Tune.

In addition, we have expanded the Related Work section to discuss the works mentioned above, so that the positioning of our method with respect to these works is clearer. If any important references have been omitted from the added discussion, please let us know and we will add them accordingly.

MethodINR-LAINR-AIAINA-LAINA-AIAOBJ-LAOBJ-AIA
RanPAC76.07±0.8581.18±0.9458.16±0.4666.73±1.4763.79±0.1271.98±1.47
EASE68.32±0.2474.34±0.7548.89±0.4258.03±1.2055.14±0.3364.23±1.64
InfLoRA77.74±0.2681.78±0.5758.76±1.4165.52±1.4357.53±1.3067.62±1.28
APER with adapter67.25±1.2173.13±1.5449.22±0.0358.37±1.1755.52±0.2264.12±1.40
SSIAT + ACL79.13±0.2282.80±0.3363.91±0.3969.84±1.4265.22±0.2672.95±1.24
SW-Tune81.25±0.2585.25±0.3665.24±0.1572.00±1.5165.23±0.4173.01±1.11
AuthorsOpenReview ↗

Response to Reviewer 7bS1 (2/2)

W3 & Q3. Further analysis on training overhead

We appreciate the reviewer’s constructive suggestion to provide a cost breakdown and to analyze how the training cost scales to larger models. This suggestion prompted us to conduct a more fine-grained cost analysis. In doing so, we found that the timing scopes in the original appendix were not fully aligned across methods—some entries included incidental debugging overhead. We have therefore recomputed all training times under a unified protocol and updated the appendix accordingly. This issue does not affect the memory-consumption or inference measurements reported in the paper. We would like to thank the reviewer for the suggestion that made us aware of this oversight.

The following table reports training time in seconds and the component-wise cost breakdown on ImageNet-A Inc20 under a consistent scope. In particular, the ViT-L experiment provides an additional evaluation of this cost on a larger backbone. On ViT-B, SW-Tune takes 1.36× the training time of L2P, while on ViT-L its training time is lower than L2P and close to SSIAT. Further cost breakdown shows that most of the time is spent on core training and head alignment, while GSF accounts for 6.35% on ViT-B and 10.07% on ViT-L. These results indicate that the additional training cost of SW-Tune does not hinder its scalability.

MethodsViT-B (RTX 3090)ViT-L (RTX PRO 6000)
L2P6521356
DualPrompt6151275
CODA-P6751240
InfLoRA10551580
SSIAT7501208
MOS8421529
TUNA10211591
SLCA9231533
SLCA++9281546
SW-Tune8871226
SW-Tune / L2P1.36x0.90x
ItemTime Proportion with ViT-BTime Proportion with ViT-L
Core Training44.55%55.62%
Head Alignment43.52%26.35%
GSF6.35%10.07%
Other5.58%7.96%

Regarding how much inference is needed to offset the extra training time, we provide the estimates below using ViT-B. The results show that the extra training time is offset after processing at most 53,334 images. On the larger ViT-L backbone, SW-Tune already shows nearly the lowest training time among the baselines. Overall, we would like to emphasize that although SW-Tune incurs additional training cost relative to some methods, this cost is justified by its better performance and higher inference efficiency. We have added the above discussion in the revised manuscript.

MethodTrain Time (s)Inference Latency (ms)Inference images to offset extra training time
SW-Tune8876.99-
L2P65211.8248,655
DualPrompt61512.0953,334
CODA-P67513.7631,315
InfLoRA105512.930
SSIAT75012.0826,916
MOS842363.34127
TUNA1021132.530

Q2. Ablations over five class orders, and the contribution of head alignment.

The table below reports the performance of the single-component variants under the same five class orders. For ease of comparison, we also list the mean paired improvements of full SW-Tune over each ablation variant. All ablations use the same head alignment and other training settings, so that the analysis isolates the independent contributions of the core mechanisms. Both PBF and GSF improve over full fine-tuning, and combining them gives the best results on all three datasets. These results indicate that both components of our method can effectively improve continual learning performance and can be effectively combined.

MethodINR-LAINR-AIAINA-LAINA-AIAOBJ-LAOBJ-AIA
Full fine-tuning75.43±0.5482.72±0.5453.79±0.6065.27±1.0551.93±0.7765.08±0.97
PBF only79.53±0.4084.40±0.3461.99±0.7670.45±1.3759.39±0.7070.47±1.06
GSF only76.79±0.4583.18±0.5456.48±1.0866.44±1.0253.76±0.5966.03±1.00
SW-Tune81.25±0.2585.25±0.3665.24±0.1572.00±1.5165.23±0.4173.01±1.11
Compared withINR LA gainINR AIA gainINA LA gainINA AIA gainOBJ LA gainOBJ AIA gain
Full fine-tuning+5.82+2.53+11.45+6.73+13.30+7.93
PBF only+1.72+0.85+3.25+1.55+5.84+2.54
GSF only+4.46+2.07+8.76+5.56+11.47+6.98

Regarding head alignment, we would like to clarify that it is a widely adopted post-processing step in existing methods, and the comparisons remain fair. To further separate the contribution of the training strategy from this post-processing, as requested, we report full SW-Tune without head alignment, together with three strong baselines under the same no-head-alignment setting on ImageNet-A. The results show that SW-Tune still outperforms the strongest baseline, TUNA, by 3.76 pp in LA and 4.64 pp in AIA. This indicates that head alignment is useful, but the advantage of SW-Tune is also clearly present when it is removed from all compared methods, i.e., the gain is not merely due to head alignment. We have added the above discussion in the revised manuscript.

MethodINA-LAINA-AIA
SSIAT50.28±0.7359.75±0.88
MOS44.85±2.4855.30±2.06
TUNA52.58±1.2260.82±1.53
SW-Tune56.34±0.5765.46±1.74
Reviewer_7bS1OpenReview ↗

Response to the Rebuttal

Dear authors, Thank you for the extra explanation and experiments. Your additional explanation and experiments have resolved most of my concerns. I have one last concern about the time cost. Could you clarify which timing boundaries differed in the original table and what was included in the “incidental debugging overhead”? The revisions affect methods by very different amounts: SW-Tune changes from 1,413 to 887 seconds and SSIAT from 1,302 to 750 seconds, whereas L2P changes from 765 to 652 seconds and MOS remains at 842 seconds. I wonder why, across methods, the “incidental debugging overhead” affects the time differently.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.