- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SW-Tune for exemplar-free class-incremental learning with pre-trained models. The method fine-tunes the backbone directly through two operations. Periodic Block Freezing (PBF) leaves a small set of evenly spaced Transformer blocks trainable and freezes the rest. Gradient Spectral Flattening (GSF) applies a power transformation to the singular values of linear-layer gradients, spreading update energy more evenly across the existing singular directions while preserving the Frobenius norm. The authors interpret PBF as reducing “effective depth” and GSF as increasing “effective width,” and combine both with an existing classifier alignment procedure.
SW-Tune improves performance on three visual benchmarks with distribution shifts, without adding auxiliary modules to the backbone at inference. However, the experiments offer stronger support for the method’s empirical effectiveness than for its proposed mechanism or its technical contribution relative to prior work.
Critical strengths
The paper addresses the practical challenge of adapting pre-trained models to new tasks in continual learning without introducing additional inference overhead. The proposed method is simple and achieves consistent improvements over existing approaches on three challenging benchmarks with substantial distribution shifts.
Critical weaknesses
The proposed method achieves promising results, but several aspects require further clarification. In particular, the connection to shallow-wide architectures and the mechanism by which GSF mitigates cross-task interference are not sufficiently established. The following concerns should be addressed.
-
Insufficient justification for the shallow-wide interpretation. The central motivation is that shallow and wide networks are more resistant to catastrophic forgetting. However, PBF only reduces the number of trainable blocks without changing the actual network depth, while GSF modifies the gradient spectrum without increasing its exact rank. It remains unclear whether these operations reproduce the optimization properties of genuinely shallow and wide networks. Further theoretical or experimental evidence is needed to establish this connection.
-
Insufficient evidence for the proposed mechanism of mitigating cross-task interference. The paper assumes that gradients are concentrated in a few dominant singular directions and that this concentration increases interference between tasks. However, the authors do not directly analyze the singular-value distributions of the original gradients or establish a relationship between gradient concentration and catastrophic forgetting. Moreover, the analysis in Figure 6 compares the complete SW-Tune framework with full fine-tuning, making it difficult to isolate the contribution of GSF. The absolute cosine similarity between task-level parameter updates also does not directly measure harmful gradient interference. More targeted experiments are needed to support the proposed mechanism.
-
Insufficient discussion and comparison with null-space continual learning methods. Existing methods, such as Adam-NSCL and Gradient Projection Memory (GPM), mitigate cross-task interference by constraining parameter updates using subspaces derived from previous tasks. In contrast, GSF redistributes the current gradient across its existing singular directions without explicitly considering historical task information. It is therefore unclear why spectral flattening alone should reduce interference with previously learned knowledge. The paper should discuss the relationship between GSF and these methods, clarify their differences and potential advantages, and provide comparisons under consistent experimental settings.
Questions for authors
please refer to the weakness.
4 comments
Response to Reviewer iyr3 (2/2)
W3. Relationship to and comparison with null-space methods such as Adam-NSCL and GPM.
We appreciate this suggestion and have expanded the related work discussion accordingly. If any important references have been omitted from the added discussion, please let us know and we will add them accordingly. In particular, GPM [1] and Adam-NSCL [2] use historical task information to restrict updates: GPM maintains bases of old activation subspaces, while Adam-NSCL uses historical feature statistics to identify approximate null-space directions. GSF differs in that it only transforms the current gradient and redistributes its energy, without maintaining a historical projection subspace. The old-feature-subspace projection experiment described above suggests that this redistribution can nevertheless reduce update overlap with old representations.
To compare the resulting balance between stability and plasticity, we evaluated them under a common setting. Specifically, all methods use PBF and SGD with momentum 0.9 on ImageNet-A. For the historical-subspace methods, the full SGD candidate update, including momentum, is projected. We therefore denote these implementations as GPM-SGD-adapted and NSCL-SGD-adapted.
The results below show that GSF and both projection methods reduce forgetting compared with vanilla SGD. The projection methods achieve lower AF, while GSF retains higher AOA (75.58 vs. 71.25 and 70.91). As a result, GSF achieves better overall LA and AIA, indicating that it reduces forgetting while preserving more capacity to learn new tasks. In addition, GSF requires no persistent historical projection state. The adapted GPM and NSCL implementations maintain approximately 73.00 MiB and 384.75 MiB of such state, respectively.
| Update rule, with PBF | LA (higher is better) | AIA (higher is better) | AF (lower is better) | AOA (higher is better) | Historical projection state (MiB) |
|---|---|---|---|---|---|
| Vanilla SGD | 60.70 | 72.32 | 19.82 | 78.54 | 0 |
| GSF-SGD | 65.17 | 74.55 | 11.56 | 75.58 | 0 |
| GPM-SGD-adapted | 64.85 | 73.85 | 7.12 | 71.25 | 73.00 |
| NSCL-SGD-adapted | 64.54 | 73.61 | 7.07 | 70.91 | 384.75 |
[1] Saha, Gobinda, Isha Garg, and Kaushik Roy. "Gradient projection memory for continual learning." ICLR 2021. [2] Wang, Shipeng, et al. "Training networks in null space of feature covariance for continual learning." CVPR 2021.
Thank you for the additional experiments and responses.
Some of my concerns have addressed, I will maintain my original score.
We are grateful for your valuable comments and your recognition of our work. Your insightful suggestions have helped significantly improve the quality of our manuscript. Thank you very much for your time and effort in reviewing our work.
Response to Reviewer iyr3 (1/2)
We appreciate the reviewer's recognition of our method's simplicity and consistent improvements without additional inference overhead. To further address the reviewer's concerns, we have:
Meanwhile, we have revised the paper according to the comments, and all the revisions made in response to your comments are highlighted in green in the PDF.
We thank the reviewer for this valuable comment regarding the evidence connecting our method to genuinely shallow and wide networks. We clarify that although our method does not explicitly change network width or depth, it can still obtain the stability benefits observed in shallow and wide networks. Specifically, we focus on two properties: representation stability and the spectral diversity of updates. To further examine this connection, we compare Swin-T with Swin-S at the same initial width, and Swin-S with Swin-L at the same total number of blocks. As shown below, CKA measures representation stability and energy-based effective rank measures the spectral diversity of updates; both are averaged over the last block of each stage. Compared with Swin-S, the shallower Swin-T exhibits higher CKA (0.9063 vs. 0.8391), whereas the wider Swin-L exhibits higher energy-based effective rank (43.61 vs. 36.40). In particular, when comparing Swin-S + SW-Tune with Swin-S, we observe analogous trends, suggesting that our method shares similarities with shallow and wide networks in terms of representation stability and the spectral diversity of updates.
We thank the reviewer for the valuable comment on analyzing the proposed mechanism for mitigating cross-task interference. To address these points, we conducted the following additional analyses. In particular, we agree that the absolute cosine similarity between task-level parameter updates does not directly measure harmful gradient interference. Inspired by the protection of old-input feature subspaces in GPM [1] and Adam-NSCL [2], we replaced it with a measure of how much of the actual parameter-update energy falls into a subspace representing important old-task features. Specifically, for each matrix, the old-feature subspace is spanned by the leading eigenvectors of the old-task input second-moment matrix, with old tasks equally weighted and the basis truncated at 99% of the input-moment energy.
First, we keep PBF fixed and vary the exponent p to examine the relationship between spectral concentration and forgetting on ImageNet-A. A smaller p yields stronger flattening, while p=1 leaves the gradient unchanged. As p decreases, the energy-based effective rank increases from 57.65 to 197.29, while forgetting decreases from 19.82% to 8.96%. This shows that less concentrated updates are accompanied by less forgetting.
Second, we extend the representation and spectral analyses to isolate the contribution of PBF and GSF. As shown in the table below, PBF accounts for much of the improvement in CKA, whereas GSF accounts for much of the improvement in energy-based effective rank and old-subspace update energy fraction. These comparisons distinguish the roles of the two components: PBF mainly improves representation stability, while GSF benefits the spectral diversity of updates. In particular, GSF lowers the old-subspace update energy fraction. This suggests that GSF directs a relatively larger share of update energy outside the important old-task features subspace, thereby reducing cross-task interference. We have added the above discussion in the revised manuscript.