- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to balance the problem of reducing forgetting and learning new knowledge in the continual learning scenario of long task sequences. The paper points out that as tasks accumulate in existing gradient projection methods, problems such as protected direction sets stopping updates and directional gradient updates being prohibited may occur, making it difficult to learn new tasks.
In response to this problem, the paper proposes the SC-GPM method, which maintains a fixed-size matrix on each layer of the network (depending only on the input dimension of the layer) to record the degree of use of feature directions by historical task inputs. On each task boundary, it solves a convex allocation problem with the minimum gradient capacity constraint to update the gradient protection direction and protection strength. Experiments show that on multiple long task sequence datasets, the SC-GPM method surpasses existing gradient projection methods in the final average accuracy and is better than the naive experience replay method under an equal budget on most datasets.
Critical strengths
This paper identifies an overprotection problem in existing gradient projection-based continual learning algorithms over long task sequences and demonstrates the existence of this problem through theoretical and experimental evidence. The paper proposes SC-GPM algorithm, which balances the reduction of forgetting and the learning of new knowledge by updating the gradient protection direction and strength at the task boundary. Experiments show that the method has a certain effect.
Critical weaknesses
- The storage capacity required by SC-GPM is related to the network size, and the paper does not discuss which layers of the network should be selected for gradient protection. If the method can only be implemented on small-scale networks, it may limit the paper's practical applicability.
- In terms of experiments, it only compares against the most naive replay-based method ER, without comparing with more advanced algorithms (e.g., XDER, Boschini et al., 2022; Dynamic ER, Yan et al., 2021; DER++, Buzzega et al., 2020).
Questions for authors
Q1 [Clarification]. Is there a way to apply gradient projection protection to only part of the layers, or to use other techniques such as dimensionality reduction, to reduce the method's overhead on large networks?
Q2 [Suggestion]. Can comparisons with more advanced replay-based methods under the same budget be added?
6 comments
Weakness 2 and Question 2: stronger replay baselines
Answer. We thank the reviewer for suggesting these comparisons and we have included DER++ (Buzzega et al., 2020) and X-DER (Boschini et al., 2022) to provide additional context on replay-based methods. Our intention was not to omit strong baselines, but rather to focus our study on the loss of learning capacity in GPM-based methods over long task sequences, with ER as the replay reference, following GPM, TRGP and SGP. Results for all three image streams at the same memory budget as SC-GPM are tabulated in Table R3-3. X-DER on ImageNet-1K-100 is still running; its time per task in Table R3-3 is the mean over the first 71 tasks, and we will post the completed row in a follow-up comment.
Note. We adapt X-DER to task-specific heads, omitting components that require a shared output layer. We do not include Dynamic ER (Yan et al., 2021), which adds a feature extractor for every task and therefore does not fit our fixed-backbone setting. Instead, we compare against DER++.
Table R3-3. Replay methods and SC-GPM at the same memory budget over three seeds. Time is wall-clock training time in minutes per task, measured as in Table 9 of the paper, with the ratio to SC-GPM in parentheses. Bold marks the best mean on each completed stream.
| Stream | Method | ACC | LA | BWT | AAA | Time (min/task) |
|---|---|---|---|---|---|---|
| DomainNet-101 | ER | 30.60 ± 0.64 | 78.30 ± 0.35 | −48.18 ± 0.78 | 41.18 ± 0.50 | 9.47 ± 0.10 (1.6×) |
| DER++ | 41.05 ± 1.24 | 81.63 ± 0.32 | −40.98 ± 1.11 | 51.97 ± 0.77 | 14.98 ± 0.28 (2.6×) | |
| X-DER | 36.70 ± 2.54 | 81.09 ± 0.45 | −44.84 ± 2.46 | 50.65 ± 0.68 | 28.96 ± 0.36 (5.0×) | |
| SC-GPM | 70.13 ± 0.30 | 75.81 ± 0.41 | −5.74 ± 0.13 | 62.96 ± 0.33 | 5.80 ± 0.04 | |
| OmniBenchmark-160 | ER | 14.10 ± 0.12 | 51.24 ± 0.26 | −37.38 ± 0.31 | 22.02 ± 0.54 | 5.13 ± 0.02 (1.5×) |
| DER++ | 20.32 ± 0.55 | 54.92 ± 0.56 | −34.82 ± 0.50 | 28.61 ± 0.61 | 8.05 ± 0.10 (2.3×) | |
| X-DER | 20.53 ± 0.65 | 54.89 ± 0.19 | −34.57 ± 0.53 | 29.24 ± 0.94 | 15.39 ± 0.02 (4.4×) | |
| SC-GPM | 41.53 ± 0.70 | 44.46 ± 0.65 | −2.95 ± 0.05 | 45.03 ± 0.66 | 3.51 ± 0.02 | |
| ImageNet-1K-100 | ER | 31.93 ± 0.68 | 73.43 ± 0.28 | −41.92 ± 0.75 | 38.82 ± 0.34 | 23.54 ± 0.04 (1.8×) |
| DER++ | 41.37 ± 0.72 | 75.94 ± 0.18 | −34.92 ± 0.55 | 45.91 ± 0.47 | 37.69 ± 0.06 (2.8×) | |
| X-DER | running | running | running | running | 73.42 ± 0.18 (5.5×), first 71 of 100 tasks | |
| SC-GPM | 57.19 ± 0.68 | 66.32 ± 0.24 | −9.23 ± 0.48 | 53.67 ± 0.56 | 13.28 ± 0.11 |
Revision. We will add the storage-format and layer-subset results (Tables R3-1 and R3-2) to Appendix D with a summary in Section 4, and DER++ and X-DER to Table 2. We hope these results resolve both concerns and would be grateful if the reviewer would take them into account in the final rating. We are happy to answer any further questions.
References
- Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the eXtended DER-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5497–5512, 2022. doi: 10.1109/TPAMI.2022.3206549.
- Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. In Advances in Neural Information Processing Systems, volume 33, pp. 15920–15930, 2020.
- Vinay V. Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations, 2021.
- Shusen Wang, Luo Luo, and Zhihua Zhang. SPSD matrix approximation vis column selection: Theories, algorithms, and extensions. Journal of Machine Learning Research, 17(49):1–49, 2016.
- Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3014–3023, 2021.
The two sets of results we said we would post are now complete: the full-stream dimensionality reduction experiments (Table R3-4) and the X-DER row on ImageNet-1K-100 (Table R3-3).
Weakness 1 and Question 1: dimensionality reduction results
We have tabulated the results of the dimensionality reduction experiments on DomainNet-101 in Table R3-4. We repeat the definitions here for the reviewer's convenience. We study two methods. The first retains the top- directions and leaves the others unprotected. The second is an isotropic approximation that retains selected directions and replaces the remaining eigenvalues with their average. In our experiments, the isotropic approximation performed better than simply selecting the top-. Setting and reduces the state to 25.1% and 12.5% of the budget, respectively.
Table R3-4. Dimensionality reduction on DomainNet-101 over three seeds, next to reduced precision at the same memory budget. Memory is a percentage of the full-matrix FP64 SC-GPM budget (). ACC is mean ± standard deviation; LA and BWT are means.
| Method | Stored state | Memory (% of ) | ACC | LA | BWT |
|---|---|---|---|---|---|
| SGP | Appended basis and importance | 90.1 | 66.49 ± 0.55 | 71.68 | −5.25 |
| SC-GPM (paper) | Full matrix, FP64 | 100 | 70.13 ± 0.30 | 75.81 | −5.74 |
| SC-GPM | Lower triangle, FP32 | 25.1 | 70.17 ± 0.24 | 75.92 | −5.81 |
| SC-GPM | Isotropic, | 25.1 | 67.49 ± 0.48 | 77.05 | −9.66 |
| SC-GPM | Top-, | 25.1 | 38.35 ± 1.81 | 79.98 | −42.05 |
| SC-GPM | Lower triangle, FP16 | 12.5 | 69.90 ± 0.37 | 75.80 | −5.96 |
| SC-GPM | Isotropic, | 12.5 | 64.96 ± 0.41 | 76.99 | −12.15 |
| SC-GPM | Top-, | 12.5 | 15.65 ± 0.24 | 80.85 | −65.85 |
Weakness 2 and Question 2: completed X-DER results
We provide the completed results for X-DER on ImageNet-1K-100 in Table R3-3. X-DER reaches 40.45% final accuracy, 16.74 points below SC-GPM and 0.92 points below DER++.
Table R3-3 (ImageNet-1K-100, completed). Three seeds. Bold marks the best mean.
| Method | ACC | LA | BWT | AAA | Time (min/task) |
|---|---|---|---|---|---|
| ER | 31.93 ± 0.68 | 73.43 ± 0.28 | −41.92 ± 0.75 | 38.82 ± 0.34 | 23.54 ± 0.04 (1.8×) |
| DER++ | 41.37 ± 0.72 | 75.94 ± 0.18 | −34.92 ± 0.55 | 45.91 ± 0.47 | 37.69 ± 0.06 (2.8×) |
| X-DER | 40.45 ± 0.19 | 75.86 ± 0.32 | −35.77 ± 0.30 | 47.13 ± 0.34 | 73.26 ± 0.18 (5.5×) |
| SC-GPM | 57.19 ± 0.68 | 66.32 ± 0.24 | −9.23 ± 0.48 | 53.67 ± 0.56 | 13.28 ± 0.11 |
Revision. We will add Table R3-4 to Appendix D together with the storage-format results, and the completed X-DER row to Table 2.
Weakness 1 and Question 1: memory overhead and larger networks
Answer. We thank the reviewer for their suggestion and question. SC-GPM's state memory footprint remains fixed as tasks accumulate and is quadratic in layer width. To address the reviewer's concerns, we examined three ways to reduce SC-GPM's overhead: compact storage, protecting a subset of layers, and dimensionality reduction. All three act on each layer separately and do not depend on the number of tasks, so they apply unchanged to larger networks.
1. Compact storage. Although SC-GPM's state matrix is symmetric, we currently do not exploit this, incurring the cost of all Dl2 entries (where Dl is the layer's input dimension). Leveraging this can directly halve state memory. We also study reducing state precision. We note that storing the state at FP32 and exploiting matrix symmetry reduces the state footprint to 25% of the baseline FP64 SC-GPM at iso-accuracy. Details across precisions are provided in Table R3-1.
The small accuracy change we observe is consistent with the stability of SC-GPM's allocation (Appendix C.4). For a fixed capacity reserve κ, the change in the protection matrix cannot exceed the change in the stored state:
∥P~⋆−P⋆∥F≤∥Ω~−Ω∥F.
For state entries within FP16's normal representable range, rounding changes each entry by at most approximately 0.05%. The corresponding change in protection at that task boundary is therefore at most 0.05% of the state's Frobenius norm.
Table R3-1. Memory usage and accuracy on DomainNet-101 under different state precision formats over three seeds. Memory denotes the protection state, reported as a percentage of the full-matrix FP64 SC-GPM budget (B) and in MB. Baseline results use FP64 and are taken from Tables 2 and 8 of the paper.
2. Protecting a subset of layers. SC-GPM allocates protection independently for each layer, so it can protect a selected subset. We tested the effects of partial protection by first freezing the early layers after task 1 and protecting the remaining convolutions, motivated by evidence that early features are more widely shared across tasks (Ramasesh et al., 2021).
Protecting 15, 10, or 5 of the 20 convolutional layers in ResNet-18 reduced final accuracy by 0.64, 1.88, or 3.99 points, respectively (Table R3-2). The first two settings remain above SGP with all layers protected (66.49%).
Most of the state is in the later, wider layers, so freezing the early layers saves 1.5%, 6.2%, or 24.9% of the total state, respectively. Layer selection therefore mainly reduces the number of projections and eigendecompositions needed at each task boundary.
Table R3-2. SC-GPM with a subset of layers protected on DomainNet-101 (three seeds). The remaining layers are frozen after task 1. SGP with all layers protected is included for comparison.
3. Dimensionality reduction. We consider two approximations. The first is a standard low-rank approximation that retains the top k eigenpairs and discards the rest. The second retains only k selected directions and their eigenvalues while replacing the remaining eigenvalues with their average. This shared average represents the accumulated usage in the remaining directions and forms an isotropic background, while the retained directions capture deviations from it, similar to Wang et al. (2016). We then recompute protection from this compressed representation.
In an initial 10-task experiment on DomainNet, the second approximation stores the state in one quarter of the memory of the uncompressed state and matches its final accuracy to within 0.15 points. Experiments on the full streams are in progress, and we will post the results in a follow-up comment.