acceptodds
Under review as a conference paper at ICLR 2027

Scaling Gradient Projection for Continual Learning Over Long Task Sequences via Spectral Consolidation

Abstract

Gradient-projection methods address catastrophic forgetting in continual learning (CL) by retaining important activation subspaces and restricting subsequent parameter updates to minimally interfere with learned tasks. While these methods are competitive on standard CL benchmarks, their protection mechanisms become overly restrictive over long task sequences. We observe that in fixed-capacity models, protected subspaces formed by accumulating directions from successive tasks eventually saturate and stop admitting new directions. Even when constraints on parameter updates are relaxed, as in soft-projection variants, accumulated importance can ossify initially soft protection into effectively hard constraints. To address these limitations, we introduce Spectral Consolidation Gradient Projection Memory (SC-GPM), which combines activation usage across tasks in a bounded per-layer operator and recomputes the protected directions and their strengths after each task. We formulate the trade-off between protecting past tasks and reserving capacity for new learning as a convex optimization problem and derive an exact spectral solution. The solution reserves a specified amount of capacity for new learning by relaxing protection on less-used directions. Across DomainNet, OmniBenchmark, and ImageNet-1K streams of 100–160 tasks, SC-GPM improves final average accuracy by 2.18–3.64 percentage points over the strongest evaluated gradient-projection baseline on each dataset, with higher learning accuracy on all three. On domain-incremental Permuted MNIST with 200 tasks, SC-GPM outperforms the strongest evaluated gradient-projection baselines by 2.48 percentage points in final average accuracy and 1.24 points in accuracy averaged throughout learning (average anytime accuracy).

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, gradient projection, catastrophic forgetting, stability–plasticity trade-off, spectral consolidation, long task sequences

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cbQJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to balance the problem of reducing forgetting and learning new knowledge in the continual learning scenario of long task sequences. The paper points out that as tasks accumulate in existing gradient projection methods, problems such as protected direction sets stopping updates and directional gradient updates being prohibited may occur, making it difficult to learn new tasks.

In response to this problem, the paper proposes the SC-GPM method, which maintains a fixed-size matrix on each layer of the network (depending only on the input dimension of the layer) to record the degree of use of feature directions by historical task inputs. On each task boundary, it solves a convex allocation problem with the minimum gradient capacity constraint to update the gradient protection direction and protection strength. Experiments show that on multiple long task sequence datasets, the SC-GPM method surpasses existing gradient projection methods in the final average accuracy and is better than the naive experience replay method under an equal budget on most datasets.

Critical strengths

This paper identifies an overprotection problem in existing gradient projection-based continual learning algorithms over long task sequences and demonstrates the existence of this problem through theoretical and experimental evidence. The paper proposes SC-GPM algorithm, which balances the reduction of forgetting and the learning of new knowledge by updating the gradient protection direction and strength at the task boundary. Experiments show that the method has a certain effect.

Critical weaknesses
  1. The storage capacity required by SC-GPM is related to the network size, and the paper does not discuss which layers of the network should be selected for gradient protection. If the method can only be implemented on small-scale networks, it may limit the paper's practical applicability.
  2. In terms of experiments, it only compares against the most naive replay-based method ER, without comparing with more advanced algorithms (e.g., XDER, Boschini et al., 2022; Dynamic ER, Yan et al., 2021; DER++, Buzzega et al., 2020).
Questions for authors

Q1 [Clarification]. Is there a way to apply gradient projection protection to only part of the layers, or to use other techniques such as dimensionality reduction, to reduce the method's overhead on large networks?

Q2 [Suggestion]. Can comparisons with more advanced replay-based methods under the same budget be added?

6 comments

Weakness 1 and Question 1: memory overhead and larger networks

Answer. We thank the reviewer for their suggestion and question. SC-GPM's state memory footprint remains fixed as tasks accumulate and is quadratic in layer width. To address the reviewer's concerns, we examined three ways to reduce SC-GPM's overhead: compact storage, protecting a subset of layers, and dimensionality reduction. All three act on each layer separately and do not depend on the number of tasks, so they apply unchanged to larger networks.

1. Compact storage. Although SC-GPM's state matrix is symmetric, we currently do not exploit this, incurring the cost of all Dl2D_l^2 entries (where DlD_l is the layer's input dimension). Leveraging this can directly halve state memory. We also study reducing state precision. We note that storing the state at FP32 and exploiting matrix symmetry reduces the state footprint to 25% of the baseline FP64 SC-GPM at iso-accuracy. Details across precisions are provided in Table R3-1.

The small accuracy change we observe is consistent with the stability of SC-GPM's allocation (Appendix C.4). For a fixed capacity reserve κ\kappa, the change in the protection matrix cannot exceed the change in the stored state:

∥P~⋆−P⋆∥F≤∥Ω~−Ω∥F.\lVert \tilde P^\star - P^\star \rVert_F \leq \lVert \tilde\Omega - \Omega \rVert_F .

For state entries within FP16's normal representable range, rounding changes each entry by at most approximately 0.05%. The corresponding change in protection at that task boundary is therefore at most 0.05% of the state's Frobenius norm.

Table R3-1. Memory usage and accuracy on DomainNet-101 under different state precision formats over three seeds. Memory denotes the protection state, reported as a percentage of the full-matrix FP64 SC-GPM budget (BB) and in MB. Baseline results use FP64 and are taken from Tables 2 and 8 of the paper.

MethodStored stateMemory (% of BB)Memory (MB)ACCLABWT
GPMAppended basis88.915.9858.76±1.7263.49±0.57−4.78±1.17
SGPAppended basis and importance90.116.1966.49±0.5571.68±0.49−5.25±0.08
CODE-CL (uses 2.13× the budget)Conceptor and per-task state21338.2863.29±0.6063.13±0.520.17±0.11
SC-GPM (paper)Full matrix, FP6410017.9770.13±0.3075.81±0.41−5.74±0.13
SC-GPMLower triangle, FP6450.19.0170.13±0.3075.81±0.41−5.74±0.13
SC-GPMLower triangle, FP3225.14.5070.17±0.2475.92±0.37−5.81±0.15
SC-GPMLower triangle, FP1612.52.2569.90±0.3775.80±0.42−5.96±0.31

2. Protecting a subset of layers. SC-GPM allocates protection independently for each layer, so it can protect a selected subset. We tested the effects of partial protection by first freezing the early layers after task 1 and protecting the remaining convolutions, motivated by evidence that early features are more widely shared across tasks (Ramasesh et al., 2021).

Protecting 15, 10, or 5 of the 20 convolutional layers in ResNet-18 reduced final accuracy by 0.64, 1.88, or 3.99 points, respectively (Table R3-2). The first two settings remain above SGP with all layers protected (66.49%).

Most of the state is in the later, wider layers, so freezing the early layers saves 1.5%, 6.2%, or 24.9% of the total state, respectively. Layer selection therefore mainly reduces the number of projections and eigendecompositions needed at each task boundary.

Table R3-2. SC-GPM with a subset of layers protected on DomainNet-101 (three seeds). The remaining layers are frozen after task 1. SGP with all layers protected is included for comparison.

MethodFrozen after task 1Convolutions protectedMemory (% of BB)ACCLABWT
SGPNone2090.166.49±0.5571.68±0.49−5.25±0.08
SC-GPM (paper)None2010070.13±0.3075.81±0.41−5.74±0.13
SC-GPMInput conv. + stage 11598.569.49±0.5674.70±0.69−5.26±0.34
SC-GPMInput conv. + stages 1–21093.868.25±0.5372.63±0.77−4.43±0.24
SC-GPMInput conv. + stages 1–3575.166.13±0.2769.82±0.54−3.73±0.27

3. Dimensionality reduction. We consider two approximations. The first is a standard low-rank approximation that retains the top kk eigenpairs and discards the rest. The second retains only kk selected directions and their eigenvalues while replacing the remaining eigenvalues with their average. This shared average represents the accumulated usage in the remaining directions and forms an isotropic background, while the retained directions capture deviations from it, similar to Wang et al. (2016). We then recompute protection from this compressed representation.

In an initial 10-task experiment on DomainNet, the second approximation stores the state in one quarter of the memory of the uncompressed state and matches its final accuracy to within 0.15 points. Experiments on the full streams are in progress, and we will post the results in a follow-up comment.

Weakness 2 and Question 2: stronger replay baselines

Answer. We thank the reviewer for suggesting these comparisons and we have included DER++ (Buzzega et al., 2020) and X-DER (Boschini et al., 2022) to provide additional context on replay-based methods. Our intention was not to omit strong baselines, but rather to focus our study on the loss of learning capacity in GPM-based methods over long task sequences, with ER as the replay reference, following GPM, TRGP and SGP. Results for all three image streams at the same memory budget as SC-GPM are tabulated in Table R3-3. X-DER on ImageNet-1K-100 is still running; its time per task in Table R3-3 is the mean over the first 71 tasks, and we will post the completed row in a follow-up comment.

Note. We adapt X-DER to task-specific heads, omitting components that require a shared output layer. We do not include Dynamic ER (Yan et al., 2021), which adds a feature extractor for every task and therefore does not fit our fixed-backbone setting. Instead, we compare against DER++.

Table R3-3. Replay methods and SC-GPM at the same memory budget over three seeds. Time is wall-clock training time in minutes per task, measured as in Table 9 of the paper, with the ratio to SC-GPM in parentheses. Bold marks the best mean on each completed stream.

StreamMethodACCLABWTAAATime (min/task)
DomainNet-101ER30.60 ± 0.6478.30 ± 0.35−48.18 ± 0.7841.18 ± 0.509.47 ± 0.10 (1.6×)
DER++41.05 ± 1.2481.63 ± 0.32−40.98 ± 1.1151.97 ± 0.7714.98 ± 0.28 (2.6×)
X-DER36.70 ± 2.5481.09 ± 0.45−44.84 ± 2.4650.65 ± 0.6828.96 ± 0.36 (5.0×)
SC-GPM70.13 ± 0.3075.81 ± 0.41−5.74 ± 0.1362.96 ± 0.335.80 ± 0.04
OmniBenchmark-160ER14.10 ± 0.1251.24 ± 0.26−37.38 ± 0.3122.02 ± 0.545.13 ± 0.02 (1.5×)
DER++20.32 ± 0.5554.92 ± 0.56−34.82 ± 0.5028.61 ± 0.618.05 ± 0.10 (2.3×)
X-DER20.53 ± 0.6554.89 ± 0.19−34.57 ± 0.5329.24 ± 0.9415.39 ± 0.02 (4.4×)
SC-GPM41.53 ± 0.7044.46 ± 0.65−2.95 ± 0.0545.03 ± 0.663.51 ± 0.02
ImageNet-1K-100ER31.93 ± 0.6873.43 ± 0.28−41.92 ± 0.7538.82 ± 0.3423.54 ± 0.04 (1.8×)
DER++41.37 ± 0.7275.94 ± 0.18−34.92 ± 0.5545.91 ± 0.4737.69 ± 0.06 (2.8×)
X-DERrunningrunningrunningrunning73.42 ± 0.18 (5.5×), first 71 of 100 tasks
SC-GPM57.19 ± 0.6866.32 ± 0.24−9.23 ± 0.4853.67 ± 0.5613.28 ± 0.11

Revision. We will add the storage-format and layer-subset results (Tables R3-1 and R3-2) to Appendix D with a summary in Section 4, and DER++ and X-DER to Table 2. We hope these results resolve both concerns and would be grateful if the reviewer would take them into account in the final rating. We are happy to answer any further questions.

References

  • Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the eXtended DER-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5497–5512, 2022. doi: 10.1109/TPAMI.2022.3206549.
  • Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. In Advances in Neural Information Processing Systems, volume 33, pp. 15920–15930, 2020.
  • Vinay V. Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations, 2021.
  • Shusen Wang, Luo Luo, and Zhihua Zhang. SPSD matrix approximation vis column selection: Theories, algorithms, and extensions. Journal of Machine Learning Research, 17(49):1–49, 2016.
  • Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3014–3023, 2021.

The two sets of results we said we would post are now complete: the full-stream dimensionality reduction experiments (Table R3-4) and the X-DER row on ImageNet-1K-100 (Table R3-3).

Weakness 1 and Question 1: dimensionality reduction results

We have tabulated the results of the dimensionality reduction experiments on DomainNet-101 in Table R3-4. We repeat the definitions here for the reviewer's convenience. We study two methods. The first retains the top-kk directions and leaves the others unprotected. The second is an isotropic approximation that retains kk selected directions and replaces the remaining eigenvalues with their average. In our experiments, the isotropic approximation performed better than simply selecting the top-kk. Setting k=Dl/4k = D_l/4 and k=Dl/8k = D_l/8 reduces the state to 25.1% and 12.5% of the budget, respectively.

Table R3-4. Dimensionality reduction on DomainNet-101 over three seeds, next to reduced precision at the same memory budget. Memory is a percentage of the full-matrix FP64 SC-GPM budget (BB). ACC is mean ± standard deviation; LA and BWT are means.

MethodStored stateMemory (% of BB)ACCLABWT
SGPAppended basis and importance90.166.49 ± 0.5571.68−5.25
SC-GPM (paper)Full matrix, FP6410070.13 ± 0.3075.81−5.74
SC-GPMLower triangle, FP3225.170.17 ± 0.2475.92−5.81
SC-GPMIsotropic, k=Dl/4k = D_l/425.167.49 ± 0.4877.05−9.66
SC-GPMTop-kk, k=Dl/4k = D_l/425.138.35 ± 1.8179.98−42.05
SC-GPMLower triangle, FP1612.569.90 ± 0.3775.80−5.96
SC-GPMIsotropic, k=Dl/8k = D_l/812.564.96 ± 0.4176.99−12.15
SC-GPMTop-kk, k=Dl/8k = D_l/812.515.65 ± 0.2480.85−65.85

Weakness 2 and Question 2: completed X-DER results

We provide the completed results for X-DER on ImageNet-1K-100 in Table R3-3. X-DER reaches 40.45% final accuracy, 16.74 points below SC-GPM and 0.92 points below DER++.

Table R3-3 (ImageNet-1K-100, completed). Three seeds. Bold marks the best mean.

MethodACCLABWTAAATime (min/task)
ER31.93 ± 0.6873.43 ± 0.28−41.92 ± 0.7538.82 ± 0.3423.54 ± 0.04 (1.8×)
DER++41.37 ± 0.7275.94 ± 0.18−34.92 ± 0.5545.91 ± 0.4737.69 ± 0.06 (2.8×)
X-DER40.45 ± 0.1975.86 ± 0.32−35.77 ± 0.3047.13 ± 0.3473.26 ± 0.18 (5.5×)
SC-GPM57.19 ± 0.6866.32 ± 0.24−9.23 ± 0.4853.67 ± 0.5613.28 ± 0.11

Revision. We will add Table R3-4 to Appendix D together with the storage-format results, and the completed X-DER row to Table 2.

Reviewer_Rzg3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SC-GPM to optimize the gradient projection method for long task sequence learning by integrating historical statistical states and dynamically allocating protection intensity under a given lower limit of gradient update capacity. After the learning of each task, SC-GPM updates the cumulative usage matrix of historical tasks through the usage statistics matrix of the current task, recalculates the direction based on the cumulative usage matrix, and allocates the protection intensity of each direction through a convex optimization problem that limits the lower bound of the sum of update pass rates in all directions. The experiment was evaluated on three long image task sequences and Permuted MNIST, and achieved the highest final average accuracy in the first three image streams compared to the evaluated sequential learning method, improved the learning accuracy of new tasks, and the final accuracy on Permuted MNIST was higher than the strongest projection baseline. The experimental results show that in long task sequences, a better balance between new task learning and old task retention can improve the final average performance.

Critical strengths
  1. Integrates cumulative activation usage statistics, recalculation of protection directions, protection intensity distribution and updated capacity lower bounds into an accurately solvable framework.

  2. SC-GPM optimizes the trade-off between learning new tasks and retaining old tasks in the continuous learning model, and ultimately performs better in the final average accuracy of the long task sequence measured.

  3. The research questions are relatively consistent and consistent with the experiments conducted. The derivation process for the application of mathematical methods is relatively clear, and the key solution calculation processes are clearly explained.

Critical weaknesses
  1. Although this paper has set a minimum reserve for the total update volume, this reserve does not guarantee that the model can learn new tasks. In future theoretical developments, sufficient experiments should be conducted to reveal whether the model can indeed learn new tasks in the more realistic training scenario and how the two conditions relate.

  2. However, on the DomainNet and ImageNet sequences, compared to most gradient-projection methods, SC-GPM suffers from a higher average decline in the performances of the old tasks, relative to the values attained at the time of learning the tasks first. Although a larger FA subsequent to learning is obtained, SC-GPM does not robustly outperform in old task retention, i.e., a trade-off between new and old tasks remains.

  3. SC-GPM has yet to resolve the issue of severe forgetting in scenarios involving a shared classification head, and its absolute performance remains relatively limited.

Questions for authors
  1. In the more realistic training scenario considered in this paper, will new training tasks be capable of actually using the reserve parameter update capacity?

  2. To what extent is catastrophic forgetting attributable to the shared classification head? How do the learning accuracy,backward transfer, and final average accuracy change if projection passes through similarly applied rules on the classification head?

6 comments

We thank the reviewer for the detailed review and for highlighting our exact solution, which also improves final accuracy over long task sequences. Our response covers Weakness 3 with Question 2, with new experiments, and Weakness 2. We are finishing the experiments for Question 1 and will post those results as soon as they are complete.

Weakness 3 and Question 2: forgetting and the shared classification head

Answer. We observe that applying SC-GPM to the classification head consistently improves final accuracy and reduces forgetting. On PMNIST with 200 tasks, final accuracy increases from 17.82% to 48.87% while learning accuracy decreases by 2.41 points. Forgetting, measured by the magnitude of BWT, decreases by 42–64% on PMNIST and 17.5% on DomainNet-6 (Tables R2-1 and R2-2).

Setup. In the paper, we applied the general standard for gradient projection, i.e., SC-GPM protects only the backbone. For PMNIST, one ten-way head is shared by all tasks and trains freely, so forgetting in the head is outside what the backbone projection controls. To answer the question, we applied the same SC-GPM rule to the shared head and left everything else unchanged. We evaluate both versions on PMNIST and on DomainNet-6, a domain-incremental benchmark with six tasks and a shared head. On DomainNet-6, α\alpha was selected on that dataset's validation data. We compare the results over five seeds below.

Protecting the head also narrows the gap between SC-GPM and ER on both PMNIST and DomainNet-6. On PMNIST with 200 tasks, ER's lead in final accuracy falls from 49.16 to 18.11 points, with SC-GPM achieving this without storing any examples.

Table R2-1. SC-GPM with a trainable or projected shared head, over five seeds (1–5). These results exclude the seed used to select configurations. Bold indicates improvements from projecting the head.

BenchmarkMethodLABWTACC
PMNIST, 10 tasksFine-tuning97.43 ± 0.04−44.80 ± 2.9657.12 ± 2.69
SC-GPM, trainable head (paper)97.45 ± 0.05−12.36 ± 1.0886.33 ± 0.97
SC-GPM, projected head97.39 ± 0.07−4.46 ± 0.3293.38 ± 0.31
ER96.95 ± 0.03−1.36 ± 0.0395.73 ± 0.04
PMNIST, 200 tasksFine-tuning96.79 ± 0.01−83.65 ± 0.4913.56 ± 0.48
SC-GPM, trainable head (paper)97.07 ± 0.00−79.65 ± 0.9317.82 ± 0.92
SC-GPM, projected head94.65 ± 0.03−46.01 ± 0.4448.87 ± 0.42
ER96.59 ± 0.01−29.76 ± 0.5066.98 ± 0.49
DomainNet-6 (domain-incremental)Fine-tuning66.19 ± 4.61−28.31 ± 6.5242.60 ± 4.41
SC-GPM, trainable head (paper)67.97 ± 1.47−21.64 ± 1.5449.93 ± 2.20
SC-GPM, projected head67.51 ± 1.40−17.84 ± 1.2752.65 ± 1.70
ER68.54 ± 0.93−14.85 ± 1.7756.17 ± 2.00

Table R2-2. Impact of projecting the head, compared with a trainable head. Differences are paired by seed over the five seeds (1–5).

BenchmarkΔLAΔBWTΔACCHead's share of forgettingER's lead in ACC (trainable → projected)
PMNIST, 10 tasks−0.06+7.91+7.0563.7 ± 4.6%9.40 → 2.35
PMNIST, 200 tasks−2.41+33.63+31.0542.2 ± 1.1%49.16 → 18.11
DomainNet-6−0.45+3.80+2.7217.5 ± 2.9%6.24 → 3.52

Weakness 2: retention on DomainNet and ImageNet

Answer. The reviewer's reading is correct. SC-GPM trades a small amount of retention for improved learning. Compared with SGP, the strongest of these baselines, SC-GPM gives up 0.49 points of backward transfer on DomainNet for 4.13 points of learning accuracy and 3.64 points of final accuracy. On ImageNet, the corresponding changes are −1.46, +4.72 and +3.29 points (Table R2-3). On OmniBenchmark, SC-GPM improves both learning accuracy and backward transfer over SGP. Compared with CODE-CL, SC-GPM achieves 2.18–6.84 points higher final accuracy across the three streams, while CODE-CL retains more of what it learns and uses 2.13× the memory budget.

Crucially, we also find that SC-GPM forgets less over the longer run. Using each task's accuracy after 19 subsequent tasks as the reference, SC-GPM retains 98.98%, 97.77% and 98.32% of that accuracy at the end of DomainNet, OmniBenchmark and ImageNet, respectively. SGP retains 96.36%, 95.99% and 95.36%. Thus, although SC-GPM has lower BWT on DomainNet and ImageNet, it loses a smaller fraction of its accuracy than SGP after the first 19 subsequent tasks.

SC-GPM does offer tuning of this trade-off through α\alpha: a larger α\alpha places more emphasis on protecting earlier tasks (Appendices C.7 and F.4). For all our primary experiments, we used the same value of α\alpha on all image streams, selected on DomainNet validation data.

Table R2-3. Difference between SC-GPM and each projection baseline, using results from Table 2 of the paper. Each cell shows the change in learning accuracy, BWT and final accuracy (ΔLA / ΔBWT / ΔACC), in points. Positive values favor SC-GPM.

BaselineDomainNet-101OmniBenchmark-160ImageNet-1K-100
GPM+12.32 / −0.96 / +11.37+7.59 / +0.54 / +8.13+12.36 / −2.07 / +10.32
TRGP (uses up to 2× the budget)+12.59 / −0.85 / +11.76+7.66 / +0.49 / +8.14+12.89 / −2.97 / +9.96
SGP+4.13 / −0.49 / +3.64+2.19 / +0.09 / +2.28+4.72 / −1.46 / +3.29
Adam-NSCL+4.43 / +9.00 / +13.35+1.34 / +12.40 / +13.66+3.33 / +8.21 / +11.46
CODE-CL (uses 2.13× the budget)+12.68 / −5.91 / +6.84+8.28 / −3.22 / +5.07+10.84 / −8.76 / +2.18

Revision. We will add the projected-head results and the DomainNet-6 results to Section 5.4, with the tables added to Appendix F.3. We will add details on retention to Appendix F.2 and a brief discussion of it in Section 5.2.

The experiments for Question 1 are now complete.

Weakness 1 and Question 1: do new tasks use the reserved capacity?

We examine whether new tasks use the reserved capacity on a long sequence of natural-image tasks, using a 200-task stream, ImageNet-2K-200, built from the ImageNet-1K classes and 1,000 disjoint classes from ImageNet-21K-P (Ridnik et al., 2021). We compare SC-GPM without a capacity floor against one with the minimal reserve floor of one unit per layer, and with two further settings where the floor is approximately 8% and 13% of each layer's full gradient capacity. We keep the statistics construction, α\alpha and training settings fixed (Table R2-4).

We measure new-task gradients during the first epoch of tasks 101–200. SC-GPM maintains capacity by relaxing protection in some directions to allow updates. With the minimum reserved capacity, these directions account for 75.7% of the predicted loss decrease, close to the 75.5% expected for a gradient with no preferred direction. These results show that new tasks use the capacity kept open by the reserve.

Increasing the reserved capacity allows more of the new-task gradient to flow. At r=0.13r = 0.13, gradient energy after projection is 18% higher than when we set the reserved capacity to its minimum value. Learning accuracy over tasks 101–200 is higher in every seed, reaching 71.60% compared with 71.07% without a reserve. We do note that this trades against forgetting.

With the paper's setting, SC-GPM's learning accuracy over tasks 101–200 is 8.24 points above CODE-CL's, and its final accuracy at 200 tasks is 4.73 points higher.

Table R2-4. Use of the reserved capacity on ImageNet-2K-200, three seeds (mean ± SD). LA is over tasks 101–200; BWT and ACC are at 200 tasks. "Layers" is the number of the 20 protected layers in which the reserve engages; "Released" is the share of first-order loss decrease through released directions, and "Reference" the share expected for a gradient with no preferred direction.

MethodReserveLayersReleased (%)Reference (%)LABWTACC
CODE-CL (uses 2.13× the budget)————62.87 ± 0.10+0.34 ± 0.0863.37 ± 0.06
SC-GPMnone0——71.07 ± 0.17−4.08 ± 0.3368.24 ± 0.41
SC-GPM (paper)κl=1\kappa_l = 1375.775.571.11 ± 0.12−4.19 ± 0.3468.10 ± 0.36
SC-GPMr=0.08r = 0.085.377.476.571.26 ± 0.35−4.44 ± 0.3267.98 ± 0.30
SC-GPMr=0.13r = 0.131346.944.571.60 ± 0.02−4.88 ± 0.2967.78 ± 0.41

Revision. We will add the ImageNet-2K-200 results to Section 5, and Table R2-4 with this measurement to the appendix.

Reference

  • Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-21K pretraining for the masses. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Reviewer_tFv2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies accumulated protection in append-only memories and fixed spectral rules, proposing SC-GPM to recompute protection directions and strengths from accumulated normalized activation statistics under a gradient capacity constraint. On three image streams of 100–160 tasks and PMNIST with up to 200 tasks, it improves final accuracy over the evaluated projection baselines.

Critical strengths
  1. The convex allocation has a unique spectral solution (Theorem 3, p. 6), and the paper clearly distinguishes gradient capacity guarantees from guarantees of performance on future tasks. This makes the method interpretable and straightforward to implement.
  2. Table 2 reports final accuracy gains of 3.64, 2.28, and 2.18 percentage points over the strongest evaluated projection baseline on each stream, while Table 3 shows gains of 5.33–7.78 points over allocation within the inherited basis. These results support the practical value of the joint design on long sequences.
Critical weaknesses
  1. The capacity floor's independent contribution on image streams remains unclear: Table 3 changes the protection rule, the floor binds only in three shortcut layers (p. 9), and Figure 3 varies the reserve only on PMNIST. This limits attribution of the image-stream gains to the explicit capacity constraint.
  2. Appendix E.3 does not specify whether PMNIST configurations were selected using validation or test accuracy, and Figure 13 notes that results at the standard setting include the selection seed. This leaves selection effects unresolved without establishing data leakage.
Questions for authors
  1. Do existing image-stream results isolate removal of the capacity floor while holding the statistics construction, α, and training settings fixed?
  2. Which split was used to select PMNIST configurations, was selection kept separate from final testing, and which reported results include the selection seed?
4 comments

We are finishing the experiments for Question 1 and will post those results as soon as they are complete. This comment answers Weakness 2 and Question 2.

Weakness 2 and Question 2: selection split and seed on PMNIST

Answer. In the paper, we selected the PMNIST configurations using the test split of seed 0 and reported results over seeds 0–2, so the reported PMNIST averages included the selection seed. Selecting the configurations using the validation split of seed 0 gives the same choices. We will clarify this procedure in Appendix E.3. On the image streams, selection was kept separate from evaluation: SC-GPM's α\alpha was selected on DomainNet validation data with a seed disjoint from the evaluation seeds (Section 5.1, Appendix E.3).

To address the reviewer's concern, we kept the configurations unchanged and evaluated them on five new seeds (1–5). In PMNIST, each seed generates its own sequence of permutations, so these results evaluate the selected configurations on task sequences different from the one used for selection.

SC-GPM remains ahead of the strongest projection baseline at every horizon (Table R1-2). At 200 tasks, SC-GPM is 2.04 points ahead in final accuracy and 1.32 points ahead in AAA.

Table R1-2. SC-GPM's margin over the strongest projection baseline on PMNIST, in points.

Metric and seedsT = 50T = 100T = 150T = 200
Final accuracy, seeds 0–2 (paper)+0.22+1.79+3.29+2.48
Final accuracy, seeds 1–5+0.17+1.21+3.09+2.04
AAA, seeds 0–2 (paper)+0.64+0.36+0.76+1.24
AAA, seeds 1–5+0.66+0.78+1.06+1.32

Revision. All seed-averaged PMNIST results in the abstract, Section 1, Section 5.4 and the figures will use seeds 1–5, so the abstract's margins become 2.04 and 1.32 points. Figures 2(c) and 15 show the search grid on the selection seed, and their captions will state this.

The experiments for Question 1 are now complete.

Weakness 1 and Question 1: isolating the capacity reserve on image streams

We target a long task sequence of natural images by extending our experiments to a 200-task stream, ImageNet-2K-200, built from the ImageNet-1K classes and 1,000 disjoint classes from ImageNet-21K-P (Ridnik et al., 2021). We compare SC-GPM without a capacity floor with our current setup, which has a floor of one unit per layer, and with reserve floors of approximately 8% and 13% of each layer's full gradient capacity. We keep other aspects of the experiment, e.g., the statistics construction, α\alpha and training settings, fixed (Table R1-1).

We observe shortcut layers reaching zero gradient capacity in two of three seeds when we do not reserve any capacity. Ensuring that a minimum capacity is reserved prevents this. Our gradient measurements also show that new tasks use the reserved capacity: the released directions account for 75.7% of the predicted loss decrease, close to the 75.5% expected for a gradient with no preferred direction.

Increasing the reserve also allows more of the new-task gradient through. At r=0.13r = 0.13, gradient energy after projection is 18% higher than with the minimum reserve, and learning accuracy over tasks 101–200 is higher in every seed, reaching 71.60% compared with 71.07% without any reserved capacity. This does come at the cost of forgetting.

Removing the minimum capacity reserve changes final accuracy by less than 0.2 points, and final accuracy stays within 0.46 points across the four reserve settings we tested. With the paper's minimum reserve, SC-GPM's lead over CODE-CL, the strongest baseline on ImageNet-1K-100, grows from 3.17 points at 100 tasks to 4.73 points at 200 tasks.

Table R1-1. Isolating the capacity reserve on ImageNet-2K-200, three seeds (mean ± SD). LA is over tasks 101–200; BWT is at 200 tasks. "Layers" is the number of the 20 protected layers in which the reserve engages over tasks 101–200.

MethodReserveACC, 100 tasksACC, 150 tasksACC, 200 tasksLABWTLayers
CODE-CL (uses 2.13× the budget)—63.28 ± 0.2163.18 ± 0.0663.37 ± 0.0662.87 ± 0.10+0.34 ± 0.08—
SC-GPMnone66.76 ± 0.9467.63 ± 0.8068.24 ± 0.4171.07 ± 0.17−4.08 ± 0.330
SC-GPM (paper)κl=1\kappa_l = 166.45 ± 0.8567.30 ± 0.8268.10 ± 0.3671.11 ± 0.12−4.19 ± 0.343
SC-GPMr=0.08r = 0.0866.50 ± 0.9867.29 ± 0.7167.98 ± 0.3071.26 ± 0.35−4.44 ± 0.325.3
SC-GPMr=0.13r = 0.1366.23 ± 0.9867.00 ± 0.7467.78 ± 0.4171.60 ± 0.02−4.88 ± 0.2913

Revision. We will add the ImageNet-2K-200 results to Section 5 and Table R1-1 to the appendix.

Reference

  • Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-21K pretraining for the masses. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.

Revised manuscript uploaded

We thank the reviewer again for the careful review. We have uploaded the revised manuscript, which includes all the changes described in our responses, marked in blue. The PMNIST selection procedure and held-out-seed results are in Appendix E.3 and Section 5.4, and the capacity-reserve study on ImageNet-2K-200 is in Section 5.5 and Appendix F.5. Our general comment above summarizes the main improvements. We are happy to answer any further questions.

Your questions on how the PMNIST configurations were selected and what the capacity reserve contributes were helpful in strengthening the experiments. The revised paper now includes results on held-out seeds and a study of the reserve on the 200-task ImageNet stream.

As the discussion period comes to a close, we would welcome your feedback on these results, and we are happy to clarify any remaining point.

Comments on the paper

1 comment

Summary of the revised manuscript

We thank all reviewers for their constructive feedback, which has strengthened the paper. We have uploaded a revised manuscript that includes the changes described in our responses, with new and updated text marked in blue. The main improvements are:

  • A longer natural-image stream. On a new 200-task stream, ImageNet-2K-200, SC-GPM's lead over CODE-CL grows from 3.17 points at 100 tasks to 4.73 points at 200 tasks (Section 5.2). The same stream is used to study the capacity reserve and how new tasks use it (Section 5.5, Appendix F.5).
  • Stronger replay baselines. DER++ and X-DER are added at the same memory budget on all three image streams. SC-GPM remains 15.82–29.08 points ahead in final accuracy (Table 2, Section 5.2).
  • Protection of the shared classification head. Applying SC-GPM to the shared head raises final accuracy on PMNIST with 200 tasks from 17.82% to 48.87%, and also improves accuracy on a new domain-incremental benchmark, DomainNet-6 (Section 5.4, Table 4).
  • Lower memory overhead. Storing the state compactly reduces SC-GPM's memory to 25.1% of the original at the same accuracy. We also evaluate low-rank storage and protecting a subset of layers (Section 4, Appendix D.2).
  • Evaluation on held-out seeds for PMNIST. All PMNIST results are now reported on five seeds that exclude the selection seed, and the selection procedure is stated explicitly (Appendix E.3). SC-GPM's margins at 200 tasks are 2.04 points in final accuracy and 1.32 points in average anytime accuracy.
  • Retention analysis. Section 5.2 and Appendix F.2 now discuss the trade-off between learning and retention on the image streams.

We are grateful for the reviewers' time and are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.