- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the Storage, Compression, and Targets (SCT) framework, which unifies neural learning as the interplay between storing associations, compressing representations, and aligning neural activity with local targets. The authors prove that fully preventing task degradation requires corrections coupled to the current sample, and derive a replay-based correction mechanism that improves retention of replay-based methods for class-incremental learning.
Critical strengths
- The paper reframes learning as the layer-wise interplay of Storage, Compression, and Targets (SCT).
- The paper proposes a correction term of the forgetting drift due to backpropagation that shifts how replays are used. The authors empirically show on a class-incremental benchmark that this geometry-aware correction significantly improves retention and sample efficiency of replay-based methods.
Critical weaknesses
- The empirical comparison relies primarily on older continual-learning baselines, such as DER++. While these remain useful references, the absence of recent competitive replay-based methods makes it difficult to assess the proposed method’s advantage over the current state of the art. Evaluating the correction on stronger recent replay-based methods would help establish its general applicability.
- The evaluation does not include Split CIFAR-100 which is often used in CL study.
Questions for authors
- If possible, please add comparison with more recent work in CL.
- Please include the evaluation on CIFAR-100
2 comments
Q2: “The evaluation does not include Split CIFAR-100”
This is also a fair point. We now also consider Split-CIFAR-100 (10 tasks of 10 classes), which results we report below. The JOINT training reaches 67.64 ± 0.15 and naive fine-tuning 9.91 ± 0.04. OGD stays at the naive level (7.8–9.7), as on Split-Tiny-ImageNet.
Final class-incremental accuracy (%), mean ± sd for replay buffer size , bold for best and italic for second best:
| B = 10 | 50 | 200 | 500 | 1000 | |
|---|---|---|---|---|---|
| Split-CIFAR-10 | |||||
| DER++ | 35.48 ± 4.34 | 64.46 ± 2.54 | 76.12 ± 0.96 | 80.80 ± 0.53 | 82.59 ± 0.27 |
| HOPc + DER++ | 75 ± 2.02 | 77.65 ± 0.97 | 80.22 ± 0.49 | 82.01 ± 0.26 | 82.74 ± 0.20 |
| DER++ + STAR | 39.03 ± 2.20 | 66.16 ± 3.00 | 77.86 ± 1.17 | 80.93 ± 0.30 | 82.00 ± 0.84 |
| HOPc + DER++ + STAR | 74.73 ± 1.76 | 79.82 ± 0.39 | 82.12 ± 0.25 | 82.75 ± 0.61 | 83.28 ± 0.41 |
| DER++ + LPR | 44.15 ± 1.53 | 63.10 ± 1.79 | 75.84 ± 0.35 | 81.59 ± 0.29 | 83.37 ± 0.17 |
| HOPc + DER++ + LPR | 71.45 ± 2.35 | 73.79 ± 0.77 | 79.12 ± 0.46 | 82.07 ± 0.38 | 83.05 ± 0.22 |
| HOPc w/ GR (anchored) | 78.23 ± 0.23 | 81.93 ± 0.11 | 84.67 ± 0.15 | 84.96 ± 0.11 | 84.75 ± 0.24 |
| Split-Tiny-ImageNet | |||||
| DER++ | 15.27 ± 0.79 | 26.99 ± 0.39 | 40.28 ± 0.73 | 47.17 ± 0.19 | 51.15 ± 0.79 |
| HOPc + DER++ | 25.38 ± 2.73 | 32.72 ± 1.55 | 41.99 ± 0.41 | 48.54 ± 0.45 | 52.17 ± 0.27 |
| DER++ + STAR | 14.91 ± 0.80 | 27.32 ± 0.99 | 40.25 ± 0.73 | 47.05 ± 0.55 | 51.33 ± 0.68 |
| HOPc + DER++ + STAR | 26.25 ± 3.15 | 32.71 ± 1.08 | 41.98 ± 0.39 | 48.35 ± 0.62 | 52.19 ± 0.60 |
| DER++ + LPR | 16.01 ± 1.16 | 24.48 ± 0.95 | 37.00 ± 0.52 | 50.23 ± 0.55 | 57.12 ± 0.39 |
| HOPc + DER++ + LPR | 27.46 ± 4.72 | 31.43 ± 0.64 | 41.00 ± 0.43 | 52.09 ± 0.54 | 57.94 ± 0.50 |
| HOPc w/ GR (anchored) | n/a | 42.11 ± 0.65* | 54.92 ± 0.92* | 57.37 ± 0.44 | 57.99 ± 0.30 |
| Split-CIFAR-100 | |||||
| DER++ | 12.10 ± 0.68 | 22.26 ± 0.86 | 35.22 ± 0.37 | 44.04 ± 0.89 | 50.06 ± 0.30 |
| HOPc + DER++ | 26.08 ± 0.68 | 35.81 ± 0.85 | 41.50 ± 0.86 | 45.67 ± 0.52 | 48.12 ± 0.24 |
| DER++ + STAR | 12.63 ± 1.02 | 21.84 ± 0.58 | 35.17 ± 0.51 | 44.56 ± 0.88 | 50.18 ± 0.35 |
| HOPc + DER++ + STAR | 26.39 ± 0.47 | 36.03 ± 0.60 | 41.98 ± 0.93 | 46.18 ± 0.60 | 48.11 ± 0.37 |
| DER++ + LPR | 12.91 ± 1.28 | 27.59 ± 0.70 | 51.31 ± 0.64 | 56.74 ± 0.70 | 57.43 ± 0.64 |
| HOPc + DER++ + LPR | 26.79 ± 2.00 | 40.69 ± 1.91 | 53.78 ± 0.44 | 56.80 ± 0.60 | 58.72 ± 0.32 |
| HOPc w/ GR (anchored) | n/a | 44.90 ± 0.63* | 56.08 ± 0.18 | 57.40 ± 0.51 | 58.39 ± 0.17 |
* Below the class threshold we store the class means as a sketch (r random coordinates per class, basis free, all inside ), with one anchor per task, chosen on validation. n/a: configuration does not fit ().
We will add these results to Section 3.1, with implementation details and exhaustive grids in App. E.7. We will also discuss both methods and their overlap with HOPc in the related work.
We hope that these additions address your questions, and we thank you again for your positive assessment.
References
[1] Eskandar et al. STAR: Stability-Inducing Weight Perturbation for Continual Learning. ICLR 2025.
[2] Yoo et al. Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning. ICML 2024.
We sincerely thank the reviewer for their positive and thoughtful assessment of the paper. We added both requested evaluations under the paper's protocol: Split-CIFAR-100, and two recent replay methods, each alone and combined with HOPc. All numbers are over 5 test seeds, paired by seed.
Q1: “The empirical comparison relies primarily on older continual-learning baselines [...] Evaluating the correction on stronger recent replay-based methods would help establish its general applicability.”
Thank you for the suggestion. Since HOPc corrects the update rather than the loss, we believe the most informative test is a recent replay enhancement that is itself a plug-in, run alone and combined with HOPc. We added two:
STAR [1] (ICLR 2025) adds a KL term that keeps the network's outputs on buffer samples stable under a worst-case weight perturbation.
LPR [2] (ICML 2024) preconditions each layer's gradient with the layer's inputs on the buffer, so that each step changes the layer's outputs on past inputs little.
We re-implemented both from the original repositories and tuned them on the validation seeds for every benchmark and base, on grids that cover the official settings, extended wherever the best value sat on an edge.
We also report a sketched version of the anchored HOPc w/GR from Table 1 (Sec 3.1) that extends the method where Nclasses>Nreplays, a setting the former could not implemented by design (3 table entries in total). Each class mean is stored as a sketch, keeping its first r coordinates in a random orthonormal basis drawn from the run seed (shared by all classes), with the remaining directions filled in by one global mean shared by all classes. A single vector thus holds what the classes have in common, while each additional vector stores a summary of their diversity. We extend HOPc w/GR in this way whenever Nclasses>Nreplays>Nclasses/Ntasks, decided before evaluation.
This variant will be marked by * and will be detailed in App. 3.1.
Full table at match replay-capacity below.