acceptodds
Under review as a conference paper at ICLR 2027

Learning (and Forgetting) as Storage and Compression

Abstract

Dep neural networks exhibit strong generalization, yet the mechanisms underlying feature learning and its failure to learn under distributional shift (catastrophic forgetting) remain poorly understood. We introduce the Storage, Compression, and Targets (SCT) framework, which unifies neural learning as the interplay between storing associations, compressing representations, and aligning neural activity with local targets. We show that classical credit assignment algorithms—including backpropagation, predictive coding, and target propagation—can be formulated as specific instances of SCT rules. Through this lens, we identify compression as a primary driver of feature learning: once associations are stored, compression alone can reorganize their retrieval geometry, producing feature learning and generalization without further storage. However, the same mechanism can also drive forgetting. By studying the representation drift due to storage and compression rules, we prove that fully preventing task degradation requires corrections coupled to the current sample and, once representations reorganize, access to past inputs through replay or a generative proxy. Finally, we empirically validate our findings by deriving a replay-based correction mechanism that improves retention and sample-efficiency of replay-based methods for class-incremental learning.

Primary area:
learning theory
Keywords:
Learning theory, generalization, catastrophic forgetting, continual learning and class incremental learning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_15KZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Storage, Compression, and Targets (SCT) framework, which unifies neural learning as the interplay between storing associations, compressing representations, and aligning neural activity with local targets. The authors prove that fully preventing task degradation requires corrections coupled to the current sample, and derive a replay-based correction mechanism that improves retention of replay-based methods for class-incremental learning.

Critical strengths
  1. The paper reframes learning as the layer-wise interplay of Storage, Compression, and Targets (SCT).
  2. The paper proposes a correction term of the forgetting drift due to backpropagation that shifts how replays are used. The authors empirically show on a class-incremental benchmark that this geometry-aware correction significantly improves retention and sample efficiency of replay-based methods.
Critical weaknesses
  1. The empirical comparison relies primarily on older continual-learning baselines, such as DER++. While these remain useful references, the absence of recent competitive replay-based methods makes it difficult to assess the proposed method’s advantage over the current state of the art. Evaluating the correction on stronger recent replay-based methods would help establish its general applicability.
  2. The evaluation does not include Split CIFAR-100 which is often used in CL study.
Questions for authors
  1. If possible, please add comparison with more recent work in CL.
  2. Please include the evaluation on CIFAR-100
2 comments

We sincerely thank the reviewer for their positive and thoughtful assessment of the paper. We added both requested evaluations under the paper's protocol: Split-CIFAR-100, and two recent replay methods, each alone and combined with HOPc. All numbers are over 5 test seeds, paired by seed.

Q1: “The empirical comparison relies primarily on older continual-learning baselines [...] Evaluating the correction on stronger recent replay-based methods would help establish its general applicability.”

Thank you for the suggestion. Since HOPc corrects the update rather than the loss, we believe the most informative test is a recent replay enhancement that is itself a plug-in, run alone and combined with HOPc. We added two:

  • STAR [1] (ICLR 2025) adds a KL term that keeps the network's outputs on buffer samples stable under a worst-case weight perturbation.

  • LPR [2] (ICML 2024) preconditions each layer's gradient with the layer's inputs on the buffer, so that each step changes the layer's outputs on past inputs little.

We re-implemented both from the original repositories and tuned them on the validation seeds for every benchmark and base, on grids that cover the official settings, extended wherever the best value sat on an edge.

We also report a sketched version of the anchored HOPc w/GR from Table 1 (Sec 3.1) that extends the method where Nclasses>NreplaysN_{classes} > N_{replays}, a setting the former could not implemented by design (3 table entries in total). Each class mean is stored as a sketch, keeping its first r coordinates in a random orthonormal basis drawn from the run seed (shared by all classes), with the remaining directions filled in by one global mean shared by all classes. A single vector thus holds what the classes have in common, while each additional vector stores a summary of their diversity. We extend HOPc w/GR in this way whenever Nclasses>Nreplays>Nclasses/NtasksN_{classes} > N_{replays} > N_{classes}/N_{tasks}, decided before evaluation.

This variant will be marked by * and will be detailed in App. 3.1.

Full table at match replay-capacity below.

Q2: “The evaluation does not include Split CIFAR-100”

This is also a fair point. We now also consider Split-CIFAR-100 (10 tasks of 10 classes), which results we report below. The JOINT training reaches 67.64 ± 0.15 and naive fine-tuning 9.91 ± 0.04. OGD stays at the naive level (7.8–9.7), as on Split-Tiny-ImageNet.

Final class-incremental accuracy (%), mean ± sd for replay buffer size BB, bold for best and italic for second best:

B = 10502005001000
Split-CIFAR-10
DER++35.48 ± 4.3464.46 ± 2.5476.12 ± 0.9680.80 ± 0.5382.59 ± 0.27
HOPc + DER++75 ± 2.0277.65 ± 0.9780.22 ± 0.4982.01 ± 0.2682.74 ± 0.20
DER++ + STAR39.03 ± 2.2066.16 ± 3.0077.86 ± 1.1780.93 ± 0.3082.00 ± 0.84
HOPc + DER++ + STAR74.73 ± 1.7679.82 ± 0.3982.12 ± 0.2582.75 ± 0.6183.28 ± 0.41
DER++ + LPR44.15 ± 1.5363.10 ± 1.7975.84 ± 0.3581.59 ± 0.2983.37 ± 0.17
HOPc + DER++ + LPR71.45 ± 2.3573.79 ± 0.7779.12 ± 0.4682.07 ± 0.3883.05 ± 0.22
HOPc w/ GR (anchored)78.23 ± 0.2381.93 ± 0.1184.67 ± 0.1584.96 ± 0.1184.75 ± 0.24
Split-Tiny-ImageNet
DER++15.27 ± 0.7926.99 ± 0.3940.28 ± 0.7347.17 ± 0.1951.15 ± 0.79
HOPc + DER++25.38 ± 2.7332.72 ± 1.5541.99 ± 0.4148.54 ± 0.4552.17 ± 0.27
DER++ + STAR14.91 ± 0.8027.32 ± 0.9940.25 ± 0.7347.05 ± 0.5551.33 ± 0.68
HOPc + DER++ + STAR26.25 ± 3.1532.71 ± 1.0841.98 ± 0.3948.35 ± 0.6252.19 ± 0.60
DER++ + LPR16.01 ± 1.1624.48 ± 0.9537.00 ± 0.5250.23 ± 0.5557.12 ± 0.39
HOPc + DER++ + LPR27.46 ± 4.7231.43 ± 0.6441.00 ± 0.4352.09 ± 0.5457.94 ± 0.50
HOPc w/ GR (anchored)n/a42.11 ± 0.65*54.92 ± 0.92*57.37 ± 0.4457.99 ± 0.30
Split-CIFAR-100
DER++12.10 ± 0.6822.26 ± 0.8635.22 ± 0.3744.04 ± 0.8950.06 ± 0.30
HOPc + DER++26.08 ± 0.6835.81 ± 0.8541.50 ± 0.8645.67 ± 0.5248.12 ± 0.24
DER++ + STAR12.63 ± 1.0221.84 ± 0.5835.17 ± 0.5144.56 ± 0.8850.18 ± 0.35
HOPc + DER++ + STAR26.39 ± 0.4736.03 ± 0.6041.98 ± 0.9346.18 ± 0.6048.11 ± 0.37
DER++ + LPR12.91 ± 1.2827.59 ± 0.7051.31 ± 0.6456.74 ± 0.7057.43 ± 0.64
HOPc + DER++ + LPR26.79 ± 2.0040.69 ± 1.9153.78 ± 0.4456.80 ± 0.6058.72 ± 0.32
HOPc w/ GR (anchored)n/a44.90 ± 0.63*56.08 ± 0.1857.40 ± 0.5158.39 ± 0.17

* Below the class threshold we store the class means as a sketch (r random coordinates per class, basis free, all inside BB), with one anchor per task, chosen on validation. n/a: configuration does not fit (B≤Nclasses/NtasksB\le N_{classes}/N_{tasks}).

We will add these results to Section 3.1, with implementation details and exhaustive grids in App. E.7. We will also discuss both methods and their overlap with HOPc in the related work.

We hope that these additions address your questions, and we thank you again for your positive assessment.

References

[1] Eskandar et al. STAR: Stability-Inducing Weight Perturbation for Continual Learning. ICLR 2025.

[2] Yoo et al. Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning. ICML 2024.

Reviewer_erfpOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified theoretical framework called Storage–Compression–Targets (SCT) to explain feature learning, generalization, and catastrophic forgetting in neural networks from the perspective of storage and compression. The authors decompose neural network weight updates into three components: storage, compression, and targets, and analyze the connections between this framework and existing learning algorithms, including backpropagation, predictive coding, and target propagation. Building on this framework, the paper employs random matrix theory to investigate how compression reshapes the spectral structure of feature representations, thereby influencing generalization, overfitting, and the forgetting of previously learned knowledge. To address catastrophic forgetting, the authors further derive theoretical conditions for preserving previous knowledge and propose a correction mechanism called HOPc, which reduces interference with previously learned knowledge during new-task training by preserving the principal directions of past representations and their corresponding mappings. Finally, experiments on Split-MNIST, Split-CIFAR-10, and Split-Tiny-ImageNet demonstrate that HOPc can significantly improve continual learning performance in certain low-memory settings and can be combined with existing experience replay methods.

Critical strengths

1. A unified theoretical framework for understanding learning and forgetting.

The paper introduces the SCT framework, providing a unified perspective on feature learning, generalization, and catastrophic forgetting. By connecting compression dynamics with the spectral properties of learned representations, the authors offer a mathematically grounded explanation of how learning new information can interfere with previously acquired knowledge. This theoretical analysis provides valuable insights into the mechanisms underlying neural network learning and forgetting.

2. A theoretically motivated correction mechanism with promising empirical results.

Based on the analysis of forgetting dynamics, the authors propose HOPc, a correction mechanism designed to preserve previously learned representations. The method can be integrated with existing replay-based approaches and achieves substantial performance improvements in certain low-memory settings. These results demonstrate the potential practical value of the proposed theoretical framework for continual learning.

Critical weaknesses

1. Insufficient memory-budget fairness in the experimental comparisons.

According to Appendix E.4, HOPc requires additional storage for the principal components of previous representations and their corresponding mappings when combined with replay-based methods such as DER++. On Split-CIFAR-10, this overhead is equivalent to approximately 49 samples. Consequently, under the B=10 setting, HOPc + DER++ uses substantially more total memory than DER++, despite achieving its largest reported improvement of approximately 40 percentage points in this setting. Although the authors explicitly report the additional storage overhead, comparisons under strictly matched total memory budgets are missing. Therefore, the current results do not sufficiently establish whether the method maintains its advantages under equal memory constraints, weakening the claims regarding memory efficiency.

Questions for authors

The theoretical formulation of HOPc requires the correction term to satisfy the null-space constraint (Δπ x = 0), whereas the practical implementation in Eq. (22) does not appear to explicitly enforce this property. Could the authors clarify how Eq. (22) relates to the theoretical formulation and which theoretical guarantees remain valid after introducing the scaling factor and Adam optimizer?

2 comments

We sincerely thank the reviewer for their positive and thoughtful assessment of the paper, and for their careful reading. We hope to address your questions below. Please let us know if there's anything else you'd like us to clarify. All numbers are over 5 test seeds, paired by seed, unless marked as validation.

Q1: “Insufficient memory-budget fairness in the experimental comparisons.”

Thank you for raising this point; we agree. We have now rerun the comparison with HOPc + replay fitted inside the same total storage budget as the baseline, counting both replay memory and stored summaries.

With the storage budget BB expressed in sample-equivalent units, HOPc keeps kB=max⁡(1,min⁡(10,⌊B/(2c)⌋))k_B = \max(1, \min(10, \lfloor B / (2c) \rfloor)) components per layer, where cc is the combined storage cost of adding one component to every protected layer, including its direction, stored output, and eigenvalue. The replay buffer shrinks to pay for these summaries within the total budget. This allocation rule was fixed before running the experiments.

The table reports the advantage over DER++ at the same total storage budget: mean paired accuracy difference in percentage points over five test seeds. HOPc w/ GR denotes the anchored Gaussian-replay variant (App. E.3.2), with both Gaussian statistics and stored anchor outputs charged to the budget. We also report a sketched extension of the anchored HOPc w/GR that works for Nreplays<NclassesN_{replays}<N_{classes} and fits in BB, marked by * (details in Reviewer 15KZ Q1).

MethodB=10B=10505020020050050010001000
Split-CIFAR-10
HOPc + DER+++13.0+7.9+4.1+1.0+0.3
HOPc w/ GR+42.8+17.5+8.6+4.2+2.2
Split-CIFAR-100
HOPc + DER+++2.7+6.3+5.8+1.8−0.7
HOPc w/ GRn/a+22.6*+20.9+13.4+8.3
Split-Tiny-ImageNet
HOPc + DER+++2.6−0.8+0.1+0.5+0.8
HOPc w/ GRn/a+15.1*+14.6*+10.2+6.8

* Below the class threshold we store the class means as a sketch (r random coordinates per class, basis free, all inside BB), with one anchor per task, chosen on validation. n/a: configuration does not fit (B≤Nclasses/NtasksB\le N_{classes}/N_{tasks}).

These comparisons show that benefits remain when summaries are paid for within the storage budget. We will report these storage-matched comparisons in Section 3.1 and detail the accounting and allocation rule in Appendix E. Further comparisons, including the DER++ buffer size required to reach comparable accuracy, appear in our storage-matching response to Reviewer c34M.

Q2: “Could the authors clarify how Eq. (22) relates to the theoretical formulation and which theoretical guarantees remain valid after introducing the scaling factor and Adam optimizer?”

Thank you for this careful question. You are right that Eq. 22 does not enforce Δπx=0\Delta\pi x = 0, and we will correct the statement in Appendix E.3.1 suggesting that linear independence alone makes it exact. The theoretical statements remain valid under their stated assumptions, the clarification regards which per-step properties of the correction survive in the practical implementation.

Which theoretical guarantees remain valid after introducing the scaling factor and Adam optimizer?

Exact correction can be enforced under Adam by applying Eq. 17 to the Adam step, and in general for any optimizer by considering their effective step. We now report it, and it improves the base method on its own. App. D.3 states Eq. 17 for any base update ΔW0\Delta W_0 (SGD, Adam, or the full DER++ step). Applied to the step Adam actually takes, its guarantees hold exactly, whatever Adam's normalization. We implemented it and verified the exactness constraints to ≤10−10\le 10^{-10} (relative) over whole test runs. We report it on the test seeds with HOPc + DER++ on all benchmarks, without the restoring term, in two versions:

  • With a trust region (correction closest to Eq. 17 with norm ≤ 3× the Adam step), the current step is still kept exactly, and 99 % of the per-step drift on the protected maps is cancelled. It improves the DER++ base on its own: +24.7 / +8.6 / +9.2 / +4.9 pp at B = 10 on CIFAR-10 / CIFAR-100 / Tiny-ImageNet / MNIST, and +2.7 pp pooled over 100 paired runs.

  • Strictly exact, it also improves the base at B = 10 on CIFAR-10 and MNIST (+26.2 / +23.3 pp, 5/5 seeds), where it needs 3–5× the Adam step on average. On Tiny-ImageNet and CIFAR-100 it needs 6–16× the step on average (single steps up to 207×), and new tasks are barely learned (plasticity 0.34 vs 0.81 on Tiny-ImageNet), while the constraints still hold.

These observations are consistent with the conditioning dependence established in Prop. D.2 and App. D.3: stronger batch–memory alignment can require larger exact corrections. In our experiments, these large updates impaired learning of new tasks. Therefore, exactness guarantee holds, but its cost in plasticity grows with this overlap. A trust region trades exactness for plasticity, and the restoring term repairs the remaining drift instead of preventing it at every step.

As for the scaling factor, multiplying the exact nullspace correction by λ\lambda preserves its nullspace property, but generally weakens memory protection. For example, with ε=0\varepsilon=0 (no restoration/elastic force),

ΔW=ΔW0−λ(ΔW0M)A†\Delta W=\Delta W_0-\lambda(\Delta W_0M)A^\dagger

satisfies

\qquad \Delta WM=(1-\lambda)\Delta W_0M.$$ Therefore exact cancellation of the proposed memory drift generally requires $\lambda=1$. **Clarify how Eq. (22) relates to the theoretical formulation.** Eq. 5 (one sample, Prop. D.2) and Eq. 17 (a batch, App. D.3) apply the correction along the part of each memory that is orthogonal to the current inputs ($z^\mu_\perp$ and $A = \Pi^\perp_X M$, respectively), which is why they leave the current step unchanged ($A^\dagger X = 0$). Eq. (22) is a cheaper per-component form for the orthonormal stored directions $u_j$, and it applies the correction along $u_j$ itself. It acts as a restoring force over protected activations: these protected directions stay within 0.1–1 % of their outputs at the task boundary. For one-row batches, $\epsilon=0$ and $\lambda=1$, this is equivalent to gradient projection when applied before Adam. Here we trade the guarantees of a preventive approach (as in Eq.17) with the flexibility of a reactive one. The practical method therefore combines approximate prevention of drift with a corrective force that penalizes deviations from stored outputs. The coefficient $\lambda$ controls the strength of this intervention empirically and does not preserve the exact construction’s guarantees. See answer to Reviewer c34M (Q2) for a discussion about the step adjustment effect versus restoration. We will revise Sec. 3.1, App. E.3.1, and the Discussion to distinguish these properties from the exact guarantees of Eqs. (5) and (17).
Reviewer_c34MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Storage, Compression, and Targets (SCT) framework to explain learning and forgetting in neural networks through the interaction of these three components. It expresses the layer-wise updates of backpropagation, predictive coding, and target propagation within a common structure. The paper presents the view that compression changes how stored information is retrieved, driving generalization and overfitting while also contributing to forgetting. Building on this perspective, the authors analyze how learning new information affects past representations and propose HOPc, which protects and restores layer responses along principal feature directions. Theoretical predictions are supported by simulations and memory-recovery examples. In class-incremental learning experiments on Split-MNIST, Split-CIFAR-10, and Split-Tiny-ImageNet, the authors report performance improvements when HOPc is combined with existing replay methods, particularly under small sample-buffer settings.

Critical strengths
  1. The paper organizes different learning algorithms within a common framework of storage, compression, and layer-wise target assignment. It also provides a concrete analysis of how the prediction error on a new input and its alignment with past representations affect previously stored memories. Rather than treating learning and forgetting as separate problems, this perspective helps explain how the same update can contribute both to acquiring new knowledge and to disrupting existing knowledge.

  2. HOPc uses principal directions of past representations and their reference outputs to protect and restore layer responses. Because it corrects updates rather than replacing the loss function, it can be combined with ER, DER++, and Gaussian replay. The reported improvements over the corresponding baselines in several small-buffer settings are encouraging and demonstrate a useful connection between the proposed analytical framework and the design of a practical method for mitigating forgetting.

Critical weaknesses
  1. HOPc presents correcting memory drift while preserving learning on the current input as a key advantage. However, the current results do not sufficiently distinguish the effect of this update adjustment from that of restoring the model’s responses to stored past feature directions to their original values. Although the analysis demonstrates the importance of the restoration term, it remains unclear how much the full HOPc method improves over a restoration-only approach using the same stored information and replay setup. Such a comparison is needed to establish the practical benefit of the proposed geometric correction beyond simply maintaining past responses.

  2. When combined with replay, HOPc stores principal feature directions and their reference outputs in addition to the past-sample buffer. Therefore, performance improvements at the same buffer size cannot readily be interpreted as efficiency gains under the same total memory budget. Although the additional storage cost is disclosed, it is not sufficiently established whether HOPc retains an advantage when the baseline replay method is given an equivalent amount of additional storage. Since effectiveness under memory constraints is presented as a major contribution, comparisons should match the total storage budget, including the stored summary information.

Questions for authors

Refer to the Weaknesses section.

3 comments

We thank the reviewer for their constructive feedback and for taking the time to review our work. Both weaknesses ask for controls that we agree are needed to support our claims, and we have now run both in full on our benchmarks. We address your questions below; please let us know if there's anything else you'd like us to clarify. All numbers are over 5 test seeds, paired by seed.

Q1: “comparisons should match the total storage budget, including the stored summary information.”

Thank you; we agree that claims about memory efficiency should be tested at equal total stored memory footprint. The submission reports the summaries’ cost (App. E.4) and states before Table 1 that they are stored in addition to the replay buffer. However, those comparisons match replay-buffer capacity rather than total storage. We have now rerun the comparison with HOPc + replay fitted inside the baseline’s total storage budget.

With the storage budget BB expressed in sample-equivalent units, HOPc keeps kB=max⁡(1,min⁡(10,⌊B/(2c)⌋))k_B = \max(1, \min(10, \lfloor B / (2c) \rfloor)) components per layer, where cc is the combined storage cost of adding one component to every protected layer, including its direction, stored output, and eigenvalue. The replay buffer shrinks to pay for these summaries within the total budget. This allocation rule was fixed before running the experiments. Table 1's strongest row, HOPc w/ GR (anchored variant, App. E.3.2), keeps few Gaussian means that work both for the sampling and to anchors on those same directions, so it fits inside B in both budgets already as submitted. We also report a sketched version of the anchored HOPc w/GR that works on Nreplays<NclassesN_{replays}<N_{classes} and fits in BB, marked by * (details in Reviewer 15KZ Q1).

The table reports the advantage over DER++ at equal total memory storage: mean paired accuracy difference in percentage points over five test seeds.

MethodB=10B=10505020020050050010001000
Split-CIFAR-10
HOPc + DER+++13.0+7.9+4.1+1.0+0.3
HOPc w/ GR+42.8+17.5+8.6+4.2+2.2
Split-CIFAR-100
HOPc + DER+++2.7+6.3+5.8+1.8−0.7
HOPc w/ GRn/a+22.6*+20.9+13.4+8.3
Split-Tiny-ImageNet
HOPc + DER+++2.6−0.8+0.1+0.5+0.8
HOPc w/ GRn/a+15.1*+14.6*+10.2+6.8

* Below the class threshold we store the class means as a sketch (r random coordinates per class, basis free, all inside BB), with one anchor per task, chosen on validation. n/a: configuration does not fit (B≤Nclasses/NtasksB\le N_{classes}/N_{tasks}).

As a complementary measure, we estimate the DER++ storage budget required to reach comparable accuracy. At B=50B=50, HOPc + DER++ reaches accuracy corresponding to approximately 2.3×2.3\times its storage budget for DER++ on CIFAR-10 (about 115 replay samples) and approximately 2.0×2.0\times on CIFAR-100.

We also fitted HOPc alone, without replay during training, inside the storage budget. At B=10B=10, its mean gains over DER++ are +44+44, +26+26, +7+7, and +11+11 pp on MNIST, CIFAR-10, Tiny-ImageNet, and CIFAR-100, respectively, with improvements on all five seeds. It falls behind DER++ at B≥500B\geq500, where replays count the most.

We will report these storage-matched comparisons in Section 3.1 and provide the full results in App. E.7. App. E.4 will explicitly list all persistent storage counted for each method. We will also revise the Discussion and experimental claims to distinguish replay-buffer capacity from total storage.


Q2: “the current results do not sufficiently distinguish the effect of this update adjustment from that of restoring the model’s responses to stored past feature directions to their original values”

Thank you for requesting this comparison. We agree that the contribution of the adjustment must be tested against restoration alone, with the same stored directions, targets, and replay buffer. We have now performed this direct ablation, tuning the restoring strength separately for each arm on validation and comparing paired test seeds.

What the adjustment contributes beyond restoration.

Under the submitted protocol, adding the approximate adjustment of Eq. (22) (see Rev erfp, Q2) produces little additional final-accuracy benefit over restoration alone (+0.2 pp pooled over 225 paired runs on CIFAR10, CIFAR100 and Tiny-ImageNet; −0.4 pp on 75 runs for Split-MNIST). We will state explicitly in Sec. 3.1 that restoration accounts for most of the gain of the evaluated method in low-buffer settings, while both effects are small in high-BB regimes.

To determine whether the geometric adjustment itself can add to restoration, we applied its exact form (Eq. 17) to the step the optimizer actually takes (see Rev erfp, Q2), capped at 1× that step, on top of restoration alone. Both arms use the same stored directions, targets, replay buffer and seeds. The restoring strength (and the cap) was selected on validation seeds separately for each arm, so the comparison is against a separately tuned restoration-only baseline. Split-MNIST (DER++, MLP 784–100–100–10, all layers trained), 20 epochs per task, paired differences over 3 test seeds:

OptimizerBuffer BBAccuracy gain over restoration aloneForgetting: restoration → + adjustmentPlasticity: restoration → + adjustment
SGD10+3.4 pp (3/3)0.552 → 0.5070.991 → 0.987
SGD50+6.7 pp (3/3)0.475 → 0.3920.988 → 0.990
SGD500+0.6 pp (2/3)0.168 → 0.1610.988 → 0.989
Adam10+3.0 pp (3/3)0.457 → 0.4100.985 → 0.976
Adam50+1.1 pp (3/3)0.264 → 0.2540.964 → 0.968
Adam500+0.1 pp (2/3)0.144 → 0.1480.977 → 0.981

At small buffers the adjustment lowers forgetting at essentially unchanged plasticity, under both optimizers. Strengthening restoration alone does not close the gap: under SGD at B=50B = 50, raising its weight tenfold changes its final accuracy by only +0.4 pp, while adding the adjustment adds +6.7 pp.

Why we relax exact enforcement, and when the benefit disappears.

Prop. D.2 and App. D.3 establish that exact adjustment can require large updates when current and protected inputs are closely aligned. Without the cap, the SGD variant diverged on every seed, consistent with this conditioning problem. Capping (or ridge damping) controls the correction's magnitude while retaining its current-input nullspace property, at the cost of relaxing exact memory protection. Moreover the benefit is regime-dependent: it is largest at small buffers and vanishes at B=500B = 500, where replay already limits forgetting.

We will revise Sec. 3.1 and App. E.3.1 to distinguish the two contributions: restoration repairs accumulated deviations, whereas the step-preserving adjustment additionally reduces their formation. The new ablations demonstrate this complementary benefit for the exact adjustment applied to the optimizer's step, under both SGD and Adam, while the submitted implementation (Eq. 22) remains primarily supported by its restoring component.

Dependency on geometry's accuracy

The adjustment ablation above holds the stored directions fixed. We additionally tested the choice of those directions, keeping the summary rank kk, summary storage, replay buffer, and correction rule fixed (HOPc + DER++, B=10). Task principal components outperform principal components estimated from the replay buffer, the span of kk sampled task representations, and random directions.

task PCs (from all past inputs)PCs of the replay bufferspan of k task samplesk random directionsno restoration (DER++ base)
Split-CIFAR-1075.444.358.535.335.1
Split-Tiny-ImageNet25.415.520.916.415.4

This supports the geometric choice of what to preserve: with the same summary storage, directions capturing more of the past representations’ variance provide better retention. The spectral identity in Sec. 3.1 explains this motivation by weighting each direction’s contribution to direct pre-activation drift by its variance. We will report this ablation in appendix.

We hope that these results address your concerns, and we would be grateful if you would take them into account in your final assessment.

Thank you for the additional experiments and explanations. The comparisons under matched total memory budgets address my concern about memory efficiency, and the restoration-only comparisons help clarify the contributions of the proposed method. Considering these results and the theoretical contributions of the paper, I have decided to raise my rating to 3.

Comments on the paper

1 comment

We thank all three reviewers for their time, careful reading, and constructive feedback. Their questions substantially helped us refine the paper and its narrative, clarify the relationship between the theory and the practical method, and strengthen the experimental evidence on our benchmarks. We address each point below and will incorporate these improvements into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.