acceptodds
Withdrawn from ICLR 2027

Beyond Experience Replay: Learning Structural Memory for Online Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Experience replay is a widely adopted strategy for online continual learning (OCL), yet conventional replay memories mainly preserve historical observations and do not explicitly retain how previously learned concepts are organized. We show that continual updates can alter the relational organization of historical representations, leading to structural drift even when replay is employed. Motivated by this observation, we propose Structural Memory Consolidation (SMC), a memory-efficient framework that augments episodic replay with an additional structural memory state. SMC constructs compact structural snapshots from representative anchors at task transitions and consolidates these historical relational states during subsequent online adaptation. By preserving the organization among learned concepts rather than directly storing high-dimensional representations or constraining individual features, SMC enables more stable knowledge evolution while maintaining model plasticity. Extensive experiments on multiple online Class-IL benchmarks demonstrate that SMC consistently improves knowledge retention and representation stability across diverse memory budgets, with particularly strong benefits under memory-constrained settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Online Continual Learning, Catastrophic Forgetting, Memory Representation, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_x5F9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Structural Memory Consolidation for replay-based online class-incremental learning. At task boundaries, it selects anchors from the replay buffer and stores row-normalized similarity matrices between anchors from adjacent historical tasks. Subsequent updates reconstruct those relations and distil toward the stored matrices alongside the base training objective. Experiments use randomly initialized ResNet-18 models on CIFAR-10, CIFAR-100, ImageNet-100, and Tiny-ImageNet. The paper reports the highest final accuracy in 11 of 12 evaluated settings when either anchor-selection variant is allowed, and uses representation-drift measurements to examine the proposed mechanism.

Critical strengths

1 The proposed memory representation is compact and addresses a reasonable question. Storing relation matrices can preserve a previous encoder's geometry without retaining that encoder or full historical embeddings (Section 3.2). The adjacent-task construction limits snapshot storage to order (t−2)K2(t-2)K^2 scalars. Evaluating structural drift using class-prototype similarities rather than exactly the anchor-level training loss is also a useful diagnostic choice (Section 4.5). These are worthwhile aspects of the design, although their benefit needs a consistent evaluation and complete resource accounting.

Critical weaknesses
  1. The ablation and implementation description do not isolate the claimed contribution. Table 3 reports an increase from 12.35% to 19.64% when structural snapshots are retained but their consolidation objective is removed. Under the stated procedure, merely caching unused targets should not alter training; any remaining difference needs to be specified. Conversely, consolidation without stored structural targets is not defined.
Questions for authors
  1. Please provide the exact forgetting formula and the per-task accuracy matrices behind the ER and DVC Tiny-ImageNet entries. Were Tables 2 and 4 produced with the same runs and evaluation protocol?
  2. How does a matched relational-distillation baseline perform, and what is the complete extra storage and computation cost? In particular, how are cached anchors retained if reservoir sampling would otherwise evict them?
Reviewer_JtHuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Structural Memory Consolidation (SMC) for online class-incremental learning. The method replays past samples' relations to prevent relational drift during standard sample replay.

Critical strengths
  1. The method is simple and intuitive. Only per-task similarity matrices are added, and they are small in size.
  2. Performance is good compared to recent online continual learning baselines
Critical weaknesses
  1. How does SMC differ from Co^2L? The loss is the same, and the only difference seems to be how the similarity information is cached.
  2. Co^2L is similar to SMC and discussed in related work, but not shown in benchmarks.
  3. The values in Tables 2, 4, and 5 lack statistical significance
  4. The training latency in Figure 5 is a large-batch offline setting, which does not match the main benchmarks in the main paper's online settings.
  5. One challenge of online continual learning is the lack of task boundaries; it's unknown how this method can be adapted to such cases, and how well it performs. Since the design seems to be dependent on clearly defined task boundaries.
Questions for authors
  1. SMC never preserves a task’s original relational structure, only the structure as of the last task boundary; thus, drift can accumulate across tasks. Have the authors compared this against keeping each task’s first snapshot fixed?
  2. What's the difference between Co2L and SMC? SMC does not store the full model, so it is cheaper in memory than Co2L, but I expect this memory saving incurs some cost on relation drift?
Reviewer_ooCEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that reenacting experiences preserve samples from the past but not the relational organization among learned concepts. As a result, representations undergo “structural drift” even when recall is used. The proposed method, called structural memory consolidation (SMC), adds “structural memory” to the episodic buffer to enable incremental class learning.

Critical strengths
  1. The loss of relationships involving several reference points (anchors) can be accounted for in any reconstruction method, and storing snapshots involves very low costs.

  2. The article is well-written and well-reasoned. It includes numerous illustrations, pseudocode, and a table of hyperparameters.

Critical weaknesses
  1. “Structural memory” is merely a cached, ephemeral relational distillation, not a new type of memory. A snapshot is a similarity matrix from the encoder of the previous task, which is distilled into the current model in a manner analogous to methods such as RKD or Co2L. The only difference is caching—that is, precomputing results for reference points instead of retaining the old model. Since snapshots are recalculated at every task boundary, this constitutes standard single-stage distillation rather than a persistent historical structure. Consequently, Table 1 erroneously suggests that the relational methods KD and Co2L lack an “explicit relational state.” see the following articles: Cha, H et al., Co2L: Contrastive continual learning. ICCV 2021. Tung, F. and Mori, G., Similarity-preserving knowledge distillation, ICCV 2019 Park, W. et al., Relational knowledge distillation. CVPR 2019

So, authors may propose that such a structure could be stored and maintained during the explanation. Of course, it should take into consideration memory constraints and also suggest how to manage this knowledge. I think that the proposed algorithm could store class-level relations among all seen classes (a prototype-similarity matrix, or a sparse top-k neighbor graph per class) instead of anchor-level matrices for adjacent task pairs only. I suggest reading the following works, including a very classical survey on data streams, e.g., by Gama et al.

Tao, Xet al. Topology-preserving class-incremental learning. ECCV 2020. Arani, E. et al. Learning fast, learning slow: A general continual learning method based on complementary learning system. ICLR 2022 Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), 1–37. doi:10.1145/2523813. Zhou, D.-W. et al, A model or 603 exemplars: Towards memory-efficient class-incremental learning, ICLR 2023.

  1. No standard deviations and no number of seeds are reported. Authors should present reliable results from experiments repeated multiple times to demonstrate the impact of errors arising from the choice of optimizer. Experiments should be conducted using several to a dozen randomly selected seeds to ensure we are not dealing with so-called “seed hacking,” i.e., selecting results from multiple training runs.

  2. In my opinion the ablation studies do not isolate the so-called structural component

  3. The method is a relational distillation loss, yet no relational or topology-preserving continual-learning method is compared (i) Co2L, prototype–sample relation distillation (PRD) and topology-aware knowledge preservation. All these methods were cited but not used in the experiments. Also, methods discussed in the related work are absent: OnPro and MOSE. Additionally, the following baselines should be considered (i) topology-preserving class-incremental learning (Tao et al., 2020), (ii) X-DER (Boschini et al., 2023), the successor of DER++ (which usually outperforms DER++).

Tao, X. et al., Topology-preserving class-incremental learning. ECCV 2020). Boschini, M. et al., Class-incremental continual learning into the eXtended DER-verse. IEEE PAMI, 45(5), 5497–5512.

  1. The number of anchors K and the temperatures τ_mem and τ_cur, the method’s central hyperparameters, are not reported.

  2. Feature and structural drifts are not defined in sec. 4.

  3. There is no discussion of the metric choice

Questions for authors

See weaknesses and

  1. Can you double check and argue that the results in Tab.2. and 4 are credible – I think that using the definitione of final average accuracy and forgetting we cannot get such results, especially for Tiny-ImageNet
  2. Apart from caching teacher outputs on fixed anchors, how does SMC differ from relational distillation from the previous model (e.g., Co2L’s instance-wise relation distillation)? Since snapshots are recomputed at every boundary, in what sense are they persistent?
  3. Have you tried keeping snapshots from earlier boundaries instead of recomputing them, and how does this compare at an equal total memory budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.