Beyond Experience Replay: Learning Structural Memory for Online Continual Learning

Experience replay is a widely adopted strategy for online continual learning (OCL), yet conventional replay memories mainly preserve historical observations and do not explicitly retain how previously learned concepts are organized. We show that continual updates can alter the relational organization of historical representations, leading to structural drift even when replay is employed. Motivated by this observation, we propose Structural Memory Consolidation (SMC), a memory-efficient framework that augments episodic replay with an additional structural memory state. SMC constructs compact structural snapshots from representative anchors at task transitions and consolidates these historical relational states during subsequent online adaptation. By preserving the organization among learned concepts rather than directly storing high-dimensional representations or constraining individual features, SMC enables more stable knowledge evolution while maintaining model plasticity. Extensive experiments on multiple online Class-IL benchmarks demonstrate that SMC consistently improves knowledge retention and representation stability across diverse memory budgets, with particularly strong benefits under memory-constrained settings.