Learning to Drift: Multimodal Continual Learning via Differentiable Delta Memory

Multimodal continual learning (MCL) aims to learn from a sequence of tasks involving multiple modalities while retaining previously acquired modality-specific knowledge and cross-modal relationships. A key challenge in MCL is the drift of representations across tasks. Despite numerous representation compensation methods, most address modality-specific drift in isolation and struggle with cross-modal interactions in MCL. This paper proposes DriftMem, a novel MCL method that learns to drift through associative memory. Specifically, DriftMem integrates Differentiable Delta Memory (DDM) and Attention-guided Gradient Recovery (AGR). The former formulates associative memory as differentiable delta-rule updates, and the latter uses attention to guide restorative gradients and stabilize cross-modal interactions. To address class conflicts in memory, DriftMem further incorporates Class-structured Feature Calibration (CFC) with two auxiliary losses for reliable memory retrieval and updates. Experiments on four datasets in the general yet challenging MCL setting of class-incremental learning show remarkable performance improvements over 10 baselines, along with ablation studies supporting the significant gains from each component.