acceptodds
Withdrawn from ICLR 2027

Learning to Drift: Multimodal Continual Learning via Differentiable Delta Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal continual learning (MCL) aims to learn from a sequence of tasks involving multiple modalities while retaining previously acquired modality-specific knowledge and cross-modal relationships. A key challenge in MCL is the drift of representations across tasks. Despite numerous representation compensation methods, most address modality-specific drift in isolation and struggle with cross-modal interactions in MCL. This paper proposes DriftMem, a novel MCL method that learns to drift through associative memory. Specifically, DriftMem integrates Differentiable Delta Memory (DDM) and Attention-guided Gradient Recovery (AGR). The former formulates associative memory as differentiable delta-rule updates, and the latter uses attention to guide restorative gradients and stabilize cross-modal interactions. To address class conflicts in memory, DriftMem further incorporates Class-structured Feature Calibration (CFC) with two auxiliary losses for reliable memory retrieval and updates. Experiments on four datasets in the general yet challenging MCL setting of class-incremental learning show remarkable performance improvements over 10 baselines, along with ablation studies supporting the significant gains from each component.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-modal Continual Learning, Multi-view Learning, Representation Drift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CaoNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DriftMem, a method for multimodal class-incremental learning that compensates for representation drift rather than explicitly constraining it. DriftMem consists of three components: Differentiable Delta Memory (DDM), a linear associative memory that predicts drift residuals from semantic keys; Attention-guided Gradient Recovery (AGR), which introduces restorative gradients based on changes in cross-modal attention; and Class-structured Feature Calibration (CFC), which encourages a more discriminative structure in the semantic key space. The authors report consistent improvements over ten baselines across four datasets.

Critical strengths
  1. Addressing representation drift through compensation rather than restriction is a reasonable approach to balancing stability and plasticity.

  2. The method introduces relatively modest additional memory overhead and does not require task identity at inference.

  3. The analysis of γ_mem (Fig. 4) gives an intuitive picture of how compensation strength affects old- and new-class accuracy.

Critical weaknesses

1. Technical novelty of DDM. DDM appears to be a class-conditional linear corrector fitted on exemplars. Since CFC strongly clusters the keys by class, it would be helpful to clarify how much Mk actually differs from a per-class mean drift offset. An analysis supporting the claimed sample-dependent compensation would make the contribution more convincing. In addition, LDC (Gomez-Villa et al., 2024), which learns a projector for drift compensation, seems conceptually very close. It is cited but not compared against experimentally.

2. Source of the improvement. Because DDM is trained to output zero for new-class samples, part of the gain may come from correcting the classifier bias between old and new classes rather than from recovering drift. The trend in Fig. 4, where increasing γ_mem raises old-class accuracy and lowers new-class accuracy, also seems consistent with this interpretation. An experiment that disentangles these two effects would be valuable.

3. Multimodal specificity. DDM operates on the fused feature and is modality-agnostic, so the multimodal-specific part of the method appears to be mainly AGR. However, Table 2 shows that adding AGR to DDM degrades performance (FAA 64.19 → 58.57). I would appreciate a deeper analysis of this result. Given the chosen values of δ_att and τ_att, the Bernoulli gate may also be active almost all the time. Similarly, CFC brings a large gain even without DDM, so it is unclear whether its benefit is specific to reducing memory interference.

Questions for authors

Q1 Could you provide comparisons with LDC, or with simple feature distillation on exemplars?

Q2 Could you disentangle the effect of DDM from that of bias correction methods such as BiC or WA?

Q3 What does the distribution of π^t look like during training? Also, how do you explain the performance drop when AGR is combined with DDM?

Q4 Is the cosine similarity in Fig. 3 measured on replay samples or on held-out test samples?

Q5 Does DDM bring similar gains when applied to unimodal class-incremental learning?

3 comments
AuthorsOpenReview ↗

Resonpse to reviewer CaoN: Thank you very much for your review. Our response is as follows.

We sincerely thank Reviewer CaoN for the constructive feedback and valuable comments. We provide our point-by-point responses below. Should there be any further questions or suggestions on our submission, please let us know. We greatly value and appreciate each comment because they can help us clarify and improve our manuscript tremendously.

Q1/W1. Technical novelty of DDM

Thank you for your question and comments.

To explore sample-dependent contribution, we conduct experiments that compare DDM with class-wise compensation. For class cc, we define class-wise compensation as rct=1∣Ec∣∑i∈EcMkit\mathbf r_c^t = \frac{1}{ |\mathcal E_c|}\sum_{i\in \mathcal E_c}^{}\mathbf M\mathbf k_i^t, and use it to compensate the original feature $ \hat{\mathbf z}^t=\mathbf z^t+\gamma_{\mathrm{mem}}\mathbf M\mathbf r^t$. We calculated the average L2 distance between compensated features and corresponding historical representations, comparing these two compensation methods before and after compensation. Below are the results of AVE after training on the last task.

MethodsBeforeAfterReduction$\uparrow$
DDM Compensation15.7810.4833.59%
Class-wise Compensation15.7813.9611.53%

The comparison results show our sample-dependent improvement.

LDC learns a feature mapping from previous feature space to current feature space, and use a Nearest Class Mean (NCM) Classifier. Different from LDC, our DriftMem provides a new compensation mechanism by learning associations between current representations and corresponding representation drift. Although LDC also employ a sample-dependent compensation, it learns the map function based on observation of current task data, unabling to model drift of cross-modal interactions.

We also include the comparison results with LDC here. Each cell is reported in the format of CAA/FFM.

MethodsAVEKinetics-SoundsN24NewsUMPC-Food101
LDC42.46/28.8134.85/11.8870.92/15.7184.11/15.83
DriftMem80.36/5.4173.98/10.5981.98/10.7394.23/3.08

We will include the new results in the revision.

Q2/W2. Source of the improvement.

Thank you for this comment. We agree that the disentanglement of classifier bias correction and drift recovery is crucial. In fact, as shown in Fig. 4, increasing γmem\gamma_{\text{mem}} tends to improve old-class accuracy while new-class accuracy degrades. However, our objective is not to eliminate this trade-off, but rather to improve the attainable performance under such a trade-off through representation-level drift compensation.

We now conducted additional experiments by incorporating two representative bias correction methods, Weight Alignment (WA) and BiC, into the baseline without DDM. Each cell is reported in the format of CAA/FFM.

MethodsAVEKinetics-SoundsN24NewsUMPC-Food101
Base(w/o DDM)73.71/16.1164.85/20.8877.17/15.9591.07/7.98
Base+WA76.20/13.7566.25/19.9777.08/18.3491.67/7.25
Base+BiC75.51/15.0865.57/16.9476.76/15.8490.94/5.56
Base+DDM80.36/5.4173.98/10.5981.98/10.7395.23/3.08

Although WA and BiC provide certain improvements over the baseline, their gains are relatively limited. Our DDM achieves substantially better performance across all four datasets.

We additionally measure the L2 distance between the current feature representations and their corresponding historical representations, before and after applying DDM.

AVE Tasks (Max)BeforeAfterReduction$\uparrow$
128.8517.9137.92%
225.9916.6735.86%
320.2614.5828.04%
AVE Tasks (Mean)BeforeAfterReduction$\uparrow$
124.969.2962.78%
218.3411.7236.10%
315.7810.4833.59%
N24News Tasks (Max)BeforeAfterReduction$\uparrow$
133.2424.6325.90%
225.1422.4510.70%
323.0919.9813.47%
N24News Tasks (Mean)BeforeAfterReduction$\uparrow$
114.847.1252.02%
211.448.7723.34%
310.618.1023.66%

The results show that DDM substantially reduces the feature distance, indicating that the proposed memory-based compensation effectively recovers the historical representations.

Q3/W3. Multi-modal Specificity

We are grateful to you for your constructive suggestions.

As illustrated in line430, AGR alone may increase the overlap between the semantic representations of old and new classes, thereby reducing their discriminability. Following your suggestion, we measure the Silhouette scores of semantic keys across different ablation settings.

MethodSilhouette score
Base(only DDM)-0.0092
Base+AGR-0.0102
Base+AGR+CFC0.0704

The results shows that CFC effectively reduces memory interference, while applying AGR alone could increase memory interference.

Q4. Cosine similarity in Fig. 3

The cosine similarity in Fig. 3 is measured on test samples.

Q5. DDM for unimodal class-incremental learning

DDM also achieves comparable performance gains in unimodal settings. To further investigate this, we conduct ablation experiment using only the context modality and report the results on the AVE dataset. Each cell is reported in the format of CAA/FFM.

MethodSilhouette score
Base model57.75/14.81
Base model + DDM62.35/11.40

As illustrated in line 253-255, our delta memory is able to compensate modality-specific features and cross-modal aggregation, achieving a more fine-grained compensation. Thus, DDM plays a more significant role in multimodal scenarios.

Dear Reviewer CaoN,

We sincerely appreciate your thoughtful feedback and valuable suggestions.

We hope that our responses and clarifications have addressed your concerns. As the discussion period is approaching its deadline, we would greatly appreciate your further feedback.

If our responses have changed your assessment, we would be grateful if you could consider revisiting your rating accordingly. We are also very happy to address any remaining concerns.

Thank you for your time and consideration.

Best regards,

Submission9485 Authors

Reviewer_zgN2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses multimodal class-incremental learning, where a model learns tasks with disjoint labels, is evaluated without task identity, and may retain a small buffer of past multimodal features. Its core idea is the “representation drift dilemma”: limiting drift improves stability but hurts plasticity, while allowing drift improves plasticity but causes forgetting. DriftMem instead learns and undoes drift by adding three components to a cross-modal attention fusion head over frozen features: Differentiable Delta Memory (DDM), which maps semantic keys to replay-sample drift residuals via a delta rule and compensates features at inference; Attention-guided Gradient Recovery (AGR), which uses attention-map JS divergence to trigger bounded restoring gradients; and Class-structured Feature Calibration (CFC), which adds prototype-orthogonality and supervised contrastive losses. The paper also provides memory-update derivation, conditional convergence, a necessary-and-sufficient condition for drift reduction, and an interference bound. Experiments cover four video-audio/image-text datasets against ten baselines, with accuracy, forgetting, ablations, sensitivity studies, alternative orders, longer sequences, and visualizations.

Critical strengths
  1. The paper is well written and easy to follow. The motivation, problem setting, and technical components are presented clearly and in an accessible manner, which makes the main ideas understandable to a broad readership.
  2. Honest self-reporting in places where it costs the authors. The limitations paragraph (L482–485) names the d×d memory cost and replay-quality dependence; the results discussion concedes that CMR‑MFN achieves lower FFM in cases (L415) and that AGR alone hurts (L430); the λc study concedes that N24News FFM is worse than the zero-weight baseline (L1053–1054); Table 7 concedes that DriftMem does not always have the lowest FFM.
Critical weaknesses
  1. Memory-budget fairness is unaddressed. DriftMem’s buffers are large (340/500/1000/1200, i.e., 10.32% of AVE’s training set; Table 4), but the paper never states the baselines’ buffer sizes. The compared MMAL is exemplar-free and does not use a replay mechanism, making the comparison unfair. Moreover, after removing the replay-based DDM module from DriftMem (ablation results in Table 2), DriftMem underperforms MMAL on all three performance metrics. I therefore question the methodological advancement of this approach.
  2. The experimental record is not self-consistent, and the inconsistency is largest exactly where the hyper-parameters were chosen (major). At the adopted AVE setting, Table 6 reports 81.53 CAA / 4.36 FFM while Table 1 reports 79.23 / 6.89. A 2.3-point gap that is comparable to the paper's whole margin over the best baseline, and Table 6's values exceed anything in the memory-size study. Table 4 disagrees with Table 1 for Kinetics‑Sounds (74.32/10.91 vs 73.79/10.37 at the adopted 500) and N24News (81.55/10.50 vs 81.97/10.32 at the adopted 1000), and Table 5 disagrees for both AVE (79.43) and N24News (81.36). Since no table states a different protocol, the headline numbers cannot be reconstructed from the paper.
  3. A claim is contradicted by the paper’s own table. L1167 asserts that DriftMem “consistently maintains the best performance,” but Table 8 shows iCaRL ahead on AVE CAA (74.82 vs 74.75). This case is merely an illustrative example rather than the sole shortcoming: similar overstatements or table–text mismatches appear elsewhere, where absolute wording is used despite marginal gains or individual baseline wins.
Questions for authors

refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zgN2: Thank you very much for your review. Our response is as follows.

We thank Reviewer zgN2 for the valuable time and constructive feedback. We provide a point-to-point response as follows. Should there be any further questions or suggestions on our submission, please let us know. We greatly value and appreciate each comment because they can help us clarify and improve our manuscript tremendously.

W1. Memory-budget fairness

We would like to have a further clarification on our work.

  • For AVE and Kinetics-Sound datasets, we follow the setting in AV-CIL[1] (340 exemplar size for AVE, 500 exemplar size for Kinetics-Sound). In our experiments, we used the same buffer size for all replay-based baselines (i.e., iCaRL, SS-IL, AFC, CMR-MFN, and AV-CIL) on each of the four datasets. We will include this illustration in the revised manuscript.

  • We acknowledge that MMAL is an exemplar-free method and thus it follows a different memory paradigm from replay-based approaches. We include MMAL as a complementary baseline to provide a broader comparison across different MCL paradigms, rather than as a directly matched replay-based competitor. What's more, our primary evaluation focuses on comparisons with existing exemplar-based methods under the same memory budget. Experimental results in the paper show that DriftMem outperforms these methods across all four datasets, demonstrating its effectiveness under a controlled replay setting.

  • Regarding the ablation of DDM, the fact that this ablated variant performs worse than MMAL does not undermine the contribution of DDM. Instead, it demonstrates the importance of DDM in the overall framework.

[1] Audio-Visual Class-Incremental Learning, ICCV2023.

W2. Experimental inconsistency

Thank you for carefully identifying these inconsistencies. We would like to clarify that these discrepancies arise from the stochastic training procedure rather than differences in the experimental configurations. Specifically, our method involves Bernoulli sampling in AGR, which determines the activation of gradient recovery. Different stochastic sampling outcomes can lead to variations in the number of gradient recovery operations, thereby affecting the optimization trajectory and resulting in fluctuations in the final performance. Such effect is particularly noticeable on AVE, this is because AVE has a smaller evaluation dataset size (less than 400 samples). To address this concern and ensure a more reliable evaluation, we have conducted additional experiments using five independent random seeds and report the averaged results.

DatasetsCAA ↑\uparrowFFM ↓\downarrowFAA ↑\uparrow
AVE80.36 ± 1.375.41 ± 1.8470.56 ± 1.23
Kinetics-Sounds73.98 ± 0.7510.59 ± 1.2365.27 ± 0.98
N24News81.98 ± 0.9710.73 ± 0.8068.71 ± 1.06
UPMC-Food10195.23 ± 0.853.08 ± 0.4193.78 ± 0.56

We will include the results in the revision.

W3. Claim contradiction

Thank you for pointing out this issue. We agree that the statement at L1167 is overly strong. We will re-examine the manuscript for similar overstatements and revise the relevant descriptions accordingly, following your suggestions to improve the clarity of the paper.

Dear Reviewer zgN2,

We sincerely appreciate your thoughtful feedback and valuable suggestions.

We hope that our responses and clarifications have addressed your concerns. As the discussion period is approaching its deadline, we would greatly appreciate your further feedback.

If our responses have changed your assessment, we would be grateful if you could consider revisiting your rating accordingly. We are also very happy to address any remaining concerns.

Thank you for your time and consideration.

Best regards,

Submission9485 Authors

I remain skeptical about your response to W1. After removing the DDM module from DriftMem, its performance is inferior to that of the exemplar-free methods; therefore, your claim highlights the importance of DDM in the overall framework. However, why not incorporate a module similar to DDM into the exemplar-free methods to enable a fairer comparison?

We are grateful to you for your further feedback. We agree that removing DDM demonstrates its importance within DriftMem, rather than establishing an advantage over exemplar-free methods. Our DDM aims to address representation drift by using historical exemplars to learn associations between multimodal feature keys and historical residuals. Although DDM cannot be incorporated into exemplar-free methods, a similar idea to DDM without replay exemplars has been previously examined in LDC (Gomez-Villa et al., 2024). We follow this comment to compare LwF, LwF + LDC, and DriftMem under the same protocol. The results are shown as follows.

MethodAVEKinetics-SoundN24NewsUPMC-Food101
LwF66.85/32.8561.34/27.6278.53/16.5384.87/22.74
LwF + LDC69.08/32.5863.88/19.8377.25/13.6485.65/20.44
DriftMem80.36/5.4173.98/10.5981.98/10.7394.23/3.08

The results show that LwF+LDC has improvements over LwF. Our DriftMem outperforms LwF+LDC mainly due to our DDM uses historical exemplars to learn associations between multimodal feature keys and historical residuals, which enables more fine-grained compensation of fused representations. Besides, DriftMem's synergistic integration, i.e., Differentiable Delta Memory (DDM) for memory construction, Attention-guided Gradient Recovery (AGR) for cross-modal interaction stabilization, and Class-structured Feature Calibration (CFC) for memory consolidation, effectively addresses the challenges in MCL.

[1] Gomez-Villa et al., Exemplar-free continual representation learning via learnable drift compensation. In European Conference on Computer Vision, pp. 473-490, 2024.

Reviewer_ZNn7OpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on representation drift in multimodal class-incremental learning. As a model learns new audio-visual or image-text classes, its feature representations and cross-modal interactions can change, causing it to forget previously learned knowledge. Instead of trying to prevent these changes, the authors propose DriftMem, which learns to compensate for representation drift. The method consists of three components. Differentiable Delta Memory (DDM) works like a continuously updated correction system that learns how old representations have shifted and predicts corrections based on the current input. Attention-guided Gradient Recovery (AGR) aims to keep cross-modal attention stable during training by adjusting gradients according to changes in attention patterns. Class-structured Feature Calibration (CFC) makes semantic keys more distinguishable across classes, helping reduce interference when retrieving and updating memory. The authors evaluate DriftMem on four multimodal class-incremental learning datasets, covering both audio-visual and image-text tasks, and compare it with ten existing methods. The results show that DriftMem achieves better classification accuracy across all four datasets and generally reduces forgetting. The paper also includes ablation studies and additional analyses to examine the effectiveness of the proposed components.

Critical strengths

One strength is that the paper considers several important issues in multimodal continual learning, rather than just focusing on feature drift. It also takes into account changes in cross-modal interactions and interference between different classes during memory updates. The three proposed modules address these problems from different perspectives and complement each other, making the overall method more complete and convincing. Another strength is that the experiments are relatively comprehensive. The authors evaluate their method on four datasets covering both audio-visual and image-text tasks, and compare it with ten baseline methods. They also conduct ablation studies and additional experiments to analyze the proposed components and their effectiveness under different settings. These results provide reasonable support for the proposed method.

Critical weaknesses

Although the paper includes many experiments, some reported results are inconsistent. For example, Table 1 and Table 6 report different results on AVE under what appear to be the same default loss weights, without explaining the difference. Also, DriftMem uses replay samples, while some of the compared methods are exemplar-free, which raises questions about the fairness of these comparisons. These issues make it harder to judge how reliable the reported improvements are.

Questions for authors

Table 1 and Table 6 report different results on AVE, even though they seem to use the same default settings. Could the authors explain what causes this difference? Were different random seeds, data splits, or training settings used?

4 comments
AuthorsOpenReview ↗

Response to reviewer ZNn7: Thank you very much for your review. Our response is as follows.

We are very grateful to Reviewer ZNn7 for your appreciation and insightful comments. We would like to further clarify our experimental settings and results here.

Part 1. Experimental settings & results

Regarding the inconsistencies in the experimental results, we would like to clarify that they mainly arise from the stochastic nature of our training procedure rather than differences in experimental configurations. Specifically, our method involves Bernoulli sampling in Attention-guided Gradient Recovery (AGR) to determine whether gradient recovery is activated. Different sampling can lead to variations in the number of gradient recovery operations. To address this concern and ensure a more reliable evaluation, we have conducted additional experiments using five independent random seeds and report the averaged results.

DatasetsCAA ↑\uparrowFFM ↓\downarrowFAA ↑\uparrow
AVE80.36 ± 1.375.41 ± 1.8470.56 ± 1.23
Kinetics-Sounds73.98 ± 0.7510.59 ± 1.2365.27 ± 0.98
N24News81.98 ± 0.9710.73 ± 0.8068.71 ± 1.06
UPMC-Food10195.23 ± 0.853.08 ± 0.4193.78 ± 0.56

Part 2. Baseline fairness

Regarding the fairness of the comparisons, we acknowledge that some compared methods are exemplar-free, whereas DriftMem relies on replay samples. To clarify this distinction, we categorize the compared methods into replay-based and exemplar-free approaches. For all replay-based baselines (i.e., iCaRL, SS-IL, AFC, CMR-MFN, and AV-CIL), we use an identical buffer size across all four datasets to ensure a fair comparison under the same memory budget. Exemplar-free methods are included as a separate category to provide a broader comparison with approaches that do not rely on exemplar storage. We will clarify these experimental settings and distinctions in the revised manuscript.

Reviewer_ZNn7OpenReview ↗

Comment to the response

Thanks for the clarification and additional results. Your response addresses most of my previous concerns about the experimental consistency and fairness of the comparisons.

However, using the same replay buffer size does not necessarily show that DDM is better than simpler feature compensation methods. For example, would a simple MLP trained to predict historical feature residuals achieve similar results under the same replay budget? I think this comparison is important to better understand the actual contribution of DDM.

Overall, I appreciate the authors' response, but I would prefer to keep my original score unchanged.

We are grateful to you for your further feedback and your suggestion.

We follow this comment and evaluate a two-layer MLP. This variant DriftMem(w/o DDM) + MLP is trained with MSE loss to predict historical feature residuals under the same replay budget as DriftMem(DDM). The evaluation results (CAA/FFM) are shown as follows.

MethodAVEN24News
DriftMem(w/o DDM) + MLP74.66/12.8777.51/15.34
DriftMem(DDM)80.36/5.4181.98/10.73

From the results, we can see that this variant is inferior to DriftMem on the datasets AVE and N24News. The results indeed demonstrate the strength of DDM. The results also indicate that simple feature compensation methods such as MLP still suffer from representation drift. Our DDM can learn associations between semantic keys and drift residuals for representation compensation through differentiable delta-rule updates.

We hope that our responses and clarifications have addressed your concerns. We are also very happy to address any remaining concerns. Thank you!

Thank you but I can't change anymoer. It's not worth 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.