- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DriftMem, a method for multimodal class-incremental learning that compensates for representation drift rather than explicitly constraining it. DriftMem consists of three components: Differentiable Delta Memory (DDM), a linear associative memory that predicts drift residuals from semantic keys; Attention-guided Gradient Recovery (AGR), which introduces restorative gradients based on changes in cross-modal attention; and Class-structured Feature Calibration (CFC), which encourages a more discriminative structure in the semantic key space. The authors report consistent improvements over ten baselines across four datasets.
Critical strengths
-
Addressing representation drift through compensation rather than restriction is a reasonable approach to balancing stability and plasticity.
-
The method introduces relatively modest additional memory overhead and does not require task identity at inference.
-
The analysis of γ_mem (Fig. 4) gives an intuitive picture of how compensation strength affects old- and new-class accuracy.
Critical weaknesses
1. Technical novelty of DDM. DDM appears to be a class-conditional linear corrector fitted on exemplars. Since CFC strongly clusters the keys by class, it would be helpful to clarify how much Mk actually differs from a per-class mean drift offset. An analysis supporting the claimed sample-dependent compensation would make the contribution more convincing. In addition, LDC (Gomez-Villa et al., 2024), which learns a projector for drift compensation, seems conceptually very close. It is cited but not compared against experimentally.
2. Source of the improvement. Because DDM is trained to output zero for new-class samples, part of the gain may come from correcting the classifier bias between old and new classes rather than from recovering drift. The trend in Fig. 4, where increasing γ_mem raises old-class accuracy and lowers new-class accuracy, also seems consistent with this interpretation. An experiment that disentangles these two effects would be valuable.
3. Multimodal specificity. DDM operates on the fused feature and is modality-agnostic, so the multimodal-specific part of the method appears to be mainly AGR. However, Table 2 shows that adding AGR to DDM degrades performance (FAA 64.19 → 58.57). I would appreciate a deeper analysis of this result. Given the chosen values of δ_att and τ_att, the Bernoulli gate may also be active almost all the time. Similarly, CFC brings a large gain even without DDM, so it is unclear whether its benefit is specific to reducing memory interference.
Questions for authors
Q1 Could you provide comparisons with LDC, or with simple feature distillation on exemplars?
Q2 Could you disentangle the effect of DDM from that of bias correction methods such as BiC or WA?
Q3 What does the distribution of π^t look like during training? Also, how do you explain the performance drop when AGR is combined with DDM?
Q4 Is the cosine similarity in Fig. 3 measured on replay samples or on held-out test samples?
Q5 Does DDM bring similar gains when applied to unimodal class-incremental learning?
3 comments
Q2/W2. Source of the improvement.
Thank you for this comment. We agree that the disentanglement of classifier bias correction and drift recovery is crucial. In fact, as shown in Fig. 4, increasing tends to improve old-class accuracy while new-class accuracy degrades. However, our objective is not to eliminate this trade-off, but rather to improve the attainable performance under such a trade-off through representation-level drift compensation.
We now conducted additional experiments by incorporating two representative bias correction methods, Weight Alignment (WA) and BiC, into the baseline without DDM. Each cell is reported in the format of CAA/FFM.
| Methods | AVE | Kinetics-Sounds | N24News | UMPC-Food101 |
|---|---|---|---|---|
| Base(w/o DDM) | 73.71/16.11 | 64.85/20.88 | 77.17/15.95 | 91.07/7.98 |
| Base+WA | 76.20/13.75 | 66.25/19.97 | 77.08/18.34 | 91.67/7.25 |
| Base+BiC | 75.51/15.08 | 65.57/16.94 | 76.76/15.84 | 90.94/5.56 |
| Base+DDM | 80.36/5.41 | 73.98/10.59 | 81.98/10.73 | 95.23/3.08 |
Although WA and BiC provide certain improvements over the baseline, their gains are relatively limited. Our DDM achieves substantially better performance across all four datasets.
We additionally measure the L2 distance between the current feature representations and their corresponding historical representations, before and after applying DDM.
| AVE Tasks (Max) | Before | After | Reduction$\uparrow$ |
|---|---|---|---|
| 1 | 28.85 | 17.91 | 37.92% |
| 2 | 25.99 | 16.67 | 35.86% |
| 3 | 20.26 | 14.58 | 28.04% |
| AVE Tasks (Mean) | Before | After | Reduction$\uparrow$ |
|---|---|---|---|
| 1 | 24.96 | 9.29 | 62.78% |
| 2 | 18.34 | 11.72 | 36.10% |
| 3 | 15.78 | 10.48 | 33.59% |
| N24News Tasks (Max) | Before | After | Reduction$\uparrow$ |
|---|---|---|---|
| 1 | 33.24 | 24.63 | 25.90% |
| 2 | 25.14 | 22.45 | 10.70% |
| 3 | 23.09 | 19.98 | 13.47% |
| N24News Tasks (Mean) | Before | After | Reduction$\uparrow$ |
|---|---|---|---|
| 1 | 14.84 | 7.12 | 52.02% |
| 2 | 11.44 | 8.77 | 23.34% |
| 3 | 10.61 | 8.10 | 23.66% |
The results show that DDM substantially reduces the feature distance, indicating that the proposed memory-based compensation effectively recovers the historical representations.
Q3/W3. Multi-modal Specificity
We are grateful to you for your constructive suggestions.
As illustrated in line430, AGR alone may increase the overlap between the semantic representations of old and new classes, thereby reducing their discriminability. Following your suggestion, we measure the Silhouette scores of semantic keys across different ablation settings.
| Method | Silhouette score |
|---|---|
| Base(only DDM) | -0.0092 |
| Base+AGR | -0.0102 |
| Base+AGR+CFC | 0.0704 |
The results shows that CFC effectively reduces memory interference, while applying AGR alone could increase memory interference.
Q4. Cosine similarity in Fig. 3
The cosine similarity in Fig. 3 is measured on test samples.
Q5. DDM for unimodal class-incremental learning
DDM also achieves comparable performance gains in unimodal settings. To further investigate this, we conduct ablation experiment using only the context modality and report the results on the AVE dataset. Each cell is reported in the format of CAA/FFM.
| Method | Silhouette score |
|---|---|
| Base model | 57.75/14.81 |
| Base model + DDM | 62.35/11.40 |
As illustrated in line 253-255, our delta memory is able to compensate modality-specific features and cross-modal aggregation, achieving a more fine-grained compensation. Thus, DDM plays a more significant role in multimodal scenarios.
Dear Reviewer CaoN,
We sincerely appreciate your thoughtful feedback and valuable suggestions.
We hope that our responses and clarifications have addressed your concerns. As the discussion period is approaching its deadline, we would greatly appreciate your further feedback.
If our responses have changed your assessment, we would be grateful if you could consider revisiting your rating accordingly. We are also very happy to address any remaining concerns.
Thank you for your time and consideration.
Best regards,
Submission9485 Authors
Resonpse to reviewer CaoN: Thank you very much for your review. Our response is as follows.
We sincerely thank Reviewer CaoN for the constructive feedback and valuable comments. We provide our point-by-point responses below. Should there be any further questions or suggestions on our submission, please let us know. We greatly value and appreciate each comment because they can help us clarify and improve our manuscript tremendously.
Q1/W1. Technical novelty of DDM
Thank you for your question and comments.
To explore sample-dependent contribution, we conduct experiments that compare DDM with class-wise compensation. For class c, we define class-wise compensation as rct=∣Ec∣1∑i∈EcMkit, and use it to compensate the original feature $ \hat{\mathbf z}^t=\mathbf z^t+\gamma_{\mathrm{mem}}\mathbf M\mathbf r^t$. We calculated the average L2 distance between compensated features and corresponding historical representations, comparing these two compensation methods before and after compensation. Below are the results of AVE after training on the last task.
The comparison results show our sample-dependent improvement.
LDC learns a feature mapping from previous feature space to current feature space, and use a Nearest Class Mean (NCM) Classifier. Different from LDC, our DriftMem provides a new compensation mechanism by learning associations between current representations and corresponding representation drift. Although LDC also employ a sample-dependent compensation, it learns the map function based on observation of current task data, unabling to model drift of cross-modal interactions.
We also include the comparison results with LDC here. Each cell is reported in the format of CAA/FFM.
We will include the new results in the revision.