acceptodds
Withdrawn from ICLR 2027

Learning to Manage Memory for Domain Incremental Learning via Adaptive Replay

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world data often evolves across domains, challenging models to adapt to new distributions while retaining previously acquired knowledge. Replay-based methods for Domain Incremental Learning (DIL) address this challenge by interleaving current data with samples from previous domains. However, as the model and domains evolve, some stored memories may become stale, redundant, or less useful for replay. The challenge is therefore not simply to preserve memory, but to keep it useful. Inspired by neuroscience studies that view forgetting as an active and functional process of memory regulation, we propose Reinforcement learning-based Adaptive Memory (RAM), a framework that treats memory as an adaptive resource rather than a static collection of raw samples. We further present a Mutual-Information Sequential Adaptation Bound that relates cross-domain error bounds to the mutual information between semantic-related representations and domain identity. This analysis motivates separating semantic- and domain-related representations, whose drift provides structured signals for assessing memory throughout training. Guided by drift-based priors and performance feedback across all seen domains, RAM learns a lightweight policy to actively retain, repair, or remove memory entries under a fixed budget. Experiments on four DIL benchmarks demonstrate that RAM consistently improves accuracy and reduces forgetting across different backbones, outperforming the evaluated baselines and highlighting the value of active memory regulation for continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
domain-incremental learning, feature disentanglement, memory management

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_q7KBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAM, a replay-based Domain Incremental Learning (DIL) framework that treats memory management as a reinforcement learning problem. Instead of maintaining a static replay buffer, RAM stores semantic and domain representations for each memory sample and learns a policy that decides whether to retain, repair, or remove samples. The policy is guided by representation drift and validation feedback. The paper also derives a mutual-information-based adaptation bound to motivate disentangling semantic and domain representations. Experimental results on DomainNet, Camelyon17, iWildCam, and Rotated-MNIST show improvements over replay baselines such as ER, DER++, X-DER, and AdaER.

Critical strengths
  • The motivation is clear and intuitive. The central observation that stored replay samples can become stale under evolving domains is reasonable and underexplored relative to sample selection literature.
  • Strong empirical results. The reported gains over ER, DER++, X-DER, AdaER and several prompt-based approaches are non-trivial across multiple datasets and architectures.
  • The proposed method could orchestrate multiple integration into one coherent system.
Critical weaknesses
  • Novelty. The repair action sounds novel at first glance. However, inspecting Eq. (13), repair merely updates stored features using an EMA: z^{old} = (1−\beta)z^{old} + \beta z^{new}. Overall, the memory management systems uses standard operations for the flow of experience replay.
  • The reinforcement learning controller's role is actually questionable in the pipeline as the proposed heuristic rule already partitions the drift plane. Thus, the policy may simply learn the handcrafted rule.
  • The paper presents a "Mutual-Information Sequential Adaptation Bound". This theoretical proof mostly plays a motivational role for the proposed algorithm rather than providing new guarantees.
  • The theoretical arguments built upon the Wasserstein framework depend on an approximately constant covariance assumption throughout memory updates, limiting the validity of the conclusions in realistic settings. Thus, the empirical gains could not be well-justified.
  • The ablations remove components individually, but there is no clean decomposition demonstrating each algorithm related to reinforcement learning, disentanglement, and repair operation in memory.
Questions for authors
  • Regarding the assumption that covariance remains approximately stable during memory updates. How sensitive is the method to violations of this assumption, and do the theoretical conclusions still hold under realistic feature distributions?
  • Since replay is performed using the original raw samples, while the "repair" operation only updates stored feature representations via EMA, what is the precise mechanism through which repair improves replay quality?
  • Have the authors analyzed cases where the learned policy disagrees with the drift-based heuristic? If so, are these disagreements responsible for the observed gains over heuristic policies?
Reviewer_rxYoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that in replay-based domain-incremental learning, the memory buffer should be actively maintained rather than stored as a static archive. The proposed method consists of (i) a two-branch encoder that splits each input into a semantic representation used for prediction and a domain representation, and (ii) a method that measures normalized semantic and domain drifts between the stored and the current representations.

Critical strengths
  1. The paper focuses on how stored instances should be maintained over time, not only which ones to store. Viewing memory as a resource that must be actively nurtured may not be particularly innovative, but it is a useful perspective and, in my opinion, still represents an important research challenge.

  2. The paper is pretty clearly written and well illustrated, with pseudocode and a notation table.

Critical weaknesses
  1. The main weakness is that REPAIR only moves the stored representations toward the current encoder (Eq. 13). These stored vectors are never used for training or inference. It enter only the drift computation. REPAIR is therefore equivalent to RETAIN plus a reset of the item’s drift reference, which only changes the policy’s future decisions. There ara many approaches in this as drift compensation of stored features or prototypes see: Yu, L., Twardowski, B., Liu, X., Herranz, L., Wang, K., Cheng, Y., Jui, S., and van de Weijer, J. (2020). Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020). arXiv:2004.00440. Gomez-Villa, A., Goswami, D., Wang, K., Bagdanov, A. D., Twardowski, B., and van de Weijer, J. (2024). Exemplar-free continual representation learning via learnable drift compensation. In European Conference on Computer Vision (ECCV 2024), pp. 473–490. Springer.

or replacing outdated buffer entries with samples from the current distribution; see, e.g., Ashrafee, A. et al., (2026). Holistic continual learning under concept drift with adaptive memory realignment. Transactions on Machine Learning Research (TMLR), 01/2026. (Ashrafee et al., TMLR 2026).

  1. In Eq. 14, the validation set data is accumulated, which means it may exceed the buffer size. How, then, is the data in the buffer managed? Therefore, the reference methods do not have access to such data, so the comparison is not reliable or fair. The validation data should be included in the memory limit. The domain loss and adversarial loss (Eq. 5–6) are expected values with respect to D_t, and Algorithm 1 computes them solely on the current mini-batch. All these samples share the domain label t. So, the domain classifier sees a single class, and the gradient inversion term cannot induce domain invariance. Was data from the buffer taken into account during the loss calculation? If so, this should be clearly indicated. It is also unclear how drift is calculated in the “no separation” variant.

  2. What is the sense of the following theoretical analysis (in the context of the propsed method) (i) Lemma 3.1 concerns a fixed representation, not a sequentially updated encoder, and it does not involve the memory at all. (ii) D.2–D.4 – it focuses on stored features that the learner never uses (what was mentioned earlier. (iii) Lemma D.16 that does not concern learned layers.

  3. The baselines are not credibly chosen. At least the following algorithms should be added MIR and ASER which were cited in the paper but did not used.  Also, the following methods should be considered: GSS, Rainbow Memory, UDIL, methods employing learned replay policies, Concept-drift-aware replay (Korycki, Ł. and Krawczyk, B. Class-incremental experience replay for continual learning under concept drift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2021, CLVision), pp. 3649–3658. arXiv:2104.11861)and memory realignment triggered by a statistical test on predictive uncertainty (Ashrafee et al., Holistic continual learning under concept drift with adaptive memory realignment. Transactions on Machine Learning Research (TMLR), 01/2026.).

  4. High-drift samples are also likely to be atypical or long-tail ones, which, following, need to be memorized for good generalization (Feldman, V. (2020). Does learning require memorization? A short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC 2020). arXiv:1906.05271.). This matters for iWildCam (182 imbalanced species), where removal may systematically target rare classes.

  5. The experimental protocol may disadvantage the baselines.

  6. Related work is incomplete, therefore many important baselines are missing.

Questions for authors

See weaknesses. Additional questions related to comparative study: How were the ViT and ResNet backbones pretrained, and how were the baseline hyperparameters set? Why is SGD with learning rate 0.1 used for ViT and the prompt-based methods? Do the results hold across several domain orders?

Reviewer_TrREOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAM, an adaptive replay framework for domain-incremental learning. It separates semantic and domain-related representations, measures their drift, and uses an RL-based policy to retain, repair, or remove memory entries. The policy is guided by drift-based rules and validation feedback. The authors also provide a mutual-information adaptation bound and a Wasserstein-based analysis to support the method. Experiments on four benchmarks (Rotated-MNIST, DomainNet, Camelyon17, and iWildCam) show improved accuracy and reduced forgetting over the compared baselines. Ablation studies and comparisons with heuristic policies further evaluate the proposed components.

Critical strengths

RAM consistently improves accuracy and reduces forgetting over the evaluated baselines across four benchmarks. The ablation studies and comparisons with heuristic policies further support the effectiveness of the proposed approach.

Critical weaknesses
  1. The paper presents active memory maintenance as a problem distinct from replay selection. However, AdaER [2] already updates replay buffers, while X-DER [3] revises stored predictions. Although the authors argue that existing approaches rely on heuristics or lack interpretability, it remains unclear what research gap these limitations leave beyond the proposed implementation. The related work [1], which also studies adaptive replay under a different data-access setting, is not discussed.

  2. RAM favors removing samples with high semantic and domain drift, but such samples could also be important examples at risk of forgetting. Appendix D.3 analyzes when removing outliers can reduce distribution discrepancy, but does not establish that high-drift samples satisfy these conditions or have low replay utility. The ablation results demonstrate the effectiveness of the overall method, but do not directly validate this assumption.

[1] 2024 – CVPR Workshops – Adaptive Memory Replay for Continual Learning

[2] 2024 – Neurocomputing – AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning

[3] 2023 – TPAMI – Class-Incremental Continual Learning into the eXtended DER-Verse

Questions for authors
  1. What specific limitations of AdaER and X-DER does RAM address that their existing memory-update mechanisms cannot?

  2. Does high semantic and domain drift imply the mean-outlier condition in Appendix D.3? If not, what justifies favoring removal for these samples?

  3. Is there direct experimental evidence that high-drift samples have lower replay utility, beyond the overall performance improvements of RAM?

Reviewer_Fj24OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reinforcement learning-based Adaptive Memory (RAM), a framework for Domain Incremental Learning (DIL) that dynamically manages replay memory to mitigate catastrophic forgetting and improve adaptation to evolving domains. The method introduces a Mutual-Information Sequential Adaptation Bound to motivate the disentanglement of semantic- and domain-related representations. Based on their representation drift, RAM employs a lightweight reinforcement learning policy to selectively retain, repair, or remove stored samples under a fixed memory budget. The policy is guided by drift-based priors and performance feedback across previously encountered domains. Extensive experiments on four DIL benchmarks demonstrate improved accuracy and reduced forgetting across different backbones, highlighting the effectiveness of adaptive memory management.

Critical strengths

The paper addresses an important problem in domain-incremental learning by introducing adaptive memory management to mitigate catastrophic forgetting. Its key strength lies in integrating representation drift with a learned policy for dynamic memory regulation. The experiments cover multiple benchmarks and backbones, with consistent improvements over existing methods and additional ablation studies supporting the overall effectiveness of the framework.

Critical weaknesses
  1. The REPAIR operation updates stored historical representations, but the updated features are not directly utilized in the replay loss. This raises questions about how the proposed repair mechanism effectively contributes to improving model performance.

  2. The reinforcement learning policy employs a global reward shared across multiple memory operations, making it difficult to distinguish the individual contributions of different decisions. This may introduce ambiguity in policy optimization and affect training stability.

  3. The memory management strategy primarily relies on semantic and domain representation drift, which may not adequately reflect sample importance, redundancy, or the actual contribution of historical samples to knowledge retention.

  4. The reinforcement learning policy requires validation feedback from all previously encountered domains, potentially limiting its applicability to realistic DIL scenarios. Moreover, this additional access to historical data raises concerns about the fairness of comparisons with methods operating under stricter data access constraints.

  5. The RL reward combines validation improvement, action diversity, and drift-guided alignment, but the experiments only vary their weights within limited nonzero ranges. The authors should provide component-wise ablations that remove each reward term to establish its individual contribution.

  6. Although RAM and the baselines share the same memory budget, their actual buffer sizes may differ, and some insertion schedules cannot saturate the buffer even without deletion. Comparisons under matched memory sizes and varying budgets are needed to ensure fair evaluation.

Questions for authors

Please see the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.