acceptodds
Withdrawn from ICLR 2027

CARES: Coverage-Aware Replay Exemplar Selection for Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Replay-based Continual Learning mitigates catastrophic forgetting by keeping a buffer of past samples, making buffer composition critical under a fixed memory budget. Recent coreset-based replay methods improve buffer quality using model-derived gradients, losses, or auxiliary optimization, but incur substantial computational overhead as memory is repeatedly updated. We propose CARES, a continual model-optimization-free approach for buffer construction that instead exploits the geometry of a frozen pretrained feature space using class-aware, density-weighted coverage. We evaluate CARES across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings. On Tiny-ImageNet, CARES is 2.3–8.8 faster than existing coreset-based selectors. Although CARES achieves this speedup using no training signal from the continual learner, it ranks among the state of the art across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings, outperforming the strongest baseline by up to 7.58 percentage points under class imbalance while also attaining higher average accuracy on Tiny-ImageNet. Code will be released upon publication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Continual Learning, Coreset Selection, Online Learning, Class-Incremental

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YMtTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes CARES, a coreset selection method for continual learning that does not require model optimization. The experimental results demonstrate the efficiency and effectiveness of the proposed method.

Critical strengths
  1. The proposed coreset selection method avoids model optimization and bilevel optimization, making it computationally efficient.
  2. The writing is generally clear, and the equations and algorithm are well presented.
  3. The experiments are comprehensive and cover different evaluation protocols.
Critical weaknesses
  1. As acknowledged in line 186, the coreset selection method used in this work was originally proposed in [1]. Therefore, the selection method itself is not novel. While it is well established that selecting representative samples for replay can improve continual learning performance, this work does not sufficiently explain why the method from [1] is particularly suitable for continual learning.
  2. Beyond the empirical results, this work provides limited explanation of why CARES outperforms model-optimization-based coreset selection methods. Model-optimization-based methods typically have explicit objectives that guide sample selection. This work should further explain the relationship between the coverage objective and coreset effectiveness in continual learning, and why this objective can lead to better performance.
  3. The proposed method relies on an external feature encoder, specifically a frozen ImageNet-pretrained Inception-V3 model. Its effectiveness may therefore depend on the quality of the pretrained representations. It is unclear whether CARES would remain effective on domains that differ substantially from the pretraining distribution.
  4. The efficiency analysis does not sufficiently account for the computational cost associated with the pretrained encoder.

[1]. SubZeroCore: A Submodular Approach with Zero Training for Coreset Selection

Questions for authors

In line 176, the authors raise the question of how to evaluate the value of a sample when its contribution to the trained model is realized in the future rather than at the time of selection. How does the proposed method address this issue? In particular, how does the coverage-based selection objective account for the future utility of selected samples in continual learning?

Reviewer_hqNfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CARES, a continual model-optimization free approach for replay-buffer construction for online CL. It adapts training-free geometric coreset selection to a continually evolving replay buffer of fixed capacity. Evaluation conducted on CIFAR-10, CIFAR-100 and TInyImagenet shows faster memory updates compared to existing methods.

Critical strengths

Coreset selection is an important problem in CL, especially given the large-size of datasets and streaming setting. This is of interest to the community. The use of density-aware coreset selection yields faster memory updates, which would also be interesting for future references.

Critical weaknesses

Originality: CARES builds on top of SubZeroCore for online continual learning. It is however challenging to see the contribution of the method. Sections 3.2 and 3.3 are essentially standard rehearsed-based CL with the incorporation of SubZeroCore. The application seems straightforward which makes the positioning of the method challenging. Class-aware allocation is also a common technique in CL, thus is not really a contribution of the paper.

Significance: While online CL and the used experimental evaluation would have been significant a few years ago, its current significance is unclear to me. Even the most recent baseline considered by the paper is from 2023 with CIFAR-10, CIFAR-100 and TinyImagenet datasets. My concern thus stems from the limited experimental evaluation. Even on the limited evaluation, the gains of the proposed method remain limited in all tables. It even increases forgetting in many scenarios. Further the ablation in Table 6 shows marginal gain across components further making it hard to dissect the significance of the method.

Clarity: The paper clarity could also be significantly improved. The paper does not cite tables, the algorithm and tables are positioned out of context. Subsections in section 3 also lack the dissection of the method from existing work, which could be improved in writing.

Questions for authors

Please refer to the weaknesses above.

Reviewer_zbhYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CARES, a replay selection method for continual learning based on frozen pretrained features. The method combines a reservoir-sampled candidate pool with the existing replay buffer, allocates memory approximately equally across classes, and selects examples within each class using SubZeroCore’s density-weighted facility-location objective. Selection uses no gradients or losses from the continual learner and requires no auxiliary training.

Critical strengths

-Decoupling replay selection from learner optimization addresses a practical problem. Buffer selection is repeated throughout training, so avoiding additional backward passes or auxiliary optimization can be useful even without improving accuracy in every setting. -Several reported gains are substantial.

Critical weaknesses

-The contribution relative to the closest prior methods is not sufficiently established. Section 3.3 explicitly adopts the density score and weighted facility-location objective from SubZeroCore. However, SubZeroCore already performs class-wise selection: this is stated in its Section 2.1 and in the definition of its objective in Section 3.2. The Global SubZeroCore variant in Table 6 therefore does not represent the closest adaptation of that method. GCR also already uses a reservoir candidate pool, reconstructs memory from old and new examples, and assigns equal per-class budgets. The paper needs to explain what CARES adds beyond combining these mechanisms. The ablation results leave this question open.

-The update schedule itself is ambiguous. Algorithms 1 and 2 reconstruct memory after each task, whereas Section 3.2 describes a protocol-defined update stage. The relationship between tasks, incoming stream batches, and reconstruction stages is not specified for either protocol. This affects when new examples become available for replay, how often selection is performed, and the runtime comparison.

Questions for authors

-What are the concrete differences between CARES and the original class-wise SubZeroCore selector placed within the same staging, quota, and memory-update framework? -Do all variants in Table 6 share the same encoder, candidate pool, and update schedule? -What exactly triggers staging-pool reset and memory reconstruction in each protocol? -Were the baseline runtimes measured again on the reported hardware? The abstract describes a 2.3–8.8× memory-update speedup, while Figure 1 reports end-to-end runtime. Please distinguish training, feature extraction, and selection costs. The baseline times underlying these ratios also match those in BCSR’s Appendix D, Table 8; please clarify whether they were reused or independently measured

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.