acceptodds
Under review as a conference paper at ICLR 2027

What to Represent and What to Remember: Complementary Replay for Federated Class-Incremental Learning

Abstract

Exemplar replay-based federated class-incremental learning aims to preserve previously acquired knowledge by replaying selected samples while learning new classes. Existing methods independently score samples in the full feature space and replay those with the highest importance. However, they overlook complementarity within the limited replay set, resulting in redundant selections and insufficient coverage of the historical representation space accumulated across clients and tasks. To address these limitations, this study proposes a federated REpresentation Coverage-Aware repLay Learning framework, termed RECALL, which constructs a compact and complementary replay memory through global critical-feature identification and residual-based exemplar selection. Specifically, it consists of three modules. The global critical feature selection module selects feature coordinates for reconstructing class-balanced current and previous model representations, reducing feature redundancy. The complementary exemplar selection module combines local residual preselection with class-wise global coordination to reduce cross-client redundancy and improve replay coverage. The coverage-weighted feature distillation module constrains representation drift along the selected critical directions, preserving historical knowledge during incremental learning. We also bound full-class reconstruction error for fixed representations in the selected coordinates, accounting for local screening loss and client quotas. Experiments across three benchmark datasets show that RECALL consistently outperforms strong baselines by 1%–8%, validating the effectiveness of complementary and coverage-aware replay.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated learning, Federated class-incremental learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JS7VOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method, RECALL, for federated class-incremental learning. It tries to address the issue of complementarity in the replay set. The proposed approach is based on the idea of feature selection. Specifically, the critical-feature identification and the residual-based exemplar selection are designed. RECALL is evaluated with three datasets with varying clients and splits. The proposed approach shows some advantage.

Critical strengths
  1. The idea of FCIL with feature selection to my knowledge is new;
  2. presentation of this paper is good;
  3. experimental results are positive.
Critical weaknesses
  1. why is the proposed method evaluated with CIFAR10? this dataset is too easy for FCIL evaluations. I suggest replacing it with more challenging dataset such as CUB, etc.

  2. resnet18 is used as a backbone. Such backbone network is too small for current standard of continual learning. I also wonder how such approach performs under more complex backbone networks.

  3. it seems to me that all algorithms were re-run under the same computational environment. Although it is fine, the hyper-parameter selection of other methods needs to be explained. How do you guarantee fair experimental conditions for all consolidated algorithms?

  4. Upper bound should be provided in the main table.

Questions for authors

please address the 4 issues aforementioned in the critical weaknesses.

Reviewer_1TdnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RECALL to address redundancy among replay samples in Federated Class-Incremental Learning. RECALL combines Global Critical Feature Selection, Complementary Exemplar Selection and Coverage-Weighted Feature Distillation to improve the coverage and retention of historical representations.

Critical strengths

1.The idea of linking feature selection with exemplar selection is reasonably motivated.

Critical weaknesses

1.The method description is difficult to follow. The main text introduces a large number of symbols and formulas, but lacks a framework diagram or specific examples. It is difficult to intuitively understand how the three modules are connected and what operations the client and server perform respectively.

2.Ablation does not fully validate the core design. Removing CWFD entirely does not establish the benefit of coverage weights over standard feature distillation. The evaluation of GCFS also lacks comparisons with simpler selection methods that retain the same number of features.

Questions for authors

1.GCFS selects features one by one, requiring multiple sequential aggregations. How many sequential communication rounds are required to construct the replay memory once? Does the runtime reported in Table 4 include feature extraction, communication and waiting for synchronization?

2.How big are the screening loss, κ and quota-blocked residual energy in Theorem 1 in actual experiments? Are the theoretical bounds tight enough to explain the observed improvements in coverage?

3.Could the authors report old-class accuracy or average forgetting to clarify whether the gains in overall accuracy are accompanied by reduced forgetting? The current setup uses a per-task memory budget. Does RECALL retain its advantage when the total memory budget is fixed?

Reviewer_WJFcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECALL, a three-component framework for federated class-incremental learning that combines global critical-feature selection, class-wise cross-client complementary exemplar selection, and coverage-weighted feature distillation. The method is coherent and reports consistent gains, but it needs clearer novelty positioning and broader empirical validation.

Critical strengths

The paper addresses an important problem in federated class-incremental learning. RECALL provides a coherent framework that coordinates feature selection, complementary exemplar replay, and feature distillation while keeping raw data local. The experiments show consistent improvements across multiple datasets and settings.

Critical weaknesses
  1. The novelty appears to lie mainly in integrating existing ideas, while comparisons with representative diversity-aware or coreset-based replay methods are missing.

  2. The empirical validation is limited to ResNet-18 and relatively moderate non-IID settings, and key choices such as client-class weighting and memory allocation are insufficiently justified.

Questions for authors
  1. The novelty claim regarding exemplar complementarity should be positioned more carefully. Recent studies have already considered diversity or marginal contribution in replay selection. Krutsylo (2024) explicitly studies inter-batch diversity and removes redundant neighboring samples; Tong et al. (2025) evaluate samples by the performance gain obtained when they are added to a coreset; and Yanowsky and Weinshall (2026) use complementary supervised and self-supervised embeddings for replay selection. Therefore, RECALL’s novelty appears to lie in its specific combination rather than in introducing complementarity itself. The authors should clarify this distinction and compare with recent diversity-aware selectors under the same memory budget.

[1] A. Krutsylo, “The Inter-batch Diversity of Samples in Experience Replay for Continual Learning,” AAAI, 2024.

[2] R. Tong et al., “Coreset Selection via Reducible Loss in Continual Learning,” ICLR, 2025.

[3] D. Yanowsky and D. Weinshall, “Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers,” arXiv:2604.08336, 2026.

  1. The experimental protocol does not specify how the test data are partitioned or whether evaluation is centralized or client-specific. Moreover, the authors should justify the use of 1/\sqrt{n_{i,c}}, which gives each client-class approximately equal total weight regardless of its sample size. It is unclear whether this assumption is appropriate under heterogeneous data distributions, particularly when some local classes contain very few samples. An ablation with alternative weighting strategies would strengthen the justification.

  2. RECALL combines approximately class-balanced client-side quotas with residual-based selection, but their interaction is not examined. Under long-tailed data, this allocation may assign memory without considering class complexity, while residual maximization may also favor outliers.

  3. The paper lacks an overall framework diagram. Please add one to clearly illustrate the roles and relationships of the three proposed modules.

  4. The experiments only consider Dirichlet parameters beta=0.5 and 1.0, representing relatively mild-to-moderate heterogeneity. Please include more severe non-IID settings, e.g., beta=0.1 and 0.01, to better evaluate the robustness of RECALL under highly heterogeneous client distributions.

  5. Most individual components of RECALL, such as diversity-aware replay, coreset-based exemplar selection, class-balanced memory, and feature distillation, have been explored previously. Its novelty therefore appears to lie mainly in their integration. The authors should clarify the specific technical advances beyond a straightforward combination of existing techniques.

  6. All experiments use only ResNet-18. Please evaluate RECALL with at least one additional backbone to demonstrate its architectural generality.

  7. Preserving sample diversity does not necessarily imply a direct correlation with performance improvements; the authors need to specify the conditions under which this paper applies.

  8. The article states that previous approaches resulted in a large number of duplicate or redundant samples, but there are no experiments to support this claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.