acceptodds
Withdrawn from ICLR 2027

ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated continual learning must integrate new tasks over time without losing earlier-task knowledge. Most existing methods attach an anti-forgetting mechanism to the client-trained, server-aggregated loop of federated learning, which holds back new learning to preserve earlier knowledge and burdens resource-constrained clients. We propose ReSCENE, which structurally mitigates catastrophic forgetting by having each client upload a small condensed surrogate of its local data while the server keeps the surrogates of past tasks and trains the global model on them together with the current task surrogates. For efficient server memory, we introduce temporal herding, which selects the more recent surrogates from the pool accumulated over a task into a compressed buffer. Our study provides a theoretical analysis showing that this buffer can represent the original task data more closely than full accumulation of all surrogates. Across CIFAR-10, CIFAR-100, and TinyImageNet, ReSCENE achieves the strongest accuracy over seven baselines, by up to points of average accuracy, while requiring as little as of the client computation and up to less upload than the model-update baselines. ReSCENE further demonstrates its effectiveness when scaled to larger client populations and larger models while remaining efficient, which makes it a practical method for federated continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, Federated continual learning, Federated class-incremental learning, Catastrophic forgetting, Dataset condensation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q54ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a novel federated continual learning approach, namely ReSCENE where the catastrophic forgetting problem is handled in the server-side. Each client uploads a small condensed surrogate of its local data while the server keeps the surrogate of the past data and induces a global model. In addition, a temporal herding method is proposed selecting recent surrogates from the pool into a compressed buffer. The proposed approach shows some improvement over recent works.

Critical strengths
  1. the idea of surrogate to compress local data is interesting.
  2. FCL is a challenging problem.
  3. numerical results are promising.
Critical weaknesses
  1. why is the proposed method evaluated with CIFAR10? this dataset is too easy for FCIL evaluations. I suggest replacing it with more challenging dataset such as CUB, etc.

  2. resnet18 is used as a backbone. Such backbone network is too small for current standard of continual learning. I also wonder how such approach performs under more complex backbone networks.

  3. No experiment is done for varied global communication rounds. This should be done for all methods across all datasets.

  4. Benchmark methods are old. It is necessary to compare the performance of the proposed methods with recent state of the art methods published in 2025 onward.

Questions for authors

please kindly address the four major points abovementioned.

Reviewer_8GFzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a surrogate-based federated continual learning framework in which clients communicate condensed representations of their local data instead of model updates. The server jointly trains on current and previously retained surrogates to mitigate catastrophic forgetting. The paper also introduces temporal herding to maintain a bounded surrogate buffer.

Critical strengths

The approach is interesting and offers reductions in client-side computation and communication overhead. The experimental results show promising improvements over existing methods. The authors also provide a detailed resource analysis, including server-side computation, and investigate the effect of reducing server training epochs.

Critical weaknesses

The three main concerns are as follows:

  1. A classical rule in FL has been not to share client size data. Although clients do not transmit raw data, this still raises potential privacy concerns. While a differential privacy analysis is mentioned, there is no analysis. It is important to clarify whether the guarantees apply to the actual algorithmic configuration. An evaluation against reconstruction or inference attacks would strengthen the privacy assessment.
  2. Regarding temporal herding: Theorem 1 establishes an advantage over full accumulation under a conditioninvolving stale bias and approximation errors. However, the authors argue that this condition becomes increasingly satisfied because older surrogates become more stale, which is not necessarily true as the global model approaches convergence. Furthermore,it is not clear how improved class-mean approximation translates to better classification performance or reduced catastrophic forgetting. The authors should clarify the practical validity of the theorem’s assumptions and their connection to the learning objective. Assumption 2 imposes a coherent directional structure on the surrogate representation errors and assumes that the magnitude of the stale bias decreases monotonically across communication rounds. This is very restrictive for FL non iid settings.
  3. End-to-end training efficiency remains unclear: The method transfers most of the training burden to the server, and the framework assumes synchronous communication, making server-side training part of every round’s latency. Reporting wall-clock training time or time-to-target accuracy would provide a more complete assessment of the practical efficiency gains.

Some desired results:

  1. A more interesting theoretical insight would be on catastrophic forgetting. My intuition says that if they are able to show that catastrophic forgetting is less than or equal to surrogate approx error + some optimization error due to SGD + distributional shift error, it would be very insightful . Seemingly, the distributional shift error does not come in anywhere in the analysis, which makes me worry about the over all work.
Questions for authors

Please rebut the weaknesses.

Reviewer_9DEROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses catastrophic forgetting in federated continual learning. It moves replay storage and model training to the server to reduce the computing and storage burden on clients. At each communication round, clients upload small sets of synthetic images representing their current data, and the server trains on current data surrogates together with saved surrogates from previous tasks. To save storage, the paper use temporal herding, which selects a smaller replay buffer whose average features match those of recently generated samples.

Critical strengths

The paper investigate interesting problem that move the training and storage to server to save computing and storage at client side. The method is simple to implement. Also, the writing is clean and easy to follow.

Critical weaknesses
  1. The contribution of the proposed works need to be positioned more carefully, as some components have already been on related works. For example, Surrogate upload by distribution matching is FedDM [1], Buffer selection by herding toward a class mean is the exemplar selection of iCaRL [2], [3] combines standard class-balanced sampling with logit adjustment. Also, FedCIL [4] consolidates the global model on the server with generated data, and MFCL [5] and LANDER [6] train data-free generators on the server for FCIL. Condensed data as continual-learning memory was already evaluated in the DM paper [7] that ReSCENE builds on. ECoral [8] condenses exemplars for federated class-incremental learning and is the closest prior work, yet it is mentioned only once in Appendix B and never compared. Can the authors discuss and compare with [5, 6, 8] and clarify what the contribution is beyond "keep the uploaded surrogates and replay them"?

  2. The data held by the server is reported in table 4b is larger than the original dataset. Specifically, table 4b shows 98,087 synthetic images on CIFAR-100 (50,000 samples on training set) and 194,355 on TinyImageNet (100,000 on training set). Temporal herding may removes only about 20 to 40% of the pool (as B/|P| around 0.6 to 0.8), but it still a heavy data size. The claim "a small condensed surrogate" in the abstract should be considered carefully.

  3. The paper say IPC is fixed "regardless of how much local data the class has". This raise a question that when a client holds fewer than 10 images of a class, it still produces 10 surrogates per round from them, which is expansion rather than condensation. The authors need to clarify this detail.

  4. The privacy experiment in Table 13 is not a fair comparison, and it may indicate leakage rather than robustness. Specifically, the proposed method keeps 55.13 AA at σ=5, which may be due to 25 noisy steps on pixel gradients cannot remove a real image used as the starting point. Also, the baselines are switched to GroupNorm, which lowers their σ=0 accuracy, while ReSCENE keeps BatchNorm. The authors should add empirical privacy evaluation, such as nearest-neighbour similarity between surrogates and client images, membership inference, or visualizations, since dataset condensation has been shown not to provide privacy for free [9].

  5. The motivation contradicts the results on plasticity. The introduction criticizes prior methods because they "hold back new learning". However, table 6 shows that ReSCENE has the lowest final-task accuracy among the strong methods.

  6. The paper claims that existing methods never optimize classes of different tasks against each other. However, replay methods such as iCaRL [2], GLFC, Re-Fed and FedCBDR do train old-class exemplars together with new classes, so they do learn cross-task boundaries. This need to be considered carefully.

  7. The efficiency claims does not clearly show the server cost and report best cases. Table 12 gives total computation only for CIFAR-10 at task 2, the cheapest point on the cheapest dataset, and even there ReSCENE costs about 4.2× FedAvg. Communication is reported the same way, as "Up to 179× smaller upload" holds only on CIFAR-10 but maybe on Imagenet, it may be worse. The authors should add more experiments to clarify this point, and add this into limitation section if needed.

  8. The authors should explain how ReSCENE works on asynchronous and client-specific task sequences, and very task sequence (500 to 100 tasks for example) and evaluate it under the heterogeneous task-sequence settings. The author may check [10] for references.

  9. The paper claim reduce the computing and resource at the client side. It should be much better if the authors can provide the test under real resource-limited hardware.

[1] FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning - CVPR 2023

[2] iCaRL: Incremental Classifier and Representation Learning - CVPR 2017

[3] Long-tail Learning via Logit Adjustment - ICLR 2021

[4] Better Generative Replay for Continual Federated Learning - ICLR 2023

[5] A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks - NeurIPS 2023

[6] Text-Enhanced Data-free Approach for Federated Class-Incremental Learning - CVPR 2024

[7] Dataset Condensation with Distribution Matching - WACV 2023

[8] Exemplar-Condensed Federated Class-Incremental Learning - https://arxiv.org/abs/2412.18926

[9] No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy" - arXiv 2022

[10] HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning - NeurIPS 2026

Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.