acceptodds
Withdrawn from ICLR 2027

Dynamic Model Buffer for Continual Federated Deep Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated Deep Reinforcement Learning (FDRL) enables decentralized DRL training across distributed clients while preserving data privacy, yet it faces significant challenges in maintaining generalization across sequential tasks. Existing Continual FDRL (CFDRL) approaches primarily rely on regularization or replay mechanisms, both of which are constrained by fixed network capacity and thus inevitably suffer from catastrophic forgetting as tasks accumulate. Although expansion-based strategies have been explored in the Continual Learning (CL) community, their substantial hardware overhead renders them impractical for resource-constrained FDRL clients. To address this challenge, we propose a generic expansion-based CFDRL framework that can be seamlessly integrated into existing FDRL methods to equip them with enhanced CL capabilities. Our approach introduces a server-side model buffer that stores models from completed tasks, alongside a model selection strategy that identifies and reuses the most relevant prior model for each incoming task. This design effectively alleviates catastrophic forgetting by retaining the trained models from previous tasks, while improving knowledge utilization efficiency through model selection, without imposing a significant computational burden on clients. Extensive experimental results demonstrate that our method consistently enhances ten state-of-the-art (SOTA) FDRL methods and surpasses five SOTA CDRL and five CFDRL methods across multiple metrics.

Primary area:
reinforcement learning
Keywords:
Continual Federated Deep Reinforcement Learning, Knowledge memorization, Knowledge forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9UTdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dynamic model buffer framework for continual federated deep reinforcement learning (CFDRL) to address catastrophic forgetting and improve knowledge reuse across sequential tasks. The method maintains previously trained models in a server-side buffer and selects the most suitable model for each new task based on temporal-difference (TD) errors, avoiding network expansion on resource-constrained clients. The framework can be integrated into existing federated reinforcement learning algorithms without modifying their client-side architectures. Experiments across robotic navigation, robotic manipulation, and MuJoCo benchmarks demonstrate substantial performance improvements over several existing methods.

Critical strengths

Strength 1: The server-side model buffer is a straightforward way to preserve previously learned policies without increasing the size of individual client models. The framework can be incorporated into different FDRL algorithms with minimal architectural modification, making it potentially useful for resource-constrained federated learning environments. This practical flexibility is a meaningful contribution.

Strength 2: The authors evaluate the method with ten FDRL algorithms and compare it against five CDRL and five CFDRL approaches. The experiments cover different environments, task sequences, and client configurations.

Critical weaknesses

Weakness 1: My biggest concern is the limited methodological novelty and questionable assumptions about task identification. The central idea of storing previously trained models and selecting one for reuse is relatively straightforward and closely related to existing policy libraries, checkpoint selection, and continual learning approaches. The main technical component is TD-error-based model selection, but the paper does not convincingly establish that this criterion reliably identifies the most suitable policy for a new task. More importantly, the framework assumes that the server can interact directly with the current task environment to collect transitions for model selection. This assumption may not hold in practical federated reinforcement learning, where environments and interaction data are distributed across private clients. It also raises questions about the claimed privacy-preserving and deployment advantages. These limitations substantially weaken both the novelty and general applicability of the proposed framework.

Weakness 2: Although the paper presents a theoretical analysis, the performance bounds rely on assumptions about policy similarity and bounded deviations that are not clearly connected to the proposed buffer-selection mechanism. In particular, the analysis does not establish why minimizing TD errors should consistently lead to better policy returns.

Questions for authors
  1. How does the server obtain interaction samples from the current task in a practical federated setting where environments and data are private to individual clients? Could model selection be performed using locally computed statistics without transferring raw interaction data?

  2. What theoretical or empirical evidence supports using TD error as a reliable measure of policy suitability? Could a model have a low TD error but still produce poor actions or low cumulative reward? How does the proposed selection compare with directly evaluating candidate policies or using simpler similarity-based criteria?

  3. How do server memory, model-selection time, and communication overhead scale with the number of tasks? Can the authors clarify which assumptions in the theoretical analysis specifically guarantee an advantage for TD-error-based selection over alternative initialization strategies?

Reviewer_i83fOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors aim to resolve catastrophic forgetting in the Continual Federated Deep Reinforcement Learning (CFDRL) setting. The authors point out that since in FL clients may be highly resource-constrained, it is important to not impose network-expansion overhead.

To solve the problem, the authors propose maintaining a server-side model buffer to store the aggregated global model at the end of each task. At the arrival of new task, the server executes a random policy in the environment to collect a mini-batch of 1,000 transitions, evaluates the Temporal Difference error of all stored models on this mini-batch, selects the model with the lowest error, and uses that to warm-start local training. This mechanism does not require any change in the federated learning aggregation method or modify the RL training.

The authors show the efficacy of their method on several CFDRL settings and tasks.

Critical strengths
  • The underlying problem is very critical and important. With the considerable cost of RL, it is very important to update the model for new tasks and preserve the performance on the prior ones without substantial computation burden on the clients.
  • As the authors show in their paper, the mechanism is independent of the federated aggregation or local RL algorithm and can be integrated with various desired methods.
Critical weaknesses

The main critique to this paper is lack of novelty. The proposed algorithm is independent is not specifically designed for the federated and RL setting. It is basically storing different versions of models trained on different tasks and properly selecting a base for the new one. This idea has been explored in many different settings. This paper has only proved that it also works in federated RL as well.

  • The authors frame the problem as learning a policy model that generalizes across all tasks, however the proposed method is actually a selection of models that each trained on each task.

  • There are multiple assumptions in the paper that I find strange. For example, the common scenario in FL is that the server only aggregates the models but in this paper, the server also interacts with the environment to collect the transitions. Also, the CL setting is not very clear, are the tasks isolated and we are aware of the task id or not?

Not a weakness but a suggestion:

  • The tables are crowded with several numbers very close to each other and their descriptions are very short.
  • In page 2, last paragraph, line 6, there is "?" which probably refers to a missing citation.
Questions for authors

Can you explain what Standard deviation = 0 means in the table? (such as table 2, 3, 4)

Reviewer_wGBrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose a method for Federated Deep Reinforcement Learning in continual setting, which introduces a serverside model buffer that stores models from after each completed task and a model selection strategy that identifies and reuses the most relevant prior model for each incoming task. By doing this, the work aim to avoid new update to negatively affected the knowledge learned from older tasks, and encourage knowledge sharing, hence reduces the affect of catastrophic forgetting.

Critical strengths
  1. The paper solve an interesting problem, which is well needed for reinforcement learning.
  2. The paper shows clear motivation. It identifies the lack of computing at client-side, which cannot handle expansion-based strategies from continual learning line of work, then propose the method to focus storage and computing usage on server.
Critical weaknesses
  1. The method currently includes components that are studied by the other works. For example, saving task-specific policy heads and selecting policies for unknown tasks using environment feedback, explicitly including TD-error feedback have been studied by [1]. Selecting policies from a library using observations from an unidentified task have been studied in [2]. Maintaining multiple federated models and let clients select the model with the lowest local loss have been studied in [3,4]. Can the authors provide clear, comprehensive discussion on related works, then clarify where the contribution of the proposed work stand out ?
  2. The paper base on TD error from critic network to select the proper model. This raises the question that the critic network may output good q-value, but the actor may not be good. It should help reader to understand better if the author can do a deep analysis and experiments on how and why does TD error can be informative enough to help select good model.
  3. The paper assume at server, there will be an environment that provide samples. However, details about this global environment is not provided clearly, given that there are variances of each task provided to different clients ( as stated on stated on page 6, §5.1 “Experiment Settings,” item (2) “Benchmarks” that “To construct FDRL settings, we introduce client heterogeneity by randomly perturbing each client’s environmental parameters, such as target positions and movement velocities, within a 10% range of their nominal values.”). As multiple variances of task available, what variance does the global environment support ?
  4. The paper say that on server-side, it "interact with the environment using a random policy to collect a mini-batch of transitions." This random sampling may not include task-specific transition. This raise the question if the TD error on random sampling actually allow good model selection that allow good knowledge sharing.
  5. The other problem with the global enviroment is that without clear definition, it may cause information leak across clients, which is serious problem to federated setting.
  6. The authors also need clear justification and supported experiment about how the method scale with more clients, more tasks, more heterogeneous data (different task in each client task sequences, and across clients). For example, now the method assume save the model of each task. If the number of tasks raise to 500 or 1000, it will be extremely expensive for both storage needed to save those model and the cost for model selection. The authors may check these setting in [5].

[1] Same State, Different Task: Continual Reinforcement Learning without Interference - AAAI 2022

[2] Bayesian Policy Reuse - https://arxiv.org/abs/1505.00284

[3] An Efficient Framework for Clustered Federated Learning - NeurIPS 2020

[4] Federated Learning under Distributed Concept Drift - AISTATS 2023

[5] HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning - Neurips 2026

Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.