- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a dynamic model buffer framework for continual federated deep reinforcement learning (CFDRL) to address catastrophic forgetting and improve knowledge reuse across sequential tasks. The method maintains previously trained models in a server-side buffer and selects the most suitable model for each new task based on temporal-difference (TD) errors, avoiding network expansion on resource-constrained clients. The framework can be integrated into existing federated reinforcement learning algorithms without modifying their client-side architectures. Experiments across robotic navigation, robotic manipulation, and MuJoCo benchmarks demonstrate substantial performance improvements over several existing methods.
Critical strengths
Strength 1: The server-side model buffer is a straightforward way to preserve previously learned policies without increasing the size of individual client models. The framework can be incorporated into different FDRL algorithms with minimal architectural modification, making it potentially useful for resource-constrained federated learning environments. This practical flexibility is a meaningful contribution.
Strength 2: The authors evaluate the method with ten FDRL algorithms and compare it against five CDRL and five CFDRL approaches. The experiments cover different environments, task sequences, and client configurations.
Critical weaknesses
Weakness 1: My biggest concern is the limited methodological novelty and questionable assumptions about task identification. The central idea of storing previously trained models and selecting one for reuse is relatively straightforward and closely related to existing policy libraries, checkpoint selection, and continual learning approaches. The main technical component is TD-error-based model selection, but the paper does not convincingly establish that this criterion reliably identifies the most suitable policy for a new task. More importantly, the framework assumes that the server can interact directly with the current task environment to collect transitions for model selection. This assumption may not hold in practical federated reinforcement learning, where environments and interaction data are distributed across private clients. It also raises questions about the claimed privacy-preserving and deployment advantages. These limitations substantially weaken both the novelty and general applicability of the proposed framework.
Weakness 2: Although the paper presents a theoretical analysis, the performance bounds rely on assumptions about policy similarity and bounded deviations that are not clearly connected to the proposed buffer-selection mechanism. In particular, the analysis does not establish why minimizing TD errors should consistently lead to better policy returns.
Questions for authors
-
How does the server obtain interaction samples from the current task in a practical federated setting where environments and data are private to individual clients? Could model selection be performed using locally computed statistics without transferring raw interaction data?
-
What theoretical or empirical evidence supports using TD error as a reliable measure of policy suitability? Could a model have a low TD error but still produce poor actions or low cumulative reward? How does the proposed selection compare with directly evaluating candidate policies or using simpler similarity-based criteria?
-
How do server memory, model-selection time, and communication overhead scale with the number of tasks? Can the authors clarify which assumptions in the theoretical analysis specifically guarantee an advantage for TD-error-based selection over alternative initialization strategies?