Dynamic Model Buffer for Continual Federated Deep Reinforcement Learning

Federated Deep Reinforcement Learning (FDRL) enables decentralized DRL training across distributed clients while preserving data privacy, yet it faces significant challenges in maintaining generalization across sequential tasks. Existing Continual FDRL (CFDRL) approaches primarily rely on regularization or replay mechanisms, both of which are constrained by fixed network capacity and thus inevitably suffer from catastrophic forgetting as tasks accumulate. Although expansion-based strategies have been explored in the Continual Learning (CL) community, their substantial hardware overhead renders them impractical for resource-constrained FDRL clients. To address this challenge, we propose a generic expansion-based CFDRL framework that can be seamlessly integrated into existing FDRL methods to equip them with enhanced CL capabilities. Our approach introduces a server-side model buffer that stores models from completed tasks, alongside a model selection strategy that identifies and reuses the most relevant prior model for each incoming task. This design effectively alleviates catastrophic forgetting by retaining the trained models from previous tasks, while improving knowledge utilization efficiency through model selection, without imposing a significant computational burden on clients. Extensive experimental results demonstrate that our method consistently enhances ten state-of-the-art (SOTA) FDRL methods and surpasses five SOTA CDRL and five CFDRL methods across multiple metrics.