Communication-Efficient Federated Natural Policy Gradient with Provable Convergence Properties

This paper studies communication-efficient federated reinforcement learning for closed-loop control of dynamical systems. In our setup, multiple agents learn a common policy through a central server without sharing raw trajectory data. Standard federated natural policy gradient (FedNPG) requires each agent to transmit both a policy gradient and a Fisher information matrix (FIM), creating a substantial communication burden as the policy dimension grows. We propose FedNPG-ED, which reduces this burden by compressing each local FIM through eigendecomposition (ED). Each agent transmits only $q_k$ dominant eigenpairs and a scalar parameter $p_k$ that approximates the discarded eigenvalues. The server reconstructs the local curvature approximations, aggregates them using sample-based weights, and updates the shared policy. In the linear quadratic regulator (LQR) setting, our analysis characterizes linear convergence to a neighbourhood of the global optimum and conditions for maintaining closed-loop stability throughout learning. Our analysis for the LQR setup separates the roles of the two design parameters $q_k$ and $p_k$. The contraction factor for convergence and the residual error depend on the step size and $p_k$ only through their ratio, so that $p_k$ is chosen as small as the step-size condition permits, whereas $q_k$ fixes the per-round payload. Simulations on linear and nonlinear systems with linear and neural actors demonstrate the method's practical performance. With unequal local sample sizes, FedNPG-ED closely matches uncompressed FedNPG while transmitting substantially less data and outperforms an ADMM-based FedNPG baseline at comparable communication cost.