Adaptive Federated Reinforcement Learning with Problem-Parameter-Agnostic Stepsizes

Federated reinforcement learning (FRL) has emerged as a promising framework for decentralized decision-making in privacy-sensitive environments. However, its practical deployment remains challenging due to extensive hyperparameter tuning requirements, particularly under non-stationary data distributions and heterogeneous environments across agents. To address this limitation, we propose a problem-parameter-free FRL framework that eliminates manual stepsize selection through adaptive momentum-based updates, while addressing environment heterogeneity via hierarchical control variates. Building on this framework, we develop two algorithms: PFedPG-VR, which integrates an adaptive scheme into variance-reduced policy gradient updates, and PFedPG-HA, which refines the approach using Hessian-aided corrections. Through rigorous theoretical analysis, we prove that both algorithms achieve state-of-the-art convergence rates, with a sample complexity of $\mathcal{O}({ϵ}^{-3})$ and a communication complexity of $\mathcal{O}({ϵ}^{-2})$. In addition, they enjoy linear speedups with respect to the number of agents and local update steps at each global round. Notably, our methods eliminate the reliance on environment heterogeneity bounds required in previous FRL approaches, significantly broadening their applicability. Extensive experiments on benchmark FRL tasks further demonstrate the superior performance and robustness of our methods compared to existing baselines.