- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the personalized federated linear stochastic approximation problem, where the coefficient matrices 's and the right-hand-side vectors 's can be different across agents. It proposes an algorithm that allows the agents to collaborate while achieving local optimal solutions. In short, the agents use a mixture of local gradient and the global aggregated gradient to update their own parameter estimation. When heterogeneity is small enough, the algorithm is proven to achieve both linear speedup and personalization.
Critical strengths
Strengths: The paper is well-written and easy to follow. The technical part looks solid and clear to me. Personalization in federated reinforcement learning is under-researched; therefore, the exploration in this regard is commendable.
Critical weaknesses
Weaknesses: 1. The linear speedup relies on low heterogeneity, which is not convincing. If the heterogeneity is low, do we really need personalization? In [1], they already showed in their Theorem 2 that FedTD can converge to a neighbourhood of each agent's optimal solution, where the size of the neighbourhood is determined by heterogeneity. Therefore, I believe the message given by this paper is not adequate. Could the authors compare with [1]?
-
The paper misses several papers on personalized federated reinforcement learning, such as [2]. In the Personalized Federated Learning section, [3] is missing as well.
-
The algorithm structure that uses a mixture of local and global gradients is not novel. For example, [4] also uses weighted gradients.
[1] Han Wang, Aritra Mitra, Hamed Hassani, George J. Pappas, James Anderson, Federated TD Learning with Linear Function Approximation under Environmental Heterogeneity, TMLR 2024
[2] Guojun Xiong, Shufan Wang, Daniel Jiang, Jian Li, On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations, ICLR 2025
[3] Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai, Exploiting Shared Representations for Personalized Federated Learning, ICML 2021
[4] El Mahdi Chayti, Sai Praneeth Karimireddy, Sebastian U. Stich, Nicolas Flammarion, Martin Jaggi, Linear Speedup in Personalized Collaborative Learning, arXiv
Questions for authors
See Weaknesses.
3 comments
- [W3] The algorithm structure that uses a mixture of local and global gradients is not novel. For example, [4] also uses weighted gradients.
We agree with the reviewer that mixing local and global gradients is not, by itself, a novel mechanism. We have revised the manuscript to acknowledge related approaches and to make clear that our main contribution lies in the theoretical guarantees rather than the mixing mechanism.
In particular, [4] studies personalized collaborative learning through a different formulation: the objective is to optimize a designated target task using information from auxiliary tasks. The authors analyze weighted gradient averaging and a bias-correction approach inspired by control variates.
By contrast, our objective is to solve all agents' personalized learning problems simultaneously, rather than using auxiliary agents primarily to improve the solution of one designated task. We establish non-asymptotic convergence guarantees toward every agent's individual solution and characterize when collaboration yields a linear speedup in per-agent sample complexity.
We believe this distinction is particularly relevant in federated learning, where every participating agent has its own learning objective. We have revised the manuscript to better acknowledge prior work and emphasize these theoretical contributions.
We thank the reviewer once again for their constructive feedback and remain available to address any further questions or concerns.
[1] Han Wang, Aritra Mitra, Hamed Hassani, George J. Pappas, James Anderson, Federated TD Learning with Linear Function Approximation under Environmental Heterogeneity, TMLR 2024
[2] Guojun Xiong, Shufan Wang, Daniel Jiang, Jian Li, On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations, ICLR 2025
[3] Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai, Exploiting Shared Representations for Personalized Federated Learning, ICML 2021
[4] El Mahdi Chayti, Sai Praneeth Karimireddy, Sebastian U. Stich, Nicolas Flammarion, Martin Jaggi, Linear Speedup in Personalized Collaborative Learning, arXiv
Dear Reviewer,
As the discussion period draws to a close, we would like to ask whether any of your concerns remain unresolved following our response. We would be grateful if you could let us know and consider updating your score if you feel that your concerns have been adequately addressed.
Regarding the linear speedup, we have included an additional experiment demonstrating that linear speedup is still achieved in mildly heterogeneous settings, and that our algorithm remains effective even in highly heterogeneous settings. The results are included in the revised manuscript, and we refer you to our response to Reviewer yDw6 for further details.
Thank you again for your time and valuable feedback.
Kind regards,
The Authors
We thank the reviewer for their careful reading and constructive feedback. We appreciate the opportunity to clarify the motivation for our approach and its relationship to existing work. We address each concern below.
We thank the reviewer for raising this important point. We agree that when heterogeneity is sufficiently small and the desired accuracy is not too stringent, learning a shared solution may be sufficient. However, even mild heterogeneity can make personalization necessary when aiming for high precision.
Indeed, as established in Theorem 2 of [1], FedTD converges to a neighborhood of each agent's individual solution, whose size depends on the level of heterogeneity. Consequently, even under mild heterogeneity, this residual bias prevents FedTD from guaranteeing arbitrarily high accuracy with respect to each agent's personalized target.
In contrast, PF-LSA preserves convergence toward each agent's individual solution, regardless of heterogeneity. Moreover, when the agents' learning problems are sufficiently similar and the target accuracy is not too stringent, our analysis establishes a linear speedup in per-agent sample complexity. Thus, our objective is precisely to combine the benefits of collaboration with the ability to attain arbitrarily high accuracy on personalized problems.
Importantly, the level of heterogeneity may not be known beforehand, particularly in online learning, where agents collect observations sequentially rather than having access to complete datasets. Our method therefore does not require determining in advance whether collaboration is beneficial: the same mixing coefficient β=1/N is used across all heterogeneity regimes.
We thank the reviewer for bringing these references to our attention. We have included and discussed both [2] and [3] in the revised manuscript.
Both works exploit a shared representation across agents, together with agent-specific parameters, to obtain the benefits of collaboration. Our approach addresses a complementary setting. Rather than assuming that the agents' learning problems admit a common representation, we allow each agent to solve its own linear fixed-point equation with potentially heterogeneous stochastic operators. We establish conditions under which collaboration yields a linear sample-complexity speedup without assuming a shared representation or requiring prior knowledge of inter-agent similarities.
We have clarified these distinctions in the related-work section.