- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies personalized federated TD learning in heterogeneous environments and proposes pFedAR-TD, a single-timescale algorithm for the average-reward setting. The method learns a shared low-dimensional representation along with agent-specific parameters and reward estimates. The authors prove a convergence rate of and evaluate the method on synthetic MDPs and several Gym environments.
Critical strengths
I think the paper studies an interesting problem, especially since handling environmental heterogeneity while still benefiting from collaboration is challenging in federated RL. The theoretical analysis is quite detailed. In particular, I find the joint Lyapunov analysis of local parameter errors and subspace estimation errors interesting. I also appreciate that the authors provide an initialization procedure and a maximal-deviation analysis instead of simply assuming a good initialization.
Critical weaknesses
1. My main concern is the novelty of the proposed method. It seems to combine several existing ideas, such as shared representations, personalized heads, projected updates, and QR decomposition. Shared representation learning has already been studied by Collins et al. (2021), and Xiong et al. (2025) considered a closely related setting in personalized federated TD learning. I am not entirely convinced that the algorithm itself is sufficiently different from existing approaches.
2. I am also not fully convinced by the linear speedup claim. The main theoretical result requires , so increasing the number of agents also increases the number of samples collected by each agent. It is therefore hard to tell how much of the improvement actually comes from collaboration.
3. I have some concerns about Assumption 1. In the standard tabular average-reward setting, the TD matrix has a zero eigenvalue corresponding to the constant-value direction, so the negative-definiteness assumption does not seem to hold without additional restrictions.
4. I find the experimental evaluation somewhat limited. The synthetic experiments seem to use agent-specific reward functions, while the theoretical setup assumes a shared reward function. Also, I would have liked to see a more direct comparison with Xiong et al. (2025) and more random seeds in the Gym experiments.
[1] Xiong, G., Wang, S., Jiang, D., & Li, J. (2025). On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations. International Conference on Learning Representations (ICLR).
[2] Collins, L., Hassani, H., Mokhtari, A., & Shakkottai, S. (2021). Exploiting Shared Representations for Personalized Federated Learning. International Conference on Machine Learning (ICML), pp. 2089–2099.
Questions for authors
1. Could the authors clarify what the main novelty is compared with Xiong et al. (2025)? I would be particularly interested in understanding which parts of the proposed method or analysis go beyond existing techniques.
2. Would the linear speedup still hold if were fixed while increasing ? I think this would help clarify the actual benefit of collaboration.
3. Could the authors explain how Assumption 1 is satisfied in the tabular average-reward setting?
4. Why do the synthetic experiments use agent-specific rewards when the theoretical formulation assumes a common reward function?
I thank the authors for their detailed rebuttal and manuscript revision. My concerns are largely resolved, and I maintain my positive evaluation of the work and will engage with other reviewers and AC in further discussion. I believe the revision and incorporation of the rebuttal for all the reviews into the manuscript will greatly improve the paper.