acceptodds
Under review as a conference paper at ICLR 2027

Single-Timescale Personalized Average-Reward Federated TD Learning Across Heterogeneous Environments

Abstract

We study personalized federated TD learning, in which a collection of agents interact with different environments and jointly learn their respective value functions. Inspired by the recent success of personalized federated learning (PFL), we focus on the setting where there exists a shared linear representation and the agents' optimal weights collectively lie in an unknown linear subspace. Under canonical single-timescale updates with average-reward objectives and Markovian sampling, we propose a natural personalized federated TD algorithm, named \bf pFedAR-TD, in which agents jointly estimate a common subspace while updating agent-specific weights and mean rewards. We show that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of “misaligned” signals arising from environmental heterogeneity, and hence achieve linear speedup. Compared with the discounted-reward setting, the combination of personalization and average-reward objectives significantly complicates the convergence analysis. In particular, the additional need to learn local mean rewards results in three tightly coupled dynamics rather than two, calling for a careful balance between fine-grained characterization of error perturbations and the overall tractability of the system-level analysis. Our analysis exploits geometric properties induced by key algorithmic components such as projection, innovation filtration, and QR decomposition. A large subspace error early in training can amplify perturbations in the coupled iterates and jeopardize convergence. To address this, we further introduce an initialization phase and establish a maximal-deviation bound to keep the coupled iterates in a region where the convergence argument applies. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem.

Primary area:
reinforcement learning
Keywords:
federated reinforcement learning, reinforcement learning, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M5wJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies personalized federated TD learning in heterogeneous environments and proposes pFedAR-TD, a single-timescale algorithm for the average-reward setting. The method learns a shared low-dimensional representation along with agent-specific parameters and reward estimates. The authors prove a convergence rate of O~(r/(TK))\widetilde{O}(r/(TK)) and evaluate the method on synthetic MDPs and several Gym environments.

Critical strengths

I think the paper studies an interesting problem, especially since handling environmental heterogeneity while still benefiting from collaboration is challenging in federated RL. The theoretical analysis is quite detailed. In particular, I find the joint Lyapunov analysis of local parameter errors and subspace estimation errors interesting. I also appreciate that the authors provide an initialization procedure and a maximal-deviation analysis instead of simply assuming a good initialization.

Critical weaknesses

1. My main concern is the novelty of the proposed method. It seems to combine several existing ideas, such as shared representations, personalized heads, projected updates, and QR decomposition. Shared representation learning has already been studied by Collins et al. (2021), and Xiong et al. (2025) considered a closely related setting in personalized federated TD learning. I am not entirely convinced that the algorithm itself is sufficiently different from existing approaches.

2. I am also not fully convinced by the linear speedup claim. The main theoretical result requires L=KL=K, so increasing the number of agents also increases the number of samples collected by each agent. It is therefore hard to tell how much of the improvement actually comes from collaboration.

3. I have some concerns about Assumption 1. In the standard tabular average-reward setting, the TD matrix has a zero eigenvalue corresponding to the constant-value direction, so the negative-definiteness assumption does not seem to hold without additional restrictions.

4. I find the experimental evaluation somewhat limited. The synthetic experiments seem to use agent-specific reward functions, while the theoretical setup assumes a shared reward function. Also, I would have liked to see a more direct comparison with Xiong et al. (2025) and more random seeds in the Gym experiments.

[1] Xiong, G., Wang, S., Jiang, D., & Li, J. (2025). On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations. International Conference on Learning Representations (ICLR).

[2] Collins, L., Hassani, H., Mokhtari, A., & Shakkottai, S. (2021). Exploiting Shared Representations for Personalized Federated Learning. International Conference on Machine Learning (ICML), pp. 2089–2099.

Questions for authors

1. Could the authors clarify what the main novelty is compared with Xiong et al. (2025)? I would be particularly interested in understanding which parts of the proposed method or analysis go beyond existing techniques.

2. Would the linear speedup still hold if LL were fixed while increasing KK? I think this would help clarify the actual benefit of collaboration.

3. Could the authors explain how Assumption 1 is satisfied in the tabular average-reward setting?

4. Why do the synthetic experiments use agent-specific rewards when the theoretical formulation assumes a common reward function?

Reviewer_6qZzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies personalized averaged TD-learning--which the authors call pFedAR-TD. The agents share a common reward function, whereas the transition matrix is specified differently. This is represented as an underlying subspace between the agents, which is expressed as n×rn\times r matrix. The suggested algorithm is shown to achieve O(r/(TK))O(r/(TK)) convergence rate for subspace dimension rr and number of agents KK. This covers different scenario from the known result for discounted reward setting and also improves the dependency on TT and KK.

Critical strengths
  1. The paper tackles an underexplored regime by analyzing personalized TD learning. While shared subspace learning has been extensively studied in the federated learning community, adapting and analyzing this framework within the theoretical RL literature seems well-motivated.

  2. The authors manage to derive a solid theoretical result by consolidating different technique. In particular, the coupled three estimation processes—average rewards, the shared subspace, and agent-specific heads are successfully analyzed. Although the individual analytical tools (such as QR decomposition and principal angle analysis) are established in the prior literature, the overall integration seems to be complicated and authors manage to derive a finite-time bound result.

  3. Even though this a theory paper, the authors try to demonstrate the result by experiments.

Critical weaknesses
  1. My major concern is on Theorem 1 of how the bounds scale to the parameters. The bound is only stated in terms of TT, KK, and rr. Moreover, it only holds when TT and KK large enough, which is not discussed in the main text. From Theorem 2, this already seems to scale at the order of 1/λ81/\lambda^8, and it should be clearly discussed what other factors can be problematic. Following this discussion, the comparison with the liteature on this factor should be propely discussed.

  2. The implementations seems quite difficult to scale to large-scale problems. First, the algorithm requires a projection step to a convex ball, and second, a QR decomposition step is required. These are all interesting theoretically but it adds additional computational complexity on the implementation.

  3. The writing of the paper can be improved. For example, in the main manuscript, Lemma 5.1-5.4 are given without sufficient explanation. Moreover, Lemma 5.4 is not mentioned after its statement. The organization and presentation of technical results should be carefully re-organized for clarity of the paper.

Questions for authors

The authors assume "personalized learning" setting. Therefore, it seems more natural to assume different reward functions rather than different transition kernels between the agents. There are a quite of works on such "different reward function between the agents" in the multi-agent filed. Can the authors comment more on this setting?

Reviewer_S2pyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies personalized reinforcement learning for average-reward agents with heterogeneous transition kernels.

Critical strengths
  • The paper is easy to follow with explanations for the theoretical advances
  • It tackles the important challenge of personalization under environment heterogeneity
  • The proposed algorithm and analysis are intuitive and reasonable
  • The proposed method achieves state-of-the-art sample complexity, strictly better than that of the previous method
Critical weaknesses
  • The work inherits the structural limitations of the "shared representation and common low-dimensional subspace for personalized weights" framework; specifically,
    • Agents' value functions use the same linear approximation with a known representation;
    • Agents' personalized weights lie in the same low-dimensional subspace with a known dimension rr;
    • I regard the knowledge of the dimension a major limitation of the framework.
  • Only environment (transition kernel) heterogeneity is considered but no reward heterogeneity. Moreover, the work is constrained to evaluation but not control.
  • Related work
    • Is there any related work on Personalized FedRL on discounted return achieving a similar rate?
    • A recent work on Personalized FedRL, Zhang and Azizan (ICLR'26), might be relevant. It also studies TD under environment heterogeneity with shared representation, achieving a similar rate, but without assuming further structure.
  • The exposition of the work can be improved, especially regarding explaining and discussing theoretical consequences; some examples:
    • The abstract says "We show that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of 'misaligned' signals arising from environmental heterogeneity", but this seemingly important insight is not discussed in the main text
    • When introducing Eq. (8), better explain that a "novel filter" is introduced; otherwise it's not clear why there is a new term and what "innovation filtration" means. In addition, the transition "Thus" before Eqs. (7) and (8) is unclear.
    • The remark "it introduces 'not so small' perturbation that interplays with the errors of other dynamics nontrivially that can be harmful to the convergence and potential speedup" is unclear
    • The notation of the initialization phase is unclear. Before Section 5, it uses T′T'. Then before Lemma 5.5, TexitT_{\mathrm{exit}} appears without explanation. Additionally, should the definition of TexitT_{\mathrm{exit}} be max⁡\max, otherwise isn't it always 0? At the same location, t~0\tilde{t}_{0} and t~b\tilde{t}_{b} are introduced, and I don't know why they need to be different. On a similar note, Lemma 5.6 is difficult to parse.
    • The transition between individual errors and the Lyapunov analysis can be expanded and strengthened. Line 392 is interesting and may be worth more discussion. However, reading it, it's unclear why it works until it's explained later that an initialization phase is needed to control the second term. The clause "while ∥mt∥2≤λ/4\|m_{t}\|_{2}\le \lambda /4" is unclear here, as it needs to be established rather than automatically holding.
    • I would suggest revising all the explanations and discussions around the theoretical results to provide clear pointers for navigation. Otherwise, I am often confused about what a sentence refers to, whether it's a needed condition, property, or result to be established.

Reference

Zhang and Azizan. Personalized Collaborative Learning with Affinity-Based Variance Reduction. ICLR, 2026.

Questions for authors

See "Critical Weaknesses". Additionally, how does the complexity of the initialization phase affect the overall complexity? It's unclear from the current coarse statements.

1 comment

I thank the authors for their detailed rebuttal and manuscript revision. My concerns are largely resolved, and I maintain my positive evaluation of the work and will engage with other reviewers and AC in further discussion. I believe the revision and incorporation of the rebuttal for all the reviews into the manuscript will greatly improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.