acceptodds
Under review as a conference paper at ICLR 2027

Communication-Efficient Federated Natural Policy Gradient with Provable Convergence Properties

Abstract

This paper studies communication-efficient federated reinforcement learning for closed-loop control of dynamical systems. In our setup, multiple agents learn a common policy through a central server without sharing raw trajectory data. Standard federated natural policy gradient (FedNPG) requires each agent to transmit both a policy gradient and a Fisher information matrix (FIM), creating a substantial communication burden as the policy dimension grows. We propose FedNPG-ED, which reduces this burden by compressing each local FIM through eigendecomposition (ED). Each agent transmits only dominant eigenpairs and a scalar parameter that approximates the discarded eigenvalues. The server reconstructs the local curvature approximations, aggregates them using sample-based weights, and updates the shared policy. In the linear quadratic regulator (LQR) setting, our analysis characterizes linear convergence to a neighbourhood of the global optimum and conditions for maintaining closed-loop stability throughout learning. Our analysis for the LQR setup separates the roles of the two design parameters and . The contraction factor for convergence and the residual error depend on the step size and only through their ratio, so that is chosen as small as the step-size condition permits, whereas fixes the per-round payload. Simulations on linear and nonlinear systems with linear and neural actors demonstrate the method's practical performance. With unequal local sample sizes, FedNPG-ED closely matches uncompressed FedNPG while transmitting substantially less data and outperforms an ADMM-based FedNPG baseline at comparable communication cost.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Federated Learning, Communication-Efficient Algorithm, Multi-Agent Systems, Convergence Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_fT2oOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In federated natural policy gradient, each agent sends a large Fisher matrix to the server every round, which costs a lot of communication. This paper proposes FedNPG-ED where each agent sends only its top qq eigenvectors and pkp_k that replaces the remaining eigenvalues. The server rebuilds an approximate matrix and updates the shared policy. For LQR with a linear Gaussian policy, a perfect critic and identical agents, the paper claims linear convergence to a neighbourhood. It also claims the convergence speed depends only on ηk/pk\eta_k/p_k.

Critical strengths

For a linear Gaussian policy, the Fisher matrix splits. So only the small n×nn\times n state covariance needs compressing which provides communication saving.

Low local cost. The top eigenvectors can be computed from the score matrix. Agents never need to build or fully decompose the Fisher matrix, so the method can scale.

Critical weaknesses

In the main proof equation 105 λmin⁡(Σ^−1)\lambda_{\min}(\hat\Sigma^{-1}) is replaced by 1/pk1/p_k. But 1/pk1/p_k is the largest eigenvalue of Σ^−1\hat\Sigma^{-1}. So these main claims may not be supported.

The proof never uses the sent eigenvectors, so the theory does not explain why sending them helps.

Lemma 4 gives LL-smoothness only on S(kϵ)S(k_\epsilon), and Step 3(b) applies it between KkK_k and Kk+1K_{k+1} in (103). The stability argument in Section 4 then uses Corollary 1, which depends on that step, to conclude Kk∈S(kϵ)K_k \in S(k_\epsilon). This looks circular.

Questions for authors

In equation 105, why is λmin⁡(Σ^Kk−1)\lambda_{\min}(\hat\Sigma_{K_k}^{-1}) taken as 1/pk1/p_k and not 1/λ^11/\hat\lambda_1? Which claims about pkp_k and qq still hold after this is fixed?

Where is the error term ⟨∇J−G,Kk+1−Kk⟩\langle \nabla J - G, K_{k+1}-K_k\rangle?

Can you give an induction showing that Kk+1K_{k+1} and the segment [Kk,Kk+1][K_k, K_{k+1}] stay in S(kϵ)S(k_\epsilon)

6 comments
AuthorsOpenReview ↗

**Official Review Comment 1/6 of Submission 34118 by Reviewer fT2o**

We thank the Reviewer for the precise reading of the proof and for noting the role of the Kronecker structure and of the score-matrix computation. Each of the three weaknesses is a genuine gap, and the questions below determined the structure of the corrected analysis.

Comment 1. In the main proof equation 105 λmin⁡(Σ^−1)\lambda_{\min}(\widehat\Sigma^{-1}) is replaced by 1/pk1/p_k. But 1/pk1/p_k is the largest eigenvalue of Σ^−1\widehat\Sigma^{-1}. So these main claims may not be supported.

Response: The Reviewer is correct. By (14) and (101), λmin⁡(Σ^Kk−1)=1/Λk{\lambda_{\min}}({\widehat{\Sigma}_{K_k}}^{-1})=1/\Lambda_k with Λk=λmax⁡(Σ^Kk)\Lambda_k={\lambda_{\max}}({\widehat{\Sigma}_{K_k}}), and 1/pk1/p_k is an upper bound on λmax⁡(Σ^Kk−1){\lambda_{\max}}({\widehat{\Sigma}_{K_k}}^{-1}). The substitution made in (105) is invalid, and the claims derived from it are not supported. The corrected bound is given in Theorem 1, in which 1/Λk1/\Lambda_k appears in the first-order term and 1/pk1/p_k in the second-order term. Also see our response to Comment 4.

Change location Theorem 1 (p. 7); Appendix G.3, Steps 3(c) and 3(d) (p. 34). All changes are in blue in the revised manuscript.

AuthorsOpenReview ↗

**Official Review Comment 2/6 of Submission 34118 by Reviewer fT2o**

We thank the reviewer for this insightful observation regarding the role of the transmitted eigenvectors in our theoretical analysis.

Comment 2. The proof never uses the sent eigenvectors, so the theory does not explain why sending them helps.

Response: We agree. In the submitted proof, the compressed matrix entered only through pkp_k, so that neither the retained eigenvalues nor the eigenvectors played any role. The submitted bound is in fact the one obtained for qk=0q_k=0, where no eigenpair is sent. The revision makes the following changes.

First, the corrected bound depends on the retained eigenpairs. In Theorem 1, they enter through Λk\Lambda_k, and Remark 5 gives the exact dependence: the first-order coefficient is the alignment coefficient αk\alpha_k, which for a single agent is the eigenvector-weighted average of 1/λ^b,k1/\widehat\lambda_{b,k} and 1/pk1/p_k.

Second, we do not claim that Theorem 1 explains why sending eigenvectors helps. It is a worst-case bound based on Euclidean smoothness, and such a bound cannot favour a non-isotropic preconditioner: its certified factor is largest for κ^k=1{\widehat{\kappa}_k}=1. Proposition 1 provides the explanation. The eigenvectors determine the relative distortion γk\gamma_k of Proposition 1, which is bounded by (εf1+ρk)/pk(\varepsilon_{f_1}+\rho_k)/p_k with ρk\rho_k non-increasing in qkq_k. Once γk≤μξ/(4σˉk+1)\gamma_k\le\mu_\xi/(4\bar\sigma_{k+1}), the contraction factor is that of exact NPG and involves neither κ^k{\widehat{\kappa}_k} nor L{\mathcal{L}}. Sending eigenpairs helps because it moves the method from the regime of Theorem 1 to that of Proposition 1.

Third, the distortion bound γk\gamma_k, which was stated in Appendix G.5 without connection to the convergence result, is now stated in Remark 2 next to Proposition 1, and the alignment identity (119), written for the exact gradient, gives the first-order coefficient of the corrected bound in Remark 5. The concentration bound of Step 2(a) of the submitted proof, which was not used in Step 3, is now used to bound γk\gamma_k in Proposition 1.

We also state the limitation. Proposition 1 is a sufficient condition. For aggressive compression, such as qk=1q_k=1 in Figures 1 and 2 (Section 5), it need not hold; the closeness to uncompressed FedNPG observed there is an empirical finding not covered by the theory.

Change location Remark 2 (p. 7); Proposition 1 and the discussion after it (p. 8); Remark 5 (p. 34); Appendix G.5 (p. 35). All changes are in blue in the revised manuscript.

AuthorsOpenReview ↗

**Official Review Comment 3/6 of Submission 34118 by Reviewer fT2o**

We thank the reviewer for this important observation regarding the potential circularity in the stability and convergence arguments.

Comment 3. Lemma 4 gives LL-smoothness only on S(kϵ)\mathcal S(k_\epsilon), and Step 3(b) applies it between KkK_k and Kk+1K_{k+1} in (103). The stability argument in Section 4 then uses Corollary 1, which depends on that step, to conclude Kk∈S(kϵ)K_k\in\mathcal S(k_\epsilon). This looks circular.

Response: The argument was circular as written. Lemma 4 was applied between KkK_k and Kk+1K_{k+1} before either point, or the segment joining them, was known to lie in S(kϵ){\mathcal{S}}(k_\epsilon), and the membership was then inferred from Corollary 1. Corollary 1 removes the circularity in three ways. The smoothness constant L:=max⁡K∈Sδ(ν)∥∇2Javg(K)∥{\mathcal{L}}:=\max_{K\in{\mathcal{S}}^\delta(\nu)}\left\lVert \nabla^2{\mathcal{J}_{\mathrm{avg}}}(K) \right\rVert is defined on the fixed compact set Sδ(ν){\mathcal{S}}^\delta(\nu), chosen before the iteration and independent of the iterates. The second condition in (15) bounds the length of each update by δ\delta, so that the segment [Kk,Kk+1][K_k,K_{k+1}] lies in Sδ(ν){\mathcal{S}}^\delta(\nu) whenever Kk∈S(ν)K_k\in{\mathcal{S}}(\nu); this is Step 1 of the proof of Theorem 1 and uses no convergence statement. The one-step bound together with the budget condition (20) then returns Kk+1K_{k+1} to S(ν){\mathcal{S}}(\nu), which closes the induction. The price is two additional hypotheses, the safe-step condition and the budget condition, which the submitted theorem did not state.

Change location Section 4: sets and constants (p. 6), Corollary 1 (p. 8) and the paragraph on closed-loop stability (p. 9); Lemma 8 (p. 30); Appendix G.4 (p. 35). All changes are in blue in the revised manuscript.

Reviewer_K6kyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the setting where multiple agents with the same dynamics collaborate the design a common control policy through natural policy gradient steps. The paper provides convergence guarantees for the natural policy optimization approach.

Critical strengths

The problem of designing close-loop feedback controllers for linear systems through collaboration across multiple systems is a very important problem and has already been addressed in the literature for multiple papers.

Critical weaknesses

See for example the papers that this paper overlooks:

  1. Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approach. TMLR.
  2. Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQR. L4DC.
  3. Combining federated learning and control: A survey
  4. Policy Gradient for LQR with Domain Randomization. CDC.
  5. Asynchronous parallel policy gradient methods for the linear quadratic regulator

The paper idea is interesting and relevant, however, it seems that the paper overlooks an important piece of literature, in particular, FedLQR and MAML-LQR which are approaches that leverages the collaboration across multiple dynamical systems to learn a common stabilizing controller that performs well for all the systems and generalizes to unseen dynamics and objectives. Those papers also prove convergence for policy optimization and shows the benefit of operating on a collaborative settings. Also, those papers considers heterogeneous dynamics and not homogenous dynamics as this paper.

Questions for authors

It is not clear to me in terms of the theoretical results and the empirical results the need for considering natural policy gradient under this federated setting. It’s clear that Natural policy gradient should attain better convergence rates than naive policy gradient but there is no comparison/discussions about it in this paper.

Does the theoretical convergence analysis also leverages the gradient dominance condition? If so how that is established for the average cost?

The writing is hard to follow, the paper does not provide the cost for the LQR problem for example, it should explicitly provide it in the main body.

4 comments
AuthorsOpenReview ↗

**Official Review Comment 1/4 of Submission 34118 by Reviewer K6ky**

We thank the reviewer for this insightful comment and for the references.

Comment 1. See for example the papers that this paper overlooks:

1. Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approach. TMLR.

2. Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQR. L4DC.

3. Combining federated learning and control: A survey

4. Policy Gradient for LQR with Domain Randomization. CDC.

5. Asynchronous parallel policy gradient methods for the linear quadratic regulator

The paper idea is interesting and relevant; however, it seems that the paper overlooks an important piece of literature, in particular, FedLQR and MAML-LQR, which are approaches that leverage the collaboration across multiple dynamical systems to learn a common stabilising controller that performs well for all the systems and generalises to unseen dynamics and objectives. Those papers also prove convergence for policy optimisation and show the benefit of operating in a collaborative setting. Also, those papers consider heterogeneous dynamics, not the homogeneous dynamics considered in this paper.

Response: We agree, and we thank the Reviewer for these references. The following paragraph has been added to the related work.

"Collaborative policy optimisation for LQR: FedLQR (Wang et al., 2026) establishes stability, convergence, and sample-efficiency guarantees for federated zeroth-order policy gradient under heterogeneous dynamics. Related studies investigate personalisation (Toso et al., 2024), domain randomisation (Fujinami et al., 2025), and asynchronous policy optimisation (Sha et al., 2024); see Weber et al. (2024) for a survey. Unlike these works, we focus on reducing the communication cost of curvature information in federated NPG by transmitting dominant eigenpairs and approximating the remaining spectrum with a scalar."

Change location Section 1.1, new paragraph (p. 3); Appendix A (p. 15); reference list. All changes are blue in the revised manuscript.

We also note that while our analysis focuses on homogeneous dynamics and does not explicitly consider heterogeneous dynamics, Figure 4 shows that the proposed FedNPG-ED framework can be numerically applied to heterogeneous settings, where agents have different local dynamics. However, we focus on the homogeneous case in this paper for theoretical clarity and tractability, since our primary objective is to address the communication overhead of transmitting curvature information as a large matrix from each agent to the server in every round. We study a compression scheme that retains the dominant eigenpairs while approximating the remaining spectrum with a scalar parameter, and leave a rigorous treatment of heterogeneous dynamics for future work, which we believe is an important and challenging direction for extending the applicability of FedNPG-ED to more realistic multi-agent scenarios.

AuthorsOpenReview ↗

**Official Review Comment 2/4 of Submission 34118 by Reviewer K6ky**

It is a very interesting comment.

Comment 2. It is not clear to me in terms of the theoretical results and the empirical results the need for considering natural policy gradient under this federated setting. It's clear that Natural policy gradient should attain better convergence rates than naive policy gradient but there is no comparison/discussions about it in this paper.

Response: The Reviewer is right that the submission neither discussed nor demonstrated the advantage of NPG over PG. The revision addresses this at two levels.

Motivation. For LQR the PG direction is 2EKΣK2E_K\Sigma_K and the NPG direction is 2EK2E_K. The Fisher preconditioner removes the state-covariance factor. Theorem 7 of (Fazel et al., 2018) shows that exact NPG admits a step size and an iteration complexity with a markedly better dependence on the problem parameters than gradient descent. In a federated setting this advantage has a price. FedPG uploads dθd_\theta scalars per agent and round, whereas FedNPG uploads dθ+dθ(dθ+1)/2d_\theta+d_\theta(d_\theta+1)/2. FedNPG-ED is designed to retain the preconditioning at an uplink of dθ(qk+1)+1d_\theta(q_k+1)+1 scalars.

Theory. We state plainly what the analysis does and does not show. Theorem 1 is based on Euclidean smoothness and cannot separate NPG from PG: for qk=0q_k=0, FedNPG-ED reduces to FedPG with step size s^k{\widehat{s}_k} and κ^k=1{\widehat{\kappa}_k}=1, and the certified factor is at most c/(16L)c/(16{\mathcal{L}}). The separation follows from Proposition 1. Since ∇2Javg(K\*)[Z]=2RK\*ZΣK\*\nabla^2{\mathcal{J}_{\mathrm{avg}}}(K_\*)[Z]=2R_{K_\*}Z\Sigma_{K_\*} by (111), we have L≥2∥RK\*∥2∥ΣK\*∥2{\mathcal{L}}\ge2\left\lVert R_{K_\*} \right\rVert_2\left\lVert \Sigma_{K_\*} \right\rVert_2, so the certified PG-type factor satisfies

\qquad\text{against}\qquad \frac{2}{25}\cdot\frac{\mu_\xi\\,{\lambda_{\min}}({\mathscr{R}})}{\left\lVert R_{K_k} \right\rVert_2\left\lVert \Sigma_{K_\*} \right\rVert_2}$$ for the NPG-type factor of Proposition 1 at its largest admissible step size. The ratio of the former to the latter is at most $\tfrac{25}{16}\\,(\mu_\xi/\left\lVert \Sigma_{K_\*} \right\rVert_2)\\,(\left\lVert R_{K_k} \right\rVert_2/\left\lVert R_{K_\*} \right\rVert_2)$. Near the optimum, the certified NPG-type factor is thus larger by a factor of order $\left\lVert \Sigma_{K_\*} \right\rVert_2/\mu_\xi$, which is at least the condition number of the stationary state covariance. This comparison concerns certified factors, that is, sufficient conditions, and requires the small-distortion condition $\gamma_k$ in Proposition 1. The following sentence has been added to the introduction. "For LQR, the NPG direction removes the state-covariance factor of the policy gradient direction. An exact NPG also admits a step-size and iteration-complexity bounds that depend more favourably on the problem parameters than those of policy gradient descent (Fazel et al., 2018, Theorem 7)." **Change location** Section 1, third paragraph (p. 2) and discussion after Proposition 1 (p. 8). All changes are in blue in the revised manuscript.
AuthorsOpenReview ↗

**Official Review Comment 3/4 of Submission 34118 by Reviewer K6ky**

We thank the reviewer for this important question.

Comment 3. Does the theoretical convergence analysis also leverages the gradient dominance condition? If so how that is established for the average cost?

Response: Yes. Gradient dominance is Lemma 5, inequality (48), and it is the step that converts the descent inequality into a contraction (Step 3 of the proof of Theorem 1). It is established for the average cost as follows.

Under the Gaussian policy (11), the ergodic cost is Javg(K)=tr⁡(PKΣwˉ)+tr⁡(RΣσ){\mathcal{J}_{\mathrm{avg}}}(K)={\operatorname{tr}}(P_K\Sigma_{\bar w})+{\operatorname{tr}}({\mathscr{R}}\Sigma_\sigma), see (27), where Σwˉ=Σw+BΣσB⊤\Sigma_{\bar w}=\Sigma_w+\mathbf B\Sigma_\sigma\mathbf B^{\top} and the stationary covariance solves the Lyapunov equation (28) driven by Σwˉ\Sigma_{\bar w}. Up to the additive constant tr⁡(RΣσ){\operatorname{tr}}({\mathscr{R}}\Sigma_\sigma), this is the cost of (Fazel et al., 2018) with the initial-state covariance replaced by Σwˉ\Sigma_{\bar w}. Their cost-difference identity (111) and Lemma 11 of (Fazel et al., 2018) therefore hold verbatim, as also shown for the ergodic cost by (Yang et al., 2019):

Javg(K)−Javg(K\*)≤∥ΣK\*∥tr⁡(EK⊤RK−1EK)≤∥ΣK\*∥λmin⁡(R)∥EK∥F2.{\mathcal{J}_{\mathrm{avg}}}(K)-{\mathcal{J}_{\mathrm{avg}}}(K_\*)\le\left\lVert \Sigma_{K_\*} \right\rVert{\operatorname{tr}}\big(E_K^{\top}R_K^{-1}E_K\big)\le\frac{\left\lVert \Sigma_{K_\*} \right\rVert}{{\lambda_{\min}}({\mathscr{R}})}\left\lVert E_K \right\rVert_F^2 .

With ∇Javg(K)=2EKΣK\nabla{\mathcal{J}_{\mathrm{avg}}}(K)=2E_K\Sigma_K and λmin⁡(ΣK)≥μξ{\lambda_{\min}}(\Sigma_K)\ge\mu_\xi, this gives (48). Because the agents share the dynamics and the cost, J(i)=Javg\mathcal J^{(i)}={\mathcal{J}_{\mathrm{avg}}} for all ii, and the weights sum to one, so the average cost inherits the property. This step would not hold under heterogeneous dynamics.

The Reviewer's question exposed an imprecision. In the submission, μξ\mu_\xi was introduced in Assumption 4.2 through the second moment of the initial state, a remnant of the random-initial-state formulation of (Fazel et al., 2018). This is not the relevant quantity for the ergodic cost. In the revision,

μξ:=λmin⁡(Σw+BΣσB⊤)>0,\mu_\xi:={\lambda_{\min}}\big(\Sigma_w+\mathbf B\Sigma_\sigma\mathbf B^{\top}\big) > 0 ,

which lower-bounds λmin⁡(ΣK){\lambda_{\min}}(\Sigma_K) for every K∈KK\in{\mathcal{K}} because ΣK⪰Σwˉ\Sigma_K\succeq\Sigma_{\bar w} by (25). This is the bound (44) required in Lemma 5.

Change location Assumption 4.2 (p. 6); LQR cost (13) in Section 4 (p. 6); proof of Lemma 5 (p. 28). All changes are in blue in the revised manuscript.

AuthorsOpenReview ↗

**Official Review Comment 4/4 of Submission 34118 by Reviewer K6ky**

We thank the Reviewer for pointing out the presentation.

Comment 4. The writing is hard to follow, the paper does not provide the cost for the LQR problem for example, it should explicitly provide it in the main body.

Response: In the submission, the LQR stage cost appeared only as the reward (6) in Section 2.2; the cost was defined implicitly as Javg:=−Ravg{\mathcal{J}_{\mathrm{avg}}}:=-\mathcal R_{\mathrm{avg}} in Section 4, and its closed form was deferred to (27) in Appendix G.1. We agree that this is hard to follow. Section 4 now states the cost explicitly.

LQR cost. For K∈KK\in{\mathcal{K}} and the policy (11), each agent incurs the average cost

\mathcal J(K)=\lim_{T\to\infty}\frac1T\\,\mathbb E\Big[\sum_{t=1}^{T}\big(x_t^{\top}{\mathscr{Q}} x_t+u_t^{\top}{\mathscr{R}} u_t\big)\Big] ={\operatorname{tr}}\big(({\mathscr{Q}}+K^{\top}{\mathscr{R}} K)\Sigma_K\big)+{\operatorname{tr}}({\mathscr{R}}\Sigma_\sigma), \end{aligned}$$ where $\Sigma_K=\Sigma_{\bar w}+(\mathbf A-\mathbf BK)\Sigma_K(\mathbf A-\mathbf BK)^{\top}$ is the stationary state covariance and $\Sigma_{\bar w}:=\Sigma_w+\mathbf B\Sigma_\sigma\mathbf B^{\top}$. By homogeneity, ${\mathcal{J}_{\mathrm{avg}}}=\mathcal J^{(i)}$, with $\nabla{\mathcal{J}_{\mathrm{avg}}}(K)=2E_K\Sigma_K$, $E_K:=({\mathscr{R}}+\mathbf B^{\top}P_K\mathbf B)K-\mathbf B^{\top}P_K\mathbf A$ and $P_K$ as in (29), so the natural-gradient direction is $2E_K$. **Change location** Section 4, after (12) (p. 6). All changes are in blue in the revised manuscript. *Further changes address readability.* Table 1 in Appendix B now distinguishes the exact, sampled and compressed quantities ($\Sigma_K$, ${\bar{\Sigma}_{K_k}}$, ${\widehat{\Sigma}_{K_k}}$ and the exact and sampled gradients), which the submission denoted in a way that was easy to confuse. The sets and constants used by Theorem 1 are defined in Section 4 immediately before it. Remarks 1 and 2 have been shortened to the statements supported by the corrected analysis, and Step 3 of the appendix proof is split into three labelled parts. **Change location** Section 4, before Theorem 1 (p. 6); Remarks 1 and 2 (p. 7); Appendix B (p. 15); Appendix G.3, Step 3 (p. 33). All changes are in blue in the revised manuscript.
Reviewer_LAceOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a communication-efficient federated natural policy gradient algorithm, FedNPG-ED. In FedNPG-ED, each agent transmits only some spectral information instead of the full fisher information matrix. The paper provides convergence results for this algorithm in the LQR setting and additionally provides some experimental evaluations.

Critical strengths
  1. This paper considers a relevant problem: reducing communication for federated NPG.
  2. Proposed algorithm seems relatively simple to implement.
Critical weaknesses
  1. Correctness: there seems to be several mathematical issues in the analysis and I'm unsure if the proof goes through at all.

(i) In equation (105), the authors seem to use the largest eigenvalue of the inverse covariance matrix instead of its smallest eigenvalue, which can be very different. (ii) In equations (100-101) the authors incorrectly bound the square of a sum by sum of squares and (iii) In (103), the smoothness inequality is applied using the estimated gradient instead of the true gradient.

  1. Algorithm novelty: The proposed algorithm seems very close to SHED, except that this algorithm pertains to federated NPG.

  2. Restrictive setting: the analysis is restricted to the homogenous LQR setting with exact critics which seems very narrow in comparison to the close "competitor" FedADMM in (Lan et al 2023).

Questions for authors

Please address the weakness listed, in particular, clarify if the proof can be fixed, what the novelties are and what components extend beyond the particular LQR setting used for the analysis.

3 comments
AuthorsOpenReview ↗

**Official Review Comment 1/3 of Submission 34118 by Reviewer LAce**

We thank the Reviewer for carefully checking the proof and for acknowledging the relevance of the problem and the algorithm's simplicity.

Comment 1. Correctness: there seems to be several mathematical issues in the analysis and I'm unsure if the proof goes through at all.

(i) In equation (105), the authors seem to use the largest eigenvalue of the inverse covariance matrix instead of its smallest eigenvalue, which can be very different. (ii) In equations (100-101), the authors incorrectly bound the square of a sum by sum of squares and (iii) In equation (103), the smoothness inequality is applied using the estimated gradient instead of the true gradient.

Response: The Reviewer is correct on all three points. The corrected results in Theorem 1 and its proof have been rewritten.

(i) Equation (105). The first-order term of the descent inequality requires a lower bound on tr⁡(G,Σ^Kk−1G⊤){\operatorname{tr}}(G\\,{\widehat{\Sigma}_{K_k}}^{-1}G^{\top}), which is λmin⁡(Σ^Kk−1)∥G∥F2=∥G∥F2/Λk{\lambda_{\min}}({\widehat{\Sigma}_{K_k}}^{-1})\left\lVert G \right\rVert_F^2=\left\lVert G \right\rVert_F^2/\Lambda_k with Λk=λmax⁡(Σ^Kk)\Lambda_k={\lambda_{\max}}({\widehat{\Sigma}_{K_k}}). We substituted 1/pk1/p_k, which is λmax⁡(Σ^Kk−1){\lambda_{\max}}({\widehat{\Sigma}_{K_k}}^{-1}). In the corrected proof the two eigenvalues are kept apart: 1/Λk1/\Lambda_k enters the first-order term (104) and 1/pk1/p_k enters the second-order term (105). As the Reviewer anticipates, the two can differ greatly, and their ratio κ^k=Λk/pk{\widehat{\kappa}_k}=\Lambda_k/p_k now appears in the contraction factor and in the residual (109).

(ii) Equations (100)–(101). The step from (100) to (101) bounded (εg1+∥GKk∥2)2({\varepsilon_{g_1}}+\left\lVert G_{K_k} \right\rVert_2)^2 by εg12+∥GKk∥22{\varepsilon_{g_1}}^2+\left\lVert G_{K_k} \right\rVert_2^2 and thus missed a factor of 22. In the corrected proof, this step is no longer needed, because gradient dominance (Lemma 5) is applied directly to the exact gradient. The only place where the square of a sum is bounded is (105), where the factor 22 is present.

(iii) Equation (103). The descent inequality of Lemma 4 holds with the exact gradient ∇Javg(Kk)=GKk′\nabla{\mathcal{J}_{\mathrm{avg}}}(K_k)={G'_{K_k}}, not with the sampled gradient. The corrected inequality is (103). The resulting cross term −ηk⟨GKk′,ekΣ^Kk−1⟩-\eta_k\langle{G'_{K_k}},e_k{\widehat{\Sigma}_{K_k}}^{-1}\rangle is retained and bounded in (104); it contributes the term s^kκ^k/2{\widehat{s}_k}{\widehat{\kappa}_k}/2 to the residual coefficient in (109).

(iv) An additional error. The last line of (105) replaced ∥GKk∥F2\left\lVert G_{K_k} \right\rVert_F^2 by r∥GKk∥22r\left\lVert G_{K_k} \right\rVert_2^2 in a term with a negative sign, which requires ∥G∥F2≥r∥G∥22\left\lVert G \right\rVert_F^2\ge r\left\lVert G \right\rVert_2^2, whereas the opposite inequality holds. The corrected proof works in the Frobenius norm throughout. The rank enters only through ∥ek∥F2≤rˉ,εg12\left\lVert e_k \right\rVert_F^2\le{\bar{r}}\\,{\varepsilon_{g_1}}^2.

We have corrected Step 3 in the proof of Theorem 1 accordingly. Linear convergence to a neighbourhood of K\*K_\* and closed-loop stability of the iterates remain valid, at the cost of a weaker and differently structured statement; see Theorem 1 and Corollary 1 for the corrected results. The claims about the roles of pkp_k and qkq_k derived from (105) do not hold, and we have restated them.

Change location Section 4: Theorem 1 (p. 7), Remarks 1 and 2 (p. 7) and Corollary 1 (p. 8); Appendix G.3, Step 3 of the proof (p. 33). All changes are in blue in the revised manuscript.

AuthorsOpenReview ↗

**Official Review Comment 2/3 of Submission 34118 by Reviewer LAce**

We thank the Reviewer for giving us the opportunity to clarify the key methodological distinctions and highlight the specific contributions of our approach.

Comment 2. Algorithm novelty: The proposed algorithm seems very close to SHED, except that this algorithm pertains to federated NPG.

Response: We agree that the compression primitive, a rank-eigen-approximation completed by a scalar for the discarded spectrum, is that of SHED (Dal Fabbro et al., 2024). The submission states this in Sections 1.1 and 1.2, and we do not claim the primitive as a contribution. The revised contribution statement says so explicitly. What is specific to the present work is the following.

Mechanism. SHED compresses the Hessian of a loss on a static dataset and transmits eigenpairs incrementally, reusing those sent in earlier rounds. In our setting, the matrix to be compressed is the Fisher matrix of the current policy, re-estimated from fresh on-policy trajectories in every round, so that incremental reuse is not directly applicable and fresh eigenpairs are sent per round. For the linear Gaussian policy, the Fisher matrix factorises as (37), so only the state covariance is compressed, and the dominant eigenpairs are computed from the score matrix without forming the Fisher matrix (Section 3).

Analysis. The guarantees of SHED are obtained for (strongly) convex losses. Here the cost is non-convex, the gradient is sampled, and every iterate must remain in the stabilising set, a constraint without counterpart in SHED. The corrected analysis provides a linear rate under gradient dominance with a sampled gradient and a compressed preconditioner (Theorem 1), invariance of a sublevel set and hence closed-loop stability (Corollary 1), and a condition on the compression under which the contraction factor of exact NPG is recovered (Proposition 1).

Evaluation. The method is evaluated with linear and neural actors, on LQR and on MuJoCo tasks, under unequal local sample sizes, against uncompressed FedNPG and FedNPG-ADMM.

The following sentence has been added in Section 1.2.

"The compression primitive itself is that of SHED. Our contribution is its transfer to the policy-dependent Fisher matrix of federated NPG, which is re-estimated in every round, together with a control-theoretic analysis of the resulting method."

Change location Section 1.2, first paragraph (p. 3). All changes are in blue in the revised manuscript.

AuthorsOpenReview ↗

**Official Review Comment 3/3 of Submission 34118 by Reviewer LAce**

We thank the reviewer for this valuable comment regarding the scope of our theoretical analysis.

Comment 3. Restrictive setting: the analysis is restricted to the homogenous LQR setting with exact critics which seems very narrow in comparison to the close “competitor” FedADMM in (Lan et al., 2023).

Response: We agree that the analysis is done only for homogeneous dynamics, a linear Gaussian actor and an exact critic. These restrictions are stated in the Scope and limitations paragraph of Section 1.2. We do not claim greater generality than (Lan et al., 2023). The two analyses answer different questions, and neither contains the other.

Type of guarantee. (Lan et al., 2023) consider a discounted MDP shared by all agents, i.e., homogeneous across agents, with a general policy parameterisation, and establish convergence to an ϵ\epsilon-stationary point with O(dθ)\mathcal O(d_\theta) uplink per round. For the control problem considered here, stationarity is not sufficient. The iterates must remain in the stabilising set, which is non-convex and whose boundary is unbounded in cost, and the quantity of interest is the gap to the optimal controller. Theorem 1 and Corollary 1 address these properties. Conversely, analyses for general MDPs typically rely on bounded rewards and uniformly bounded score functions, which do not hold for LQR with a Gaussian policy, so that the LQR case is not covered by such results.

Heterogeneity. Both works assume that all agents interact with the same MDP. Our setting additionally allows unequal local sample sizes τi,k\tau_{i,k} with sample-weighted aggregation, which was not considered in (Lan et al., 2023) and introduces an additional layer of heterogeneity in the federated learning setup.

What extends beyond LQR. (a) Algorithm 1 is independent of the dynamics and of the policy class, and is evaluated with neural actors on Hopper-v5 and Humanoid-v5. (b) Lemma 6, the spectral bounds (101) and the fidelity bounds (55) and (119) are linear-algebraic and hold for any Fisher matrix. (c) The proof of Theorem 1 uses only three properties of the objective: smoothness on a compact neighbourhood of a sublevel set, gradient dominance, and compactness of sublevel sets within the domain. It therefore applies to any objective with these properties. (d) Theorem 1 holds for an arbitrary gradient error eke_k. Assumption 4.5 is used only to identify eke_k with the sampling error; with a learned critic, the same bound holds with the critic-induced bias included in the bound on ∥ek∥F\left\lVert e_k \right\rVert_F. An analysis of the critic recursion itself remains future work. The LQR-specific ingredients are the gradient-dominance constant (Lemma 5), coercivity (Lemma 1), the Kronecker factorisation (37), and Proposition 1.

The following sentence has been added to the comparison in Section 1.1.

"(Lan et al., 2023) consider a general discounted MDP shared by all agents and prove convergence to an ϵ\epsilon-stationary point, without addressing closed-loop stability or the gap to the optimal controller, whereas our theoretical analysis is restricted to the LQR setting; it establishes these two properties."

Change location Section 1.1, first paragraph (p. 2). All changes are in blue in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

**Response to the Area Chair's and the Reviewers' comments**

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all three reviewers for their careful evaluation of our submission and their thoughtful, constructive feedback. The reviews have identified important issues in the theoretical analysis, particularly in the proof of Theorem 1, that affect several statements in the submitted manuscript. We acknowledge these issues and have carefully revised the analysis to address the reviewers' concerns. The feedback has helped us improve the mathematical rigour, clarity, and presentation of our work.

Our primary contribution is a communication-efficient federated natural policy gradient framework, FedNPG-ED, together with stability and convergence guarantees. In this response, we explicitly distinguish the results that have been corrected, the claims that have been withdrawn, and the conclusions that remain valid. The revised manuscript provides the corrected theoretical statements and their complete proofs.

Each reviewer comment is reproduced in italics and followed by our response. All modifications in the revised manuscript are highlighted in blue. Equation, lemma, and section numbers appearing in the reviewers' comments and our responses refer to the submitted manuscript unless otherwise specified. For each modification, we indicate its location in the revised manuscript.

Summary of the main changes

The three reviews raised important concerns regarding the theoretical analysis, particularly the validity of the proof of Theorem 1. We acknowledge these concerns and have revised the relevant results and supporting arguments. The principal changes are summarised below.

  1. Corrected convergence result. Step 3 of the proof of Theorem 1 contained three errors identified by Reviewers LAce and fT2o, namely the eigenvalue used in (105), the gradient used in (103), and a missing factor between (100) and (101). While rechecking the proof, we found a fourth error: a reversed norm inequality in the last line of (105). The corrected one-step bound is Theorem 1. Linear convergence to a neighbourhood of K\*K_\* is retained. The contraction factor and residual error now depend on the condition number of the compressed curvature matrix, not only on the ratio ηk/pk\eta_k/p_k.

  2. Non-circular stability argument. The argument of Section 4 used Corollary 1 to place the iterates in the set on which Lemma 4 holds, while Corollary 1 itself relied on Lemma 4. It is replaced by the induction in Corollary 1, in which the smoothness constant is defined on a fixed compact set before the iteration starts.

  3. Claims on the design parameters. The statements that the rate depends on (ηk,pk)(\eta_k,p_k) only through their ratio, that pkp_k should be taken as small as the step-size condition permits, and that the error floor is independent of pkp_k are withdrawn.

  4. Role of the transmitted eigenpairs. The submitted bound did not involve the retained eigenpairs. Remark 5 gives the exact dependence of the one-step bound on the transmitted eigenvectors, and Proposition 1 shows that the contraction factor of exact NPG is recovered once the retained eigenpairs keep the relative curvature distortion below an explicit threshold.

  5. Positioning. A paragraph on federated and collaborative policy optimisation for LQR has been added, covering the five references indicated by Reviewer K6ky. The relation to SHED and to FedNPG-ADMM is stated more precisely.

  6. Presentation and experiments. The LQR cost and its closed form are now stated in Section 4, and the motivation for NPG over PG is discussed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.