- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In federated natural policy gradient, each agent sends a large Fisher matrix to the server every round, which costs a lot of communication. This paper proposes FedNPG-ED where each agent sends only its top eigenvectors and that replaces the remaining eigenvalues. The server rebuilds an approximate matrix and updates the shared policy. For LQR with a linear Gaussian policy, a perfect critic and identical agents, the paper claims linear convergence to a neighbourhood. It also claims the convergence speed depends only on .
Critical strengths
For a linear Gaussian policy, the Fisher matrix splits. So only the small state covariance needs compressing which provides communication saving.
Low local cost. The top eigenvectors can be computed from the score matrix. Agents never need to build or fully decompose the Fisher matrix, so the method can scale.
Critical weaknesses
In the main proof equation 105 is replaced by . But is the largest eigenvalue of . So these main claims may not be supported.
The proof never uses the sent eigenvectors, so the theory does not explain why sending them helps.
Lemma 4 gives -smoothness only on , and Step 3(b) applies it between and in (103). The stability argument in Section 4 then uses Corollary 1, which depends on that step, to conclude . This looks circular.
Questions for authors
In equation 105, why is taken as and not ? Which claims about and still hold after this is fixed?
Where is the error term ?
Can you give an induction showing that and the segment stay in
6 comments
**Official Review Comment 2/6 of Submission 34118 by Reviewer fT2o**
We thank the reviewer for this insightful observation regarding the role of the transmitted eigenvectors in our theoretical analysis.
Comment 2. The proof never uses the sent eigenvectors, so the theory does not explain why sending them helps.
Response: We agree. In the submitted proof, the compressed matrix entered only through , so that neither the retained eigenvalues nor the eigenvectors played any role. The submitted bound is in fact the one obtained for , where no eigenpair is sent. The revision makes the following changes.
First, the corrected bound depends on the retained eigenpairs. In Theorem 1, they enter through , and Remark 5 gives the exact dependence: the first-order coefficient is the alignment coefficient , which for a single agent is the eigenvector-weighted average of and .
Second, we do not claim that Theorem 1 explains why sending eigenvectors helps. It is a worst-case bound based on Euclidean smoothness, and such a bound cannot favour a non-isotropic preconditioner: its certified factor is largest for . Proposition 1 provides the explanation. The eigenvectors determine the relative distortion of Proposition 1, which is bounded by with non-increasing in . Once , the contraction factor is that of exact NPG and involves neither nor . Sending eigenpairs helps because it moves the method from the regime of Theorem 1 to that of Proposition 1.
Third, the distortion bound , which was stated in Appendix G.5 without connection to the convergence result, is now stated in Remark 2 next to Proposition 1, and the alignment identity (119), written for the exact gradient, gives the first-order coefficient of the corrected bound in Remark 5. The concentration bound of Step 2(a) of the submitted proof, which was not used in Step 3, is now used to bound in Proposition 1.
We also state the limitation. Proposition 1 is a sufficient condition. For aggressive compression, such as in Figures 1 and 2 (Section 5), it need not hold; the closeness to uncompressed FedNPG observed there is an empirical finding not covered by the theory.
Change location Remark 2 (p. 7); Proposition 1 and the discussion after it (p. 8); Remark 5 (p. 34); Appendix G.5 (p. 35). All changes are in blue in the revised manuscript.
**Official Review Comment 3/6 of Submission 34118 by Reviewer fT2o**
We thank the reviewer for this important observation regarding the potential circularity in the stability and convergence arguments.
Comment 3. Lemma 4 gives -smoothness only on , and Step 3(b) applies it between and in (103). The stability argument in Section 4 then uses Corollary 1, which depends on that step, to conclude . This looks circular.
Response: The argument was circular as written. Lemma 4 was applied between and before either point, or the segment joining them, was known to lie in , and the membership was then inferred from Corollary 1. Corollary 1 removes the circularity in three ways. The smoothness constant is defined on the fixed compact set , chosen before the iteration and independent of the iterates. The second condition in (15) bounds the length of each update by , so that the segment lies in whenever ; this is Step 1 of the proof of Theorem 1 and uses no convergence statement. The one-step bound together with the budget condition (20) then returns to , which closes the induction. The price is two additional hypotheses, the safe-step condition and the budget condition, which the submitted theorem did not state.
Change location Section 4: sets and constants (p. 6), Corollary 1 (p. 8) and the paragraph on closed-loop stability (p. 9); Lemma 8 (p. 30); Appendix G.4 (p. 35). All changes are in blue in the revised manuscript.
**Official Review Comment 1/6 of Submission 34118 by Reviewer fT2o**
We thank the Reviewer for the precise reading of the proof and for noting the role of the Kronecker structure and of the score-matrix computation. Each of the three weaknesses is a genuine gap, and the questions below determined the structure of the corrected analysis.
Comment 1. In the main proof equation 105 λmin(Σ−1) is replaced by 1/pk. But 1/pk is the largest eigenvalue of Σ−1. So these main claims may not be supported.
Response: The Reviewer is correct. By (14) and (101), λmin(ΣKk−1)=1/Λk with Λk=λmax(ΣKk), and 1/pk is an upper bound on λmax(ΣKk−1). The substitution made in (105) is invalid, and the claims derived from it are not supported. The corrected bound is given in Theorem 1, in which 1/Λk appears in the first-order term and 1/pk in the second-order term. Also see our response to Comment 4.
Change location Theorem 1 (p. 7); Appendix G.3, Steps 3(c) and 3(d) (p. 34). All changes are in blue in the revised manuscript.