- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of federated reinforcement learning. The authors propose FedVR-DistTD algorithm based on variance-reduced federated categorical distributional TD learning with linear function approximation. Meanwhile, the authors establish its finite-time convergence guarantees, indicating that the proposed method can converge geometrically and achieves the root-error convergence convergence rate of .
Critical strengths
The variance-reduced mechanism is considered in FedVR-DistTD, which is efficient to reduce the variance without sharing the private training trajectories, and this is crucial in decentralized setting. Meanwhile, The authors provide a solid finite-time guarantees and the corresponding analysis, and the ablation studies are conducted to evaluate the performance of each component of FedVR-DistTD.
Critical weaknesses
-
The experiments are conducted under full participation and i.i.d. data sampling strategy, which is ideal and can not reflect the practical decentralized setting.
-
In the contribution statement (line 110), the authors state that the finite-time guarantees are established under Markovian sampling. However, Assumption 1 (line 244) requires that the drawn samples for each client are i.i.d. from the distribution . The statement and the assumption are contradictory. Meanwhile, authors state that the Markovian sampling still remains open, which is also contradictory with the contribution stated in line 110.
-
The manuscript lacks numerical comparisons of communication costs of the proposed method and corresponding baselines, which is important to evaluate the method's efficiency under federated setting.
Questions for authors
-
Could authors clarify the theoretical difference between the claimed theoretical results under Markovian sampling and the i.i.d. sampling strategy in Assumption 1? Do the results still hold under these two conditions?
-
Could authors provide more detailed numerical comparisons of communication costs of different methods?
Thank you for your detailed summary and constructive suggestions. We have some issues with the theory being not rigorous and the experiments being insufficient. We have already made supplementary revisions.
We thank the reviewer for identifying this inconsistency. The original contribution statement incorrectly described our guarantees as applying to Markovian sampling. We have corrected the introduction and contribution statement and clarified the assumptions in Section3.3: our finite-time results hold for homogeneous clients with full participation under independent generative-model sampling, subject to the stated stability conditions. The additional analysis in Appendices D.4-D.6 also operates under Assumption1; it does not establish a Markovian guarantee.
The distinction is that our sampler draws a fresh state from the stationary distribution for each transition and then draws its successor from the Markov kernel. Consecutive samples along a Markov trajectory may have the same stationary marginal distribution, but they are temporally dependent. In that case, the conditional centering and independence arguments used for the control batches and local updates, including the 1/(NB) variance calculation, do not follow from Assumption1. Thus the stated guarantees hold under the independent-sampling protocol, but their validity and rates under continuous Markov trajectories have not been established. We have made this limitation explicit and leave a mixing-dependent extension for future work. AppendixG reports Markovian results as empirical evidence only.
We have added numerical communication accounting in Appendix G.1. With 16 features and 51 categorical intervals, the per-client payloads are 13,056, 19,584, and 26,112 bytes per round for FedDistTD-NoVR, SCAFFLSA-Linear-CTD, and FedVR-DistTD, respectively. These counts include parameter transfers in both directions. Under the same message protocol and representation, the per-round byte counts do not depend on whether transitions are sampled independently or along Markovian trajectories.
The table reports completed-run costs at nearly equal per-client transition budgets. FedVR-DistTD's transition count includes samples used to construct its control direction.
FedVR-DistTD sends more bytes per round but completes the nearly sample-matched run with fewer rounds and less total communication. Appendix G.4 additionally compares error trajectories under matched transition and communication budgets , showing lower tail-averaged error for FedVR-DistTD under the tested configurations.