- Relevant prior work (e.g. https://openreview.net/pdf?id=91b5jtd492) is omitted from both the discussion and evaluation.
The submission does not cite or compare against “PRPO: Collaborative Online Policy Learning in Personalized RLHF” The work studies collaboration among clients that retain their trajectories locally, using peer-derived reference policies and KL regularization to communicate learning signals. This makes it highly relevant to setting studied in the paper.
The authors should explain this distinction explicitly and identify what Fed-GRPO contributes beyond existing collaborative online policy optimization. A controlled comparison with an appropriately adapted peer-reference approach would strengthen the submission; if the settings prevent a meaningful comparison, please provide a concrete explanation.
- The convergence theorem is not established under its stated assumptions.
Theorem 1 lists Assumptions 1–4, but Appendix B introduces additional conditions:
- The proof assumes a bound on the covariance between the current-round aggregation weights and stochastic gradients, then requires that bound to shrink with the expected global gradient norm.
- The proof also introduces another covariance condition involving the adaptive communication budget and update magnitude.
These conditions do not follow from the stated smoothness, bounded-gradient, compression, and drift assumptions. They are particularly consequential because the weights, budgets, and gradients are all derived from the same training process.
Furthermore, Eq. (20) assumes that the compressor retains exactly the prescribed fraction of changed parameters. Section 4.3 instead removes whole modules until the retained count roughly matches the target. With unequal module sizes, this can overshoot substantially, invalidating the claimed contraction factor for the nominal budget.
Finally, the analysis concerns an SGD-style update and a changing weighted surrogate, whereas the experiments use AdamW. The scope of the theorem should be stated accordingly. Even a corrected theorem would not establish superiority over FedAvg on a fixed population objective.
- The compression algorithm has unresolved operational inconsistencies.
Equation (8) does not generally make B=1 lossless. For positive adaptation strength and a client whose normalized signal falls below the client average, Bk=clip(1+α(σ^k−σ^ˉ),Bmin,1)<1. Thus some clients would still discard updates, contrary to the interpretation of the B=100 experiments. A special-case override may resolve this, but none is specified.
There is also a synchronization gap in Algorithm 1. A client computes its budget using the maximum and average of all clients’ current-round reward standard deviations before sending its statistics to the server. Those quantities require an additional statistics exchange, or the algorithm must use previous-round statistics instead.
- Appendix C states that checkpoints are evaluated on all five benchmarks and that the best accuracy across checkpoints is reported. This uses evaluation data for model selection. It is also unclear whether each reported row corresponds to one selected checkpoint or combines separately selected checkpoints across benchmarks.
No independent training-seed variability or confidence intervals are reported. Generating 32 responses per prompt reduces evaluation sampling noise, but does not measure variability from training, client partitioning, or model selection. This matters because the average improvements over FGRPO are approximately 0.57 points in Table 1 and 0.52 points in Table 2. Likewise, a 0.01-point difference from centralized training cannot establish practical equivalence without uncertainty estimates.
- The origin and generality of “natural sparsity” remain unclear, and the headline communication claims are overstated.
The observation that over 96.2 of parameters remain exactly unchanged deserves a more careful explanation. The experiments use BF16 mixed precision and a learning rate of 3×10−6. Depending on parameter storage and optimizer implementation, small updates can disappear through finite precision rounding. This is a plausible alternative explanation that should be tested, rather than assuming the sparsity is an intrinsic property of GRPO. So the authors should report how unchanged parameters are detected, the precision used for parameters and deltas, optimizer-state handling, and whether the observation persists with FP32 updates, different learning rates, and longer local training.
In addition, Algorithm 1 still broadcasts a dense global model. Uplink savings alone do not establish equivalent reductions in total communication or training time. Both communication directions and compression overhead should be reported.
- The privacy claim should be scoped to retaining raw data locally. Individual model updates and reward statistics are still transmitted; the paper does not establish a formal privacy guarantee.