acceptodds
Withdrawn from ICLR 2027

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) signal-weighted aggregation that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) global reward calibration that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) adaptive sparse communication that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by and supporting up to compression under tight bandwidth budgets with only graceful accuracy degradation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
federated learning, GRPO, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_spc3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript explores the problem of federated reinforcement learning, which utilizes Group Relative Policy Optimization to fine-tune the reasoning capabilities of large language models in a privacy-constrained decentralized environment. It proposes the Fed-GRPO framework, which utilizes reward statistics naturally generated during the GRPO training process as a zero-cost signal to guide model aggregation, local training, and communication processes. Finally, the authors provide a non-convex convergence analysis and validate the framework using Qwen2.5-3B-Instruct and Qwen3-4B-Instruct on the difficulty- or subject-based mathematical reasoning dataset MATH and code generation dataset MBPP.

Critical strengths

The topic selection of the manuscript possesses certain research value, and it clearly proposes an overall framework that utilizes the built-in reward statistics of GRPO to collaboratively guide aggregation, target calibration, and sparse communication.

Critical weaknesses

The proof of convergence in the manuscript contains numerous errors, and there are also obvious typographical errors, such as the author's surname being placed outside the parentheses in all citations.

Questions for authors

(1) There are many apparent errors in the proof of convergence of the method, and it is recommended that the author carefully verify them.

(2) When B=100%, adaptive sparse communication is mathematically lossless. However, after adding +ASC (B=1.0) in Table 4, the accuracy of the test set changes.

(3) When the difficulty prompt indicates that all samples are incorrect, the intra-group advantage Aˆi=0A ˆ_i=0 and βKL=0β_{KL} =0, and the gradient remains constant at 0. How can increasing the weight ωqω_q generate an effective learning signal in this case?

(4) Is the fact that 96.2% of the parameters remain unchanged after local training due to the algorithmic characteristics of GRPO, or is it a floating-point underflow artifact caused by pure BF16 weights at an extremely small learning rate? If standard FP32 main weight training is used, what is the sparsity?

(5) The title of Table 3 is code tasks, but the benchmarks in the table are mathematical.

Reviewer_R6efOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Fed-GRPO, a federated reinforcement learning framework for collaboratively improving LLM reasoning without sharing raw training data. It leverages reward statistics from GRPO to guide client aggregation, reweight local training objectives, and adaptively compress model updates. Experiments on mathematical reasoning benchmarks report improvements over federated baselines and performance approaching centralized training, alongside substantial reductions in transmitted parameters and a configurable accuracy–communication trade-off.

Critical strengths
  1. The paper studies federated GRPO under heterogeneous data distributions, addressing the important challenge of collaborative reasoning training without centrally pooling raw training data.

  2. Reusing reward statistics for client aggregation, prompt weighting, and communication allocation provides a coherent framework without requiring an additional critic or a separate client-quality estimation model.

  3. The experiments cover mathematical reasoning and code generation, two model backbones, and different client-partition settings. Component ablations and communication-budget sweeps provide useful evidence about the method’s empirical behavior.

Critical weaknesses
  1. The evaluation protocol limits confidence in the reported improvements. Appendix C reports the best accuracy across checkpoints evaluated on the benchmark datasets, without clearly identifying an independent validation set for checkpoint selection. This introduces potential test-set selection bias, particularly if different checkpoints are selected for different benchmarks. Moreover, the average improvements over FGRPO are approximately 0.57 and 0.52 percentage points for the two model backbones, yet variability across independent training seeds is not reported. Sampling 32 responses per prompt does not establish training robustness.

  2. The communication claims require stronger qualification and matched controls. The headline compression ratios of 32× and 621× refer to parameter counts; the corresponding byte-level reductions are approximately 11× and 207× after accounting for indices. The reported costs also cover uplink transmission rather than total communication. Crucially, transmitting only changed parameters is not exclusive to reward-driven coordination. Comparisons against FedAvg and FGRPO using the same lossless encoding, together with adaptive-versus-fixed allocation under matched byte budgets, are needed to isolate the proposed method’s contribution.

  3. The convergence analysis contains unresolved assumptions and derivation issues. The proof introduces covariance conditions around Eqs. (17) and (21) that are not included in the assumptions of Theorem 1. In addition, the negative covariance lower bound stated below Eq. (21) does not justify the claimed factor of one-half in the resulting upper bound. The authors should explicitly state the additional conditions, justify their applicability to reward-dependent aggregation and compression, and correct the derivation before relying on the theorem.

The discussion could more clearly position parameter-aggregation-based federated GRPO relative to collaborative RL through experience sharing. Relevant examples include Heterogeneous Agent Collaborative Reinforcement Learning, which enables heterogeneous agents to learn from shared verified rollouts, and Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing, which introduces decentralized, asynchronous RL through rollout sharing. These approaches operate under different information-sharing assumptions and are not necessarily directly comparable baselines. Discussing them in the paper would help clarify Fed-GRPO’s scope and contribution.

Questions for authors

(1) How were checkpoints selected? Please provide results using a single checkpoint selected on held-out validation data, together with variability across independent training seeds.

(2) How much of the communication gain is specific to Fed-GRPO?

Reviewer_WV2FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Fed-GRPO, a framework for collaboratively training a shared language model with Group Relative Policy Optimization on non-IID local data. The central idea is to reuse reward statistics already computed during GRPO for three purposes: weighting client updates by reward standard deviation, reweighting prompt objectives according to their reward relative to global statistics, and allocating communication budgets before sparsifying model updates. The paper also presents a convergence analysis. Experiments cover mathematical reasoning with Qwen2.5-3B-Instruct and Qwen3-4B-Instruct, coding tasks, and federations with 5/20 clients.

Critical strengths

The problem is relevant and well motivated. Federated reinforcement learning for language models is useful when institutions can collaborate without sharing training samples.

The proposed coordination signals are inexpensive and straightforward to obtain. Reward means and group standard deviations are already available during GRPO. Using these statistics avoids introducing an additional reward model or a server-side evaluation dataset. The aggregation and calibration rules are also relatively simple to implement.

The paper provides useful ablations. Table 4 reports incremental gains from signal-weighted aggregation and reward calibration. Table 1 distinguishes parameter-count compression from byte compression and includes index overhead.

Critical weaknesses
  1. Relevant prior work (e.g. https://openreview.net/pdf?id=91b5jtd492) is omitted from both the discussion and evaluation.

The submission does not cite or compare against “PRPO: Collaborative Online Policy Learning in Personalized RLHF” The work studies collaboration among clients that retain their trajectories locally, using peer-derived reference policies and KL regularization to communicate learning signals. This makes it highly relevant to setting studied in the paper.

The authors should explain this distinction explicitly and identify what Fed-GRPO contributes beyond existing collaborative online policy optimization. A controlled comparison with an appropriately adapted peer-reference approach would strengthen the submission; if the settings prevent a meaningful comparison, please provide a concrete explanation.

  1. The convergence theorem is not established under its stated assumptions.

Theorem 1 lists Assumptions 1–4, but Appendix B introduces additional conditions:

  1. The proof assumes a bound on the covariance between the current-round aggregation weights and stochastic gradients, then requires that bound to shrink with the expected global gradient norm.
  2. The proof also introduces another covariance condition involving the adaptive communication budget and update magnitude. These conditions do not follow from the stated smoothness, bounded-gradient, compression, and drift assumptions. They are particularly consequential because the weights, budgets, and gradients are all derived from the same training process.

Furthermore, Eq. (20) assumes that the compressor retains exactly the prescribed fraction of changed parameters. Section 4.3 instead removes whole modules until the retained count roughly matches the target. With unequal module sizes, this can overshoot substantially, invalidating the claimed contraction factor for the nominal budget.

Finally, the analysis concerns an SGD-style update and a changing weighted surrogate, whereas the experiments use AdamW. The scope of the theorem should be stated accordingly. Even a corrected theorem would not establish superiority over FedAvg on a fixed population objective.

  1. The compression algorithm has unresolved operational inconsistencies.

Equation (8) does not generally make B=1B=1 lossless. For positive adaptation strength and a client whose normalized signal falls below the client average, Bk=clip(1+α(σ^k−σ^ˉ),Bmin,1)<1B_k = clip(1+\alpha(\hat{\sigma}_k - \bar{\hat{\sigma}}), B_min ,1) < 1. Thus some clients would still discard updates, contrary to the interpretation of the B=100B=100\\% experiments. A special-case override may resolve this, but none is specified.

There is also a synchronization gap in Algorithm 1. A client computes its budget using the maximum and average of all clients’ current-round reward standard deviations before sending its statistics to the server. Those quantities require an additional statistics exchange, or the algorithm must use previous-round statistics instead.

  1. Appendix C states that checkpoints are evaluated on all five benchmarks and that the best accuracy across checkpoints is reported. This uses evaluation data for model selection. It is also unclear whether each reported row corresponds to one selected checkpoint or combines separately selected checkpoints across benchmarks.

No independent training-seed variability or confidence intervals are reported. Generating 32 responses per prompt reduces evaluation sampling noise, but does not measure variability from training, client partitioning, or model selection. This matters because the average improvements over FGRPO are approximately 0.57 points in Table 1 and 0.52 points in Table 2. Likewise, a 0.01-point difference from centralized training cannot establish practical equivalence without uncertainty estimates.

  1. The origin and generality of “natural sparsity” remain unclear, and the headline communication claims are overstated.

The observation that over 96.296.2\\% of parameters remain exactly unchanged deserves a more careful explanation. The experiments use BF16 mixed precision and a learning rate of 3×10−63\times10^{-6}. Depending on parameter storage and optimizer implementation, small updates can disappear through finite precision rounding. This is a plausible alternative explanation that should be tested, rather than assuming the sparsity is an intrinsic property of GRPO. So the authors should report how unchanged parameters are detected, the precision used for parameters and deltas, optimizer-state handling, and whether the observation persists with FP32 updates, different learning rates, and longer local training.

In addition, Algorithm 1 still broadcasts a dense global model. Uplink savings alone do not establish equivalent reductions in total communication or training time. Both communication directions and compression overhead should be reported.

  1. The privacy claim should be scoped to retaining raw data locally. Individual model updates and reward statistics are still transmitted; the paper does not establish a formal privacy guarantee.
Questions for authors

How does Fed-GRPO differ from peer-reference collaboration in PRPO? Can a controlled comparison or an explanation of why such a comparison would be inappropriate?

Can you restate and prove Theorem 1 with all required covariance assumptions, clarify the conditioning on current-round statistics, and account for module-level budget rounding?

Does the reported sparsity persist with FP32 parameter updates and different learning rates? Can you demonstrate exact reconstruction in the lossless setting and compare against baselines using the same encoder?

Can you isolate reward-adaptive bandwidth allocation at matched byte budgets and show that reward standard deviation predicts actual update utility?

How does the method behave for clients with persistently near-zero success rates?

Can you specify the missing communication hyperparameters, clarify compute matching and local-only evaluation, and correct the coding-table labels and Table 5 arithmetic?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.