- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
A federated version of GRPO for RLVR fine-tuning of vision-language models under non-IID clients. Each client trains a LoRA adapter locally and uploads it plus one scalar “relative performance gain” (round-mean reward minus an EMA baseline over a clipped EMA of deviations, Eq. 14); the server maps the scalars to annealed softmax weights (Eq. 15), forms a weighted pseudo-gradient, and takes a FedAdam-style step (Eqs. 16-18). Theorem 3.1/A.1 gives an O(T^{-1/2}) + O(1) stationarity bound. Validated on four VLM backbones over Open-R1 and GEOQA, 5 clients at Dirichlet(0.05), against FedAvg/FedProx/SCAFFOLD, with RPG on/off and heterogeneity ablations.
Critical strengths
-
The mechanism is lightweight. Per round each client transmits a single extra scalar (Eq. 14) that the server converts into softmax weights (Eq. 15). Because the rule never touches the local objective, it extends to GSPO and DAPO unchanged (Sec. C.4). Federated RLVR does need some such weighting rule, and a drop-in version would be easy for others to adopt if the gain is real.
-
The only direct on/off test of RPG is Table 3, backed by the reward curves in Figs. 11-12, and on GEOQA the effect looks genuine: FGSPO rises from to and FDAPO from to . Since the RPG settings were swept on Open-R1 alone (Sec. C.5), with no reported sweep on the GEOQA test split, this is the main reason I think the idea may work.
Critical weaknesses
-
Table 1's headline comparison varies two factors at once. On the server FGRPO applies an Adam step (Eqs. 16-18), whereas FedAvg is given as a data-size weighted average (App. B.2, p. 33); FedProx is cast as an extension of FedAvg, and I could not locate SCAFFOLD's server update. Table 3 indicates this matters, though Sec. C.4 never states which server update its "w/o RPG" arm runs. On Qwen2.5-3B that "w/o RPG" arm, using data-volume weights (Sec. C.4), already sits 2.0 points above FedAvg on Open-R1 (40.66 vs 38.64) and 3.1 on GEOQA (38.66 vs 35.59). Should that arm retain the Adam step, RPG explains just 1.2 of the 3.2-point Open-R1 gain and 2.1 of 5.2 on GEOQA. Should it instead be plain FedAvg, one method lands 2-3 points apart across Tables 1 and 3, exceeding most Table 1 margins. In either case the Table 1 margins over FedAvg are not attributable to RPG alone. FedAdam with weights and with uniform weights, run on the same local GRPO pipeline, would settle this.
-
The remaining support for RPG is thin, and part of it rests on numbers scored only on the test set. The data use an 80/20 split with no validation split mentioned (Sec. 4.1), and the default row of Tables 4-7 reproduces the Table 1 test accuracy exactly, so the RPG sweep was graded on the Open-R1 test split, where the default tops every table. Averaging the 12 non-default settings gives 41.05, about +0.4 over w/o RPG (40.66), while yields 40.54, below it. The headline +1.2 sits within one std (both ), and the run count is unstated. The theory offers little: (Eq. 74) enters Thm. A.1 purely as a penalty, up to roughly at the defaults under binary rewards ( is not reported), and p. 32 describes the theorem as quantifying "a possible cost of adaptive weighting". The abstract's claim that the analysis verifies efficacy should be dropped. The gradient interference invoked as motivation in Sec. 1 is likewise never measured; checking whether high-weight client updates align better with would be a start.
Questions for authors
-
Do the FedAvg, FedProx, SCAFFOLD and "w/o RPG" rows use plain weighted averaging or the Adam step of Eqs. 17-18? This governs how the margins in Tables 1 and 3 should be read.
-
How many runs sit behind each mean std, and do the w/ and w/o RPG arms share seeds and partitions? If they do, please give paired per-seed differences for Table 3. And how were the RPG defaults selected, given that only a train/test split is described?
-
Could you include FedAdam with weights and with uniform weights (Eq. 15 reduces to when all are equal) on at least one further backbone? If RPG exceeds both beyond seed variance, I would raise my score.
-
Sec. C.3 reports the GEOQA margin over the best baseline as larger at (4.14) than at (1.48), and attributes this to better generalization. How does that fit the claim that RPG targets non-IID heterogeneity?
We appreciate the reviewer's insightful comments. We address each of the main concerns below. We will revise our manuscript accordingly.
W1 and Q1. Server updates and the gain from RPG
In Table 1, FedAvg, FedProx and SCAFFOLD averaged client models by sample size. FedProx applied its penalty locally. SCAFFOLD updated its server control variate separately. FGRPO applied FedAdam to the RPG-weighted update in Eqs. (16)–(18). Both standard-GRPO arms in Table 3 shared local and FedAdam settings, including the server learning rate. The w/o RPG weights were wi=ni/∑jnj.
With data-size weights, FedAdam exceeded FedAvg by 2.02 percentage points (pp) on Open-R1 and 3.07 pp on GEOQA. With FedAdam fixed, RPG added 1.20 and 2.10 pp, respectively. The full gains over FedAvg were 3.22 and 5.17 pp. We will specify each method's server update.
In a separate 3B comparison, Open-R1 accuracy was 38.90% with uniform FedAdam (wi=1/N) and 40.10% with RPG. On GEOQA, the corresponding accuracies were 39.89% and 41.65%.
W2. Theory and client gradients
We will revise the abstract to attribute the accuracy gain to the matched FedAdam ablation. Theorem A.1 bounds the average expected squared gradient norm. Lemma A.7 bounds the change in population gradient direction using the weight differences ωi[t]−pi and the disagreement among client gradients. The change vanishes if the weights coincide or the gradients agree. By Eq. (74) and ωi[t]≤1, the ratio ωi[t]/pi is at most min(CRPG,1/pmin).
At rounds 3, 6, 9 and 12 on both datasets, we estimated five client GRPO gradients at the same global model. Each checkpoint had 3–7 negative pairwise cosines out of 10 (43/80 overall). Removing the highest-RPG update lowered global test accuracy at all eight checkpoints by 0.57–2.60 pp. Across clients, the Spearman correlation between RPG score and leave-one-client-out accuracy drop was positive at each checkpoint (median 0.85). With updates fixed, the original score assignment gave higher one-round test accuracy than the reversed assignment at seven of eight checkpoints (mean advantage 1.14 pp across all eight).
Q2. Paired runs and choice of defaults
Tables 1 and 3 used five runs per setting. Each seed pair in the standard-GRPO ablation shared its client partition, local settings and FedAdam hyperparameters. Both Open-R1 arms had a run-level SD of 1.21 pp.
Mean paired gains (± sample SD) were 1.20 ± 0.53 pp on Open-R1 and 2.11 ± 1.39 pp on GEOQA. The rounded GEOQA arm means in Table 3 differ by 2.10 pp.
We chose an EMA history coefficient of 0.8 to halve a past reward's influence in about three rounds. On the 0–1 reward scale, the 0.05 volatility floor limits amplification of small gains after a flat history. The 0.20 cap limits how much noise suppresses later scores. A one-unit score gap gives a weight ratio of 1.49 at temperature 2.5 and 1.95 at 1.5. At an annealing rate of 0.1, the temperature moves halfway between them in about seven rounds.
Tables 4–7 use the test portion of the 80/20 Open-R1 split. All four one-factor sweeps share the default result (41.86%). The twelve alternatives averaged 41.05%, 0.39 pp above data-size-weighted FedAdam (40.66%). Eleven exceeded that baseline. The exception was σmin=0.10 at 40.54%.
Q3. FedAdam on a second backbone
On Qwen2.5-VL-7B-Instruct, we compared data-size and RPG weights under FedAdam.
RPG improved accuracy in every seed pair. The paired gains were 2.04 ± 0.32 pp on Open-R1 and 1.65 ± 0.83 pp on GEOQA (mean ± sample SD).
In the revision, we will extend the uniform-weight FedAdam comparison from the 3B backbone (W1) to the 7B backbone.
Q4. Heterogeneity in Fig. 8
Even with balanced task proportions, clients train on different prompts and sample different responses, so their progress can differ. Eq. (15) gives equal weights for equal RPG scores. From strongly non-IID to uniform, FGRPO's lead over the best baseline grew from 1.48 to 4.14 pp on GEOQA and fell from 3.02 to 2.65 pp on Open-R1. We will clarify in Sec. C.3 that FGRPO ranked first at all three levels on both datasets.