acceptodds
Under review as a conference paper at ICLR 2027

FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Data

Abstract

Recent advances in language models have established reinforcement learning as the primary paradigm for eliciting self-correction and long-chain reasoning. While group relative policy optimization (GRPO) offers superior scalability by eliminating the critic network, deploying it on a central infrastructure entails collecting a large volume of data from distributed owners, which poses communication overhead and significant privacy risks. To address these concerns, we introduce federated GRPO (FGRPO), a framework designed to decentralize the fine-tuning of reasoning models across different data owners. To account for differences in local learning progress under heterogeneous client data, FGRPO incorporates an adaptive aggregation mechanism based on relative performance gain. By characterizing each client's improvement relative to its historical baseline, the framework dynamically prioritizes effective learning trajectories regardless of local data heterogeneity. Rigorous theoretical analysis and extensive experiments are conducted to verify the efficacy of FGRPO.Our code is available at https://anonymous.4open.science/r/FGRPO-9BF8.

Primary area:
reinforcement learning
Keywords:
federated learning, reinforcement learning with verifiable rewards,GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HZ7NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

A federated version of GRPO for RLVR fine-tuning of vision-language models under non-IID clients. Each client trains a LoRA adapter locally and uploads it plus one scalar “relative performance gain” (round-mean reward minus an EMA baseline over a clipped EMA of deviations, Eq. 14); the server maps the scalars to annealed softmax weights (Eq. 15), forms a weighted pseudo-gradient, and takes a FedAdam-style step (Eqs. 16-18). Theorem 3.1/A.1 gives an O(T^{-1/2}) + O(1) stationarity bound. Validated on four VLM backbones over Open-R1 and GEOQA, 5 clients at Dirichlet(0.05), against FedAvg/FedProx/SCAFFOLD, with RPG on/off and heterogeneity ablations.

Critical strengths
  1. The mechanism is lightweight. Per round each client transmits a single extra scalar (Eq. 14) that the server converts into softmax weights (Eq. 15). Because the rule never touches the local objective, it extends to GSPO and DAPO unchanged (Sec. C.4). Federated RLVR does need some such weighting rule, and a drop-in version would be easy for others to adopt if the gain is real.

  2. The only direct on/off test of RPG is Table 3, backed by the reward curves in Figs. 11-12, and on GEOQA the effect looks genuine: FGSPO rises from 33.80±0.5433.80\pm0.54 to 36.55±0.8436.55\pm0.84 and FDAPO from 39.45±1.8239.45\pm1.82 to 44.50±0.9644.50\pm0.96. Since the RPG settings were swept on Open-R1 alone (Sec. C.5), with no reported sweep on the GEOQA test split, this is the main reason I think the idea may work.

Critical weaknesses
  1. Table 1's headline comparison varies two factors at once. On the server FGRPO applies an Adam step (Eqs. 16-18), whereas FedAvg is given as a data-size weighted average (App. B.2, p. 33); FedProx is cast as an extension of FedAvg, and I could not locate SCAFFOLD's server update. Table 3 indicates this matters, though Sec. C.4 never states which server update its "w/o RPG" arm runs. On Qwen2.5-3B that "w/o RPG" arm, using data-volume weights (Sec. C.4), already sits 2.0 points above FedAvg on Open-R1 (40.66 vs 38.64) and 3.1 on GEOQA (38.66 vs 35.59). Should that arm retain the Adam step, RPG explains just 1.2 of the 3.2-point Open-R1 gain and 2.1 of 5.2 on GEOQA. Should it instead be plain FedAvg, one method lands 2-3 points apart across Tables 1 and 3, exceeding most Table 1 margins. In either case the Table 1 margins over FedAvg are not attributable to RPG alone. FedAdam with pip_i weights and with uniform 1/N1/N weights, run on the same local GRPO pipeline, would settle this.

  2. The remaining support for RPG is thin, and part of it rests on numbers scored only on the test set. The data use an 80/20 split with no validation split mentioned (Sec. 4.1), and the default row of Tables 4-7 reproduces the Table 1 test accuracy exactly, so the RPG sweep was graded on the Open-R1 test split, where the default tops every table. Averaging the 12 non-default settings gives 41.05, about +0.4 over w/o RPG (40.66), while σmin⁡=0.10\sigma_{\min}=0.10 yields 40.54, below it. The headline +1.2 sits within one std (both ±1.21\pm1.21), and the run count is unstated. The theory offers little: CRPGC_{\rm RPG} (Eq. 74) enters Thm. A.1 purely as a penalty, up to roughly e26.7/(Npmin⁡)e^{26.7}/(Np_{\min}) at the defaults under binary rewards (ϵrpg\epsilon_{\rm rpg} is not reported), and p. 32 describes the theorem as quantifying "a possible cost of adaptive weighting". The abstract's claim that the analysis verifies efficacy should be dropped. The gradient interference invoked as motivation in Sec. 1 is likewise never measured; checking whether high-weight client updates align better with Δ[t]\Delta^{[t]} would be a start.

Questions for authors
  1. Do the FedAvg, FedProx, SCAFFOLD and "w/o RPG" rows use plain weighted averaging or the Adam step of Eqs. 17-18? This governs how the margins in Tables 1 and 3 should be read.

  2. How many runs sit behind each mean ±\pm std, and do the w/ and w/o RPG arms share seeds and partitions? If they do, please give paired per-seed differences for Table 3. And how were the RPG defaults selected, given that only a train/test split is described?

  3. Could you include FedAdam with pip_i weights and with uniform 1/N1/N weights (Eq. 15 reduces to 1/N1/N when all hih_i are equal) on at least one further backbone? If RPG exceeds both beyond seed variance, I would raise my score.

  4. Sec. C.3 reports the GEOQA margin over the best baseline as larger at μ=∞\mu=\infty (4.14) than at μ=0.05\mu=0.05 (1.48), and attributes this to better generalization. How does that fit the claim that RPG targets non-IID heterogeneity?

1 comment

We appreciate the reviewer's insightful comments. We address each of the main concerns below. We will revise our manuscript accordingly.

W1 and Q1. Server updates and the gain from RPG

In Table 1, FedAvg, FedProx and SCAFFOLD averaged client models by sample size. FedProx applied its penalty locally. SCAFFOLD updated its server control variate separately. FGRPO applied FedAdam to the RPG-weighted update in Eqs. (16)–(18). Both standard-GRPO arms in Table 3 shared local and FedAdam settings, including the server learning rate. The w/o RPG weights were wi=ni/∑jnjw_i=n_i/\sum_j n_j.

Qwen2.5-VL-3B-InstructOpen-R1 accuracy (%)GEOQA accuracy (%)
FedAvg, data-size weights38.6435.59
FedAdam, data-size weights40.6638.66
FedAdam, RPG weights41.8640.76

With data-size weights, FedAdam exceeded FedAvg by 2.02 percentage points (pp) on Open-R1 and 3.07 pp on GEOQA. With FedAdam fixed, RPG added 1.20 and 2.10 pp, respectively. The full gains over FedAvg were 3.22 and 5.17 pp. We will specify each method's server update.

In a separate 3B comparison, Open-R1 accuracy was 38.90% with uniform FedAdam (wi=1/Nw_i=1/N) and 40.10% with RPG. On GEOQA, the corresponding accuracies were 39.89% and 41.65%.

W2. Theory and client gradients

We will revise the abstract to attribute the accuracy gain to the matched FedAdam ablation. Theorem A.1 bounds the average expected squared gradient norm. Lemma A.7 bounds the change in population gradient direction using the weight differences ωi[t]−pi\omega_i^{[t]}-p_i and the disagreement among client gradients. The change vanishes if the weights coincide or the gradients agree. By Eq. (74) and ωi[t]≤1\omega_i^{[t]}\le 1, the ratio ωi[t]/pi\omega_i^{[t]}/p_i is at most min⁡(CRPG,1/pmin⁡)\min(C_{\rm RPG},1/p_{\min}).

At rounds 3, 6, 9 and 12 on both datasets, we estimated five client GRPO gradients at the same global model. Each checkpoint had 3–7 negative pairwise cosines out of 10 (43/80 overall). Removing the highest-RPG update lowered global test accuracy at all eight checkpoints by 0.57–2.60 pp. Across clients, the Spearman correlation between RPG score and leave-one-client-out accuracy drop was positive at each checkpoint (median 0.85). With updates fixed, the original score assignment gave higher one-round test accuracy than the reversed assignment at seven of eight checkpoints (mean advantage 1.14 pp across all eight).

Q2. Paired runs and choice of defaults

Tables 1 and 3 used five runs per setting. Each seed pair in the standard-GRPO ablation shared its client partition, local settings and FedAdam hyperparameters. Both Open-R1 arms had a run-level SD of 1.21 pp.

RunOpen-R1 w/o → w/ RPG (%)Gain (pp)GEOQA w/o → w/ RPG (%)Gain (pp)
139.10 → 40.00+0.9036.00 → 39.46+3.46
240.10 → 42.10+2.0037.41 → 41.02+3.61
340.70 → 41.50+0.8039.96 → 41.23+1.27
441.00 → 42.50+1.5040.45 → 40.88+0.43
542.40 → 43.20+0.8039.46 → 41.23+1.77

Mean paired gains (± sample SD) were 1.20 ± 0.53 pp on Open-R1 and 2.11 ± 1.39 pp on GEOQA. The rounded GEOQA arm means in Table 3 differ by 2.10 pp.

We chose an EMA history coefficient of 0.8 to halve a past reward's influence in about three rounds. On the 0–1 reward scale, the 0.05 volatility floor limits amplification of small gains after a flat history. The 0.20 cap limits how much noise suppresses later scores. A one-unit score gap gives a weight ratio of 1.49 at temperature 2.5 and 1.95 at 1.5. At an annealing rate of 0.1, the temperature moves halfway between them in about seven rounds.

Tables 4–7 use the test portion of the 80/20 Open-R1 split. All four one-factor sweeps share the default result (41.86%). The twelve alternatives averaged 41.05%, 0.39 pp above data-size-weighted FedAdam (40.66%). Eleven exceeded that baseline. The exception was σmin⁡=0.10\sigma_{\min}=0.10 at 40.54%.

Q3. FedAdam on a second backbone

On Qwen2.5-VL-7B-Instruct, we compared data-size and RPG weights under FedAdam.

DatasetData-size-weighted FedAdam accuracyRPG accuracy
Open-R145.64%47.68%
GEOQA49.70%51.36%

RPG improved accuracy in every seed pair. The paired gains were 2.04 ± 0.32 pp on Open-R1 and 1.65 ± 0.83 pp on GEOQA (mean ± sample SD).

In the revision, we will extend the uniform-weight FedAdam comparison from the 3B backbone (W1) to the 7B backbone.

Q4. Heterogeneity in Fig. 8

Even with balanced task proportions, clients train on different prompts and sample different responses, so their progress can differ. Eq. (15) gives equal weights for equal RPG scores. From strongly non-IID to uniform, FGRPO's lead over the best baseline grew from 1.48 to 4.14 pp on GEOQA and fell from 3.02 to 2.65 pp on Open-R1. We will clarify in Sec. C.3 that FGRPO ranked first at all three levels on both datasets.

Reviewer_jipyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FGRPO uses GRPO to fine tune reasoning models across clients with distinct tasks distributions. Each client reports relative performance gain, which is the reward improvement over a moving average. The final performance outperforms FedAvg, FedProx, and SCAFFOLD.

Critical strengths

This paper identifies an aggregation problem in conventional federated GRPO. In conventional federated GRPO, comparisons of answer rewards within each prompt make clients' learning signals hard to compare across task difficulties. Then, the proposed weighting rule adds only a scalar per client per round and leaves local GRPO training unchanged. So, it could fit existing federated pipelines with little overhead if validated. The experiments are generally solid and support the claimed argument.

Critical weaknesses
  1. It is appreciated that the authors provide the code for reproducibility and verification. I have double checked the code. However, it seems that in the released code, the FedProx and SCAFFOLD corrections do not appear to take effect, so Table 1's three baselines may all reduce to FedAvg. Both released plugins suppress a failed trainer import and use parameter names absent from saved adapters. Also, the released FGRPO script in the code computes RPG differently from Eqs. (13) to (15), so the experiments may test a rule that differs from the one that is defined and analyzed in the paper. It would be great if the authors could provide some clarifications. Otherwise, there may be some misalignment between the paper descriptions and actual implementation.
  2. There may be some claim-evidence misalignment. Some parts may have logic issues. For example, Section 2.2 argues that "the same normalized improvement can correspond to very different absolute progress across clients". Since RPG is itself a normalized improvement score (see Eq. 14), so RPG may have the same problem. The paper does not show which clients receive greater weights. For Qwen2.5-3B, FGRPO's margin over the best baseline with identical task distributions is comparable to or greater than that with strongly differing distributions (see Fig. 8). So the benefit seems unrelated to differing distributions.
  3. The math part may also have some issues. For example, the authors describes Theorem 3.1 as a convergence guarantee. However, its bound follows from the assumptions alone, so it appears to give no evidence that FGRPO converges or that RPG helps. In more details, Lemma A.2 (see Eq. 36) bounds every client's gradient norm by a constant, and the constant remainder in Eq. (102) is at least eight times the square of that constant, so Eq. (24) holds even if the parameters are never updated. In addition, Appendix A.2 itself says the theorem "quantifies a possible cost of adaptive weighting". It would be great if the authors could provide some clarifications.
  4. The authors still compare their method with old baselines from 2017-2020. Comparison with recent baselines such as pooled training, separate client training, and weighting for different client task mixes is missing. So, the SOTA seems over-claimed. For example, this paper reports neither centralized GRPO on pooled data nor local-only training. FedDisco [2] is also not discussed. It adjusts weights when clients' category proportions differ as in Section 4.1. Federated GRPO with accuracy-based weighting [3] is also not discussed.

[1] Rui Ye et al. OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning. KDD 2024. DOI 10.1145/3637528.3671582. https://doi.org/10.1145/3637528.3671582

[2] Rui Ye et al. FedDisco: Federated Learning with Discrepancy-Aware Collaboration. ICML 2023. arXiv:2305.19229. https://arxiv.org/abs/2305.19229

[3] Ziyao Wang et al. FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards. arXiv preprint 2026. arXiv:2602.00453. https://arxiv.org/abs/2602.00453

Questions for authors
  1. Do the Table 3 variants without RPG keep FGRPO's Adam-style server update with the same learning rate? How does FedAdam weighted by client data sizes perform on all four backbones?

  2. How do centralized GRPO on the pooled training data and local-only GRPO perform on both datasets?

  3. Under which additional conditions could the constant remainder in Theorem A.1 fall below the bound implied by Lemma A.2?

1 comment

We appreciate the reviewer's insightful comments. We address each of the main concerns below. We will revise our manuscript accordingly.

W1. Released plugins and RPG implementation

Anonymizing paths disrupted plugin loading. A failed trainer import can disable a correction, and unmatched adapter keys are skipped. In the Table 1 runs, FedProx applied its proximal penalty to trainable adapters and SCAFFOLD corrected their gradients with its control variate before each local step.

Eq. (13) reverses the EMA coefficients. The code uses 0.8 on the previous baseline and 0.2 on the current estimate, averaging logged mean rewards over the final window. The paper uses the full round.

Eq. (14) and the code both subtract the previous baseline. The code divides by σi[t−1]+εrpg\sigma_i^{[t-1]}+\varepsilon_{\mathrm{rpg}} before updating and clipping volatility. Algorithm 1 updates it first. Initialization and clipping preserve the σmin⁡\sigma_{\min} floor used in Lemma A.5, so Eqs. (73)–(74) still hold for Theorem 3.1.

Eq. (15) gives the standard temperature-scaled softmax over RPG scores. Two branches left in the anonymous release played no practical role in the reported experiments. One applies softmax to client reward estimates at warm start. The other uses uniform weights if all RPG scores are below −1-1. We will revise the manuscript and clean up the code.

W2. Which clients receive greater weight

GRPO compares responses to the same prompt. Clients with identical group advantages and mean reward 0.5 can have historical baselines of 0.3 and 0.7. Their RPG scores then have opposite signs, so Eq. (15) weights the first more. Equal RPG scores can represent different absolute gains when volatility differs.

Removing the highest-RPG update lowered test accuracy by 0.57–2.60 percentage points (pp) at rounds 3, 6, 9 and 12 on both datasets. With updates fixed, reversing the scores lowered accuracy at seven of eight checkpoints (mean drop 1.14 pp across all eight). We will clarify Sec. 2.2 and report per-client weight trajectories.

Clients can improve at different rates on different prompts even with balanced task proportions. From strongly non-IID to uniform (Fig. 8), FGRPO's lead changes from 3.02 to 2.65 pp on Open-R1 and from 1.48 to 4.14 pp on GEOQA, where the best baseline switches from FedProx to SCAFFOLD.

W3 and Q3. Frozen model and residual bound

Let M=ℓmax⁡rmax⁡∗S2M=\ell_{\max}r_{\max}^*S_2. Eq. (36) bounds each client gradient norm by MM, so ∥∇J(θ)∥22≤M2\Vert\nabla\mathcal J(\theta)\Vert_2^2\le M^2 even at initialization. Since Cres≥8M2C_{\rm res}\ge 8M^2 in Eq. (102), a frozen model satisfies Eq. (24). Eq. (64) bounds bias and noise by a common worst case.

Eq. (125) bounds the average expected squared gradient norm by an O(T−1/2)O(T^{-1/2}) term plus a refined residual. At the global model, Eq. (82) gives the change from reweighting as ∑i(ωi−pi)(∇Ji−∇J)\sum_i(\omega_i-p_i)(\nabla\mathcal J_i-\nabla\mathcal J). Lemma A.7 bounds this change. Table 3 measures the accuracy gain (Q1).

With a=8(εs+G∞∗/Lˉ)2/[εs2(1−β1)]a=8(\varepsilon_s+G_\infty^*/\bar L)^2/[\varepsilon_s^2(1-\beta_1)], Eq. (118) gives a refined residual below M2M^2 if

(CRPG−1)κvar2+δG(T)2+CRPGνG2(T)/E<M2/a.(C_{\rm RPG}-1)\kappa_{\rm var}^2+\delta_G(T)^2+C_{\rm RPG}\nu_G^2(T)/E < M^2/a.

For one update per fresh block (U=1U=1), suppose the first two terms sum to at most M2/a−mM^2/a-m with m>0m>0. Eq. (65) gives νG2(T)≤G2(T)2/B\nu_G^2(T)\le G_2(T)^2/B, so BE>CRPGG2(T)2/mBE>C_{\rm RPG}G_2(T)^2/m suffices. BB counts independent prompt groups per block and EE fresh blocks per client per round.

W4 and Q2. Baselines on Qwen2.5-VL-3B-Instruct

FedDisco combines data size with differences in local and target category proportions. FedMOA uses learned accuracy-objective weights within task groups. FedDisco · FedMOA

MethodOpen-R1 accuracy (%)GEOQA accuracy (%)
FedAvg38.6435.59
FedProx36.1239.28
SCAFFOLD38.8438.08
FedDisco33.4035.79
FedMOA31.9038.33
FGRPO41.8640.76
Centralized GRPO37.135.7
Local-only GRPO31.9635.60

FGRPO ranked first among the six federated methods. Pooled GRPO, local-only GRPO and FGRPO used five GPUs and 5B5B prompts per cycle (BB per client or 5B5B pooled). We matched cycle count, responses per prompt, initialization, LoRA, reward, rollout reuse and length limits. We scored all models on the same test set and averaged the five local-only accuracies equally.

Q1. FedAdam across backbones

The standard-GRPO arms in Table 3 shared all FedAdam settings, including the server learning rate.

Qwen2.5-VL (baseline → RPG)Open-R1 (%)GEOQA (%)
3B / data size40.66 → 41.8638.66 → 40.76
7B / data size45.64 → 47.6849.70 → 51.36
3B / uniform38.90 → 40.1039.89 → 41.65

We will extend the data-size comparison to Qwen3-VL-4B-Instruct and Llama-3.2-11B-Vision-Instruct.

Reviewer_xPSeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FGRPO, a GRPO framework for an FL environment with non-IID data clients. To use the GRPO framework in FL settings, the authors introduce RPG, which measures each client's relative reward improvement to the global server by checking its historical EMA and normalizing by local reward fluctuations. Based on this, the server judges the adaptive weight of the client update. Experiments on four models and two multimodal reasoning benchmarks show consistent performance gains.

Critical strengths

S1. The proposed FGRPO's RPG-based aggregation method is simple, practical, and applicable to FL settings without requiring any additional external models or modules.

S2. The experimental results show consistent performance gains across four models and two multimodal reasoning benchmarks.

Critical weaknesses

W1. The justification for the proposed RPG aggregation method is limited and not sufficiently verified. Although there is an ablation without RPG, the main comparisons use different server-side optimization mechanisms. It does not clearly distinguish how effective RPG weighting is compared with alternatives under matched settings.

W2. For the overall experiments, this work mainly uses synthetically partitioned datasets and reports the aggregate accuracy and the reward trajectories. There is no direct measurement of client-level gradient conflicts or an examination of whether higher RPG clients are really beneficial to the global objective.

Questions for authors

Q1. Why did the authors use VLMs rather than LLMs for the overall experiments in this paper? It seems FGRPO does not specifically target the VLM framework.

Q2. How many runs were conducted for this work? The main table includes standard deviation statistics, but it does not specify the number of runs.

2 comments

We appreciate the reviewer's insightful comments. We address each of the main concerns below. We will revise our manuscript accordingly.

W1. RPG weighting under a matched server update

Table 1 compares complete methods with different server updates. Our rationale for RPG is that client data size stays fixed while learning progress can vary across rounds. RPG measures each client's reward improvement relative to its own historical baseline and scales it by local reward volatility.

The standard-GRPO rows of Table 3 isolate the effect of weighting. On Qwen2.5-VL-3B-Instruct, both arms used the same local GRPO procedure and the same FedAdam server optimizer, learning rate, and other hyperparameters. Only the client weights in Eq. (16) changed. Replacing data-size weights with RPG weights raised mean total test accuracy from 40.66% to 41.86% on Open-R1 (+1.20 percentage points, pp) and from 38.66% to 40.76% on GEOQA (+2.10 pp).

We also compared RPG with uniform FedAdam (wi=1/Nw_i=1/N) in a separate evaluation on the same 3B backbone.

DatasetUniform FedAdam accuracyRPG accuracy
Open-R138.90%40.10%
GEOQA39.89%41.65%

W2. Client gradient disagreement and the contribution of high-RPG clients

We examined rounds 3, 6, 9, and 12 on each benchmark under the synthetic five-client partitions.

Gradient disagreement. At each checkpoint, we estimated the five clients' local GRPO gradients at the same global model and computed all ten pairwise cosine similarities. Although the checkpoint means ranged from −0.0217 to +0.0198, three to seven pairs had negative cosine similarity at every checkpoint, totaling 43 of 80 pairs (53.75%).

Client removal. We removed each client's update in turn and reaggregated the other four. We measured the accuracy drop as global test accuracy with all five updates minus accuracy after removal. Removing the highest-RPG client lowered accuracy at all eight checkpoints, by 0.57–2.60 pp. Across the five clients at each checkpoint, the Spearman correlation between RPG score and accuracy drop ranged from 0.40 to 0.90 (median 0.85).

Score reversal. We kept the five updates fixed and reversed the assignment of RPG scores across clients. The original assignment gave higher one-round global test accuracy at seven of eight checkpoints, with a mean advantage of 1.14 pp across all eight. The largest gain was on Open-R1 at round 6 (+3.30 pp). The exception was GEOQA at round 6 (−0.14 pp).

DatasetRoundMean cosineNegative pairsDrop after removing top RPG client (pp)Spearman ρ\rhoRPG accuracy (%)Reversed accuracy (%)
Open-R13−0.01545/101.800.6032.6031.40
Open-R16−0.02176/102.600.8038.7035.40
Open-R19+0.01985/101.500.4040.5039.40
Open-R112−0.00416/102.100.9041.9040.30
GEOQA3+0.00035/101.840.8030.9129.99
GEOQA6−0.00736/100.710.9035.2935.43
GEOQA9−0.02067/100.570.9039.5338.68
GEOQA12+0.01073/101.770.9041.6541.37

Q1. Choice of VLMs

We chose VLMs because both benchmarks require visual information. Multimodal-Open-R1-8k-Verified includes questions grounded in charts and plots, and GEOQA-8k pairs geometry diagrams with textual questions. The RPG rule uses only a client's current reward estimate and its reward history to weight its update. The same rule can therefore be used with text-only LLMs trained with GRPO and verifiable rewards. Our experiments cover four VLM backbones.

Q2. Number of runs

We ran each Table 1 setting five times.

Thank you for the additional experiments and analyses. While the new results address some of my concerns, I believe some of my broader concerns remain. Therefore, I will maintain my original assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.