acceptodds
Under review as a conference paper at ICLR 2027

Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation

Abstract

Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.

Primary area:
reinforcement learning
Keywords:
Group-Relative Policy Optimization, Robust Policy Optimization, Large Language Model Post-Training, Robust Advantage Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FjsROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a dual-channel optimizer, RoVR-GSPO, to mitigate the interference caused by potential reward perturbations in GSPO. The authors evaluate the proposed method on two models and demonstrate its effectiveness through experimental results.

Critical strengths
  1. This paper investigates and analyzes the issues arising from reward perturbations in GSPO, providing insights that help researchers better understand and apply the algorithm.
Critical weaknesses
  1. The experimental results are not entirely convincing. As shown in Table 1, most performance gains over GSPO are less than two points. Such marginal improvements may result from hyperparameter tuning or random variation, making it difficult to establish the effectiveness of the proposed approach without additional evidence, such as results across multiple runs and statistical significance tests.

  2. As shown in Table 2, the performance gains decrease as the model size increases from 4B to 7B. This trend raises concerns that the benefits of the proposed approach may diminish further on larger models, potentially limiting its practical value and scalability.

  3. To better establish the effectiveness and generalizability of the proposed algorithmic improvement, the authors should evaluate it on a broader range of tasks and across models of different sizes and model families. Such experiments would help determine whether the observed benefits consistently extend beyond the current experimental settings.

Questions for authors
  1. There is a missing citation on line 371.
2 comments

Thank you very much for your suggestions. We have carefully analyzed your comments, organized them and supplemented the corresponding experiments, hoping to address your concerns. We summarize the issues you raised as follows:

1. Statistical Significance, Multi-Seed Evidence, and Hyperparameter Neutrality

1.1 Zero hyperparameter tuning and universal fixed defaults

  1. Zero RL retuning: All training budgets and optimizer settings (LR 10−610^{-6}, batch 192/24, G=16G=16, clip [0.9997,1.0004][0.9997,1.0004], length 8,1928{,}192, 500 steps) are inherited verbatim from the parent GSPO baseline.
  2. One universal configuration across all settings: All RoVR-GSPO parameters (c=1.0, κ=1.0\kappa=1.0, K=9K=9, Bratio=8B_{\mathrm{ratio}}=8, γ=η=0.01\gamma=\eta=0.01) were fixed a priori from theoretical efficiency (V9=1.0691V_9=1.0691, within 2.1%2.1\% of π/3\pi/3, Table 8) and natural scales, and applied identically without per-task, per-model, or per-reward tuning. Sweeps over κ∈[0.5,2.0]\kappa\in[0.5,2.0], c∈[0.5,2.0]c\in[0.5,2.0], Bratio∈{4,8,16}B_{\mathrm{ratio}}\in\{4,8,16\}, and γ∈[0.002,0.05]\gamma\in[0.002,0.05] confirm every tested variant beats GSPO (+1.65%–+2.08% on Rule; +3.21%–+3.97% on RM).

1.2 Multi-run expansion (n=5n=5 seeds) and formal significance tests

In Table 1 (3 runs), RoVR-GSPO achieves 20/20 consistent wins, cutting residual error by 9.1% relatively even on near-saturated OLMo-3-7B MATH500 (89.80%→\to90.73%) while delivering +3.55%–+7.50% absolute gains on harder OOD and RM benchmarks. Expanding all four (backbone × reward) pairs to n = 5 independent seeds (Table R1), we compute two-sided Welch's tt-tests across independent runs and paired stratified bootstrap 95% CIs (10,000 replicates) across all 1,227 test problems.

Table R1: Extended n = 5 independent-seed evaluation and significance tests.

Model & FamilyRewardGSPO (%, n=5n=5)RoVR-GSPO (%, n=5n=5)Gain Δ\DeltaPaired 95% CIWelch's tt (pp-value)
Qwen3-4B-BaseRule42.65±0.5944.82±0.68+2.17[+1.42, +2.91]t=4.81t=4.81 (p=0.0014p=0.0014)
Qwen3-4B-BaseRM38.82±0.5742.92±0.76+4.11[+3.18, +5.04]t=8.63t=8.63 (p=4.0×10−5p=4.0\times 10^{-5})
OLMo-3-7B-InstRule63.36±0.3864.64±0.36+1.28[+0.74, +1.83]t=4.89t=4.89 (p=0.0012p=0.0012)
OLMo-3-7B-InstRM62.77±0.4164.74±0.44+1.97[+1.31, +2.64]t=6.52t=6.52 (p=1.9×10−4p=1.9\times 10^{-4})

All four gains are highly statistically significant (p≤0.0014p\le 0.0014) with 95% CIs strictly above zero.

2. Controlled Scaling Study: Strong and Growing Gains on Larger Models

In Table 2, comparing Qwen3-4B-Base (GSPO R-1=0.309) with OLMo-3-7B-Instruct (GSPO R-1=0.552) simultaneously changed model family, size, and alignment stage (Base vs. SFT-aligned Instruct near the ~0.56 GovReport R-1 ceiling). Holding family and Base initialization fixed across Qwen3-4B-Base, 8B-Base, 14B-Base, and 30B-A3B-Base (MoE) over n=3 seeds (Tables R2–R3) isolates true scaling:

Table R2: Multi-seed (n = 3) GovReport scaling study (ROUGE-1 / ROUGE-2 / ROUGE-L, mean ±\pm std).

ModelStage & ScaleGSPO (R-1 / R-2 / R-L)RoVR-GSPO (R-1 / R-2 / R-L)Gain Δ\Delta (R-1 / R-2 / R-L)Welch pp (R-2 / R-L)
Qwen3-4B-BaseBase (4B).309±.002 / .142±.002 / .166±.002.333±.003 / .164±.002 / .175±.002+.024 / +.021 / +.0090.0005 / 0.0084
Qwen3-8B-BaseBase (8B).384±.002 / .168±.002 / .194±.002.409±.002 / .188±.002 / .206±.002+.025 / +.020 / +.0120.0006 / 0.0032
Qwen3-14B-BaseBase (14.8B).442±.002 / .186±.002 / .215±.002.465±.002 / .205±.002 / .228±.002+.023 / +.019 / +.0130.0008 / 0.0013
OLMo-3-7B-InstInstruct (7B).552±.001 / .189±.001 / .221±.001.553±.001 / .195±.001 / .229±.001+.001 / +.006 / +.0080.0086 / 0.0030
Llama-3.1-8B-InstInstruct (8B).536±.002 / .182±.002 / .213±.002.544±.002 / .191±.002 / .224±.002+.008 / +.009 / +.0110.0041 / 0.0025
  • Sustained summarization gains across scales: Across 4B→\to8B→\to14B, ROUGE gains hold steady (+.023–.025 R-1, +.019–.021 R-2) while R-L gain increases from +.009 to +.013 (p≤0.0084p\le 0.0084); Instruct gains are also significant (p<0.009p<0.009).
  • Growing relative error reduction up to 30.5B MoE: On Math under RM (Table R3), 5-benchmark gains remain +3.97% (4B), +3.68% (8B), +3.06% (14B), and +4.33% (30.5B MoE)—a monotonically increasing relative error reduction from 6.5% (4B) →\to 7.1% (8B) →\to 7.2% (14B) →\to 11.2% (30.5B MoE).

3. Generalizability Across Model Families, Sizes, and Broader Task Domains

To establish broad algorithmic generality, we evaluate RoVR-GSPO across three distinct model families (Qwen3, OLMo-3, Llama-3.1), five model scales and architectures (4B, 7B, 8B, 14.8B Dense, and 30.5B-A3B MoE), and five diverse post-training domains: Mathematical Reasoning (Table R3), Long-Context Summarization (Table R2), Execution-Verified Code Generation, Single-Turn Function Calling, and Multi-Turn Interactive Agents (Table R4)—all using the exact same default RoVR-GSPO parameters.

Table R3: Cross-scale and cross-family mathematical reasoning (%, 3-run means; Rel. Error Red. denotes five-benchmark relative error reduction).

Family & ModelParamsRewardGSPO: MATH500 / AIME25 / AMC23RoVR-GSPO: MATH500 / AIME25 / AMC23Rel. Error Red.
Qwen3-4B-Base4BRule77.00 / 14.44 / 62.5077.47 / 15.56 / 63.333.6%
Qwen3-8B-Base8BRule82.20 / 22.22 / 70.8383.00 / 25.56 / 74.175.6%
Qwen3-30B-A3B (MoE)30.5BRule88.27 / 38.89 / 84.1789.33 / 44.44 / 88.338.4%
Qwen3-4B-Base4BRM74.53 / 8.89 / 56.6775.80 / 15.56 / 64.176.5%
Qwen3-8B-Base8BRM80.40 / 18.89 / 66.6781.80 / 24.44 / 72.507.1%
Qwen3-14B-Base14.8BRM84.80 / 30.00 / 77.5086.20 / 34.44 / 81.677.2%
Qwen3-30B-A3B (MoE)30.5BRM87.00 / 35.56 / 81.6788.73 / 43.33 / 87.5011.2%
OLMo-3-7B-Instruct7BRule89.80 / 37.78 / 86.6790.73 / 41.11 / 87.503.5%
OLMo-3-7B-Instruct7BRM88.93 / 37.78 / 86.6789.53 / 41.11 / 90.835.3%
Llama-3.1-8B-Instruct8BRule54.13 / 8.89 / 44.1755.60 / 12.22 / 48.334.1%
Llama-3.1-8B-Instruct8BRM51.73 / 7.78 / 40.0054.27 / 11.11 / 46.675.4%

For broader task domains (Table R4):

  1. Execution-verified coding (HumanEval+, MBPP+, LiveCodeBench v5): Trained with sandbox unit-test pass-rate rewards and syntax/timeout penalties, where RoVR-GSPO protects partial-pass credit contrast and stabilizes branching-token bursts.
  2. Function calling & multi-turn interactive agents: Evaluated on full 500-step ToolRL (RLLA-4K), zero-shot BFCL-v3 AST accuracy, 8-turn Search-R1 retrieval (HotpotQA EM), and ReTool Python execution (AIME2024), where dual-channel estimation handles both tool-parse cliff penalties and post-observation log-ratio spikes.

Table R4: Generalization to Execution-Verified Coding (Pass@1 %) and Agentic Tasks.

Family & BackboneMethodCoding: HumanEval+ / MBPP+ / LiveCodeBench (%)Tool-Call: ToolRL (500-step) / BFCL-v3 (%)Interactive Agent: Search-R1 HotpotQA / ReTool AIME24 (%)
Qwen3-4B-BaseGSPO70.12 / 67.20 / 26.162.294 / 75.6043.20 / 38.89
(4B Dense)RoVR-GSPO71.95 / 68.78 / 28.672.478 / 78.1045.80 / 42.22
Llama-3.1-8B-InstructGSPO67.68 / 65.08 / 21.862.215 / 73.4041.80 / 23.33
(8B Meta Family)RoVR-GSPO70.12 / 66.93 / 24.372.406 / 76.2044.60 / 26.67
Qwen3-14B-BaseGSPO79.27 / 75.13 / 39.782.615 / 83.7051.00 / 52.22
(14.8B Dense)RoVR-GSPO81.10 / 76.72 / 42.292.784 / 85.9053.60 / 55.56
Qwen3-30B-A3B-BaseGSPO81.71 / 76.98 / 43.37——
(30.5B MoE)RoVR-GSPO83.54 / 78.57 / 46.24——

Across all three model families, all scales (4B to 30.5B MoE), and all five task domains, RoVR-GSPO consistently outperforms GSPO—achieving +1.97% to +2.27% average gains on Coding (+2.51% to +2.87% on LiveCodeBench v5), +2.20% to +2.80% on BFCL-v3, and +2.60% to +3.33% on multi-turn Search-R1 and ReTool agents.


4. Response to Question

Thank you for catching this. The missing BibTeX entry qian2025toolrl (Qian et al., "ToolRL: Reward is All Tool Learning Needs," arXiv:2504.13958, 2025) has been restored, and we have verified clean compilation across the manuscript.

Reviewer_bjnVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a GRPO variant, RoVR-GSPO, a dual-channel robust optimizer that addresses sensitivity to reward outliers and token-level log-ratio perturbations. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. The authors provide stability and efficiency analyses, alongside experiments on mathematical reasoning, long-context summarization, and tool-call annotation showing improvements over GSPO.

Critical strengths
  1. Clear motivation for addressing two sources of instability.
Critical weaknesses
  1. The experiments are weak. The paper only conducts experiments on models that less than 7B and tasks only in math, which is much weaker than other published papers. More scenarios such as coding, agentic tasks should be included.

  2. Missing experiments on MoE setting, such as in the original GSPO paper.

  3. Missing comparison with other strong published baseline methods, such as Original GRPO,DAPO,Dr.GRPO,GMPO,GPG, CISPO, SPO.

Questions for authors

See weaknesses

2 comments

Thank you very much for your support and valuable comments. We have carefully reviewed them, which are very helpful. We summarize the key concerns you raised, supplemented the corresponding experiments, and provide our responses as follows.

Beyond our evaluations on OLMo-3-7B-Instruct (7B), GovReport (Table 2), and ToolRL (Sec. 5.2), we demonstrate RoVR-GSPO's superiority across 14.8B dense and 30.5B MoE backbones, execution-verified coding, multi-turn interactive agents, and all seven recent RL baselines.

1. Scaling Beyond 7B and Expanding to Coding & Agentic Scenarios

1.1 Scaling to 14.8B Dense (Qwen3-14B-Base)

With our universal fixed parameters (c=κ=1,K=9,Br=8,γ=η=0.01,G=16c=\kappa=1,K=9,B_r=8,\gamma=\eta=0.01,G=16, length 8,1928{,}192), Table R1 evaluates Qwen3-14B-Base (30.5B MoE in Sec. 2). RoVR-GSPO leads every benchmark, improving 5-benchmark accuracy by +3.06% over GSPO, +2.38% over GMPO, and +4.37% over GRPO (+4.44% on AIME25).

Table R1: 14.8B dense scaling (Qwen3-14B-Base, InternLM2-RM, %, mean ±\pm std over 3 runs).

MethodMATH500Gaokao23enMinervaMathAIME2025AMC23
Original GRPO84.00±0.3362.77±0.5330.76±0.4627.78±1.5775.83±1.18
DAPO84.40±0.3363.38±0.4231.13±0.4628.89±1.5776.67±1.18
GSPO84.80±0.3363.90±0.4231.50±0.4630.00±2.7277.50±2.04
GMPO85.13±0.2564.68±0.4231.86±0.4631.11±1.5778.33±1.18
RoVR-GSPO86.20±0.3367.10±0.5333.58±0.4634.44±1.5781.67±1.18

1.2 Execution-Verified Coding (HumanEval+, MBPP+, LiveCodeBench v5)

Trained with unit-test pass-rate rewards (Prime-RL / TACO, Ri∈[0,1]R_i\in[0,1] plus -1.0 syntax/timeout cliffs), RoVR-GSPO preserves partial-pass contrast against -1.0 cliffs (Prop. 2.1) and stabilizes branching-token bursts across 4B, 14.8B, and 30.5B MoE (Table R2).

Table R2: Code generation Pass@1 (%) across dense and MoE scales.

BackboneMethodHumanEval+MBPP+LiveCodeBench (v5)
Qwen3-4B (Dense)Original GRPO68.9066.1424.37
DAPO69.5166.6725.45
GSPO70.1267.2026.16
GMPO70.7367.4626.52
RoVR-GSPO71.9568.7828.67
Qwen3-14B (Dense)Original GRPO78.0574.3438.35
GSPO79.2775.1339.78
RoVR-GSPO81.1076.7242.29
Qwen3-30B-A3B (MoE)Original GRPO (no Replay)77.4473.8137.28
GSPO81.7176.9843.37
RoVR-GSPO83.5478.5746.24

1.3 Function Calling and Multi-Turn Interactive Agents

We evaluate 500-step ToolRL (RLLA-4K), zero-shot BFCL-v3 AST accuracy (protecting [+1.5,+4.0] partial matches from -3.0 parse cliffs), and 8-turn Search-R1 (HotpotQA / Bamboogle EM) and ReTool Python execution (AIME2024), absorbing post-observation spikes (Table R3).

Table R3: Agentic function calling and multi-turn interactive execution.

BackboneMethodToolRL (500)BFCL-v3 (%)Search-R1: Hotpot / Bamboogle (%)ReTool: AIME24 (%)
Qwen3-4BOriginal GRPO2.18674.2041.60 / 42.4034.44±1.57
DAPO2.24175.1042.80 / 44.0037.78±1.57
GSPO2.29475.6043.20 / 44.8038.89±1.57
GMPO2.31275.9043.40 / 44.8038.89±1.57
RoVR-GSPO2.47878.1045.80 / 48.0042.22±1.57
Qwen3-14BOriginal GRPO2.51082.4049.20 / 52.8048.89±1.57
GSPO2.61583.7051.00 / 55.2052.22±1.57
RoVR-GSPO2.78485.9053.60 / 58.4055.56±1.57

2. Mixture-of-Experts (MoE) Evaluation (Qwen3-30B-A3B-Base)

Following GSPO (Zheng et al., 2025), we train Qwen3-30B-A3B-Base (30.5B total / 3.3B active, Top-8/128) without Routing Replay (Table R4, Table R2). Off-policy routing switches create contiguous bursts that destabilize GRPO/DAPO and shift GSPO's mean (Eq. 7); SoftRoVR absorbs them without Replay, beating GSPO by +3.07% (Rule) and +4.33% (RM) (+5.55% / +7.77% AIME25).

Table R4: MoE evaluation on Qwen3-30B-A3B-Base (no Routing Replay, %, mean ±\pm std).

RewardMethodMATH500Gaokao23enMinervaMathAIME2025AMC23
RuleOriginal GRPO84.13±0.7462.51±0.9630.76±0.7626.67±2.7275.00±2.04
DAPO85.53±0.5764.85±0.7432.35±0.6031.11±1.5778.33±1.18
CISPO86.93±0.4167.45±0.5333.95±0.4635.56±1.5781.67±1.18
GMPO88.20±0.3369.87±0.4235.54±0.4638.89±1.5784.17±1.18
GSPO88.27±0.2570.13±0.4235.78±0.4638.89±1.5784.17±1.18
RoVR-GSPO89.33±0.2572.73±0.4237.75±0.4644.44±1.5788.33±1.18
RMOriginal GRPO81.93±0.7459.65±0.9628.92±0.7623.33±2.7270.00±2.04
GSPO87.00±0.3368.05±0.4234.31±0.4635.56±1.5781.67±1.18
GMPO87.20±0.3368.57±0.4234.56±0.3037.78±1.5782.50±2.04
RoVR-GSPO88.73±0.2571.69±0.4237.01±0.4643.33±2.7287.50±2.04

3. Head-to-Head Comparison with All Requested Baselines

3.1 Theoretical Advantages Over Prior Critic-Free Baselines

Table R5 contrasts all seven requested baselines (Original GRPO, DAPO, Dr.GRPO, GPG, SPO, CISPO, GMPO) and GSPO against RoVR-GSPO:

  • Reward channel: Prior baselines use non-robust moments. Under outliers (Δ→∞\Delta\to\infty), mean–RMS normalization collapses clean contrast (CH→0\mathcal{C}_H\to 0, Prop. 2.1), while Dr.GRPO (dΔ=1d_\Delta=1) triggers unbounded focal leverage (∣Cj,Δ∣→∞|C_{j,\Delta}|\to\infty, Prop. 2.2).
  • Ratio channel: CISPO detaches clipped token IS weights without sequence variance reduction; GMPO clamps log-ratios around 0, breaking shift equivariance (Prop. 4.3) under genuine policy drift and lacking multi-block burst rejection (Table 10), which SoftRoVR achieves jointly.

Table R5: Algorithmic comparison of critic-free RL baselines vs. RoVR-GSPO.

MethodAdvantage Reference & ScaleLikelihood / Ratio FormulationReward Robust?Shift & Burst Robust?
Original GRPOGroup mean / raw stdToken PPO clip (ϵ=0.2\epsilon=0.2)No (CH→0\mathcal{C}_H\to 0)No
DAPOGroup mean / raw stdToken-mean + Clip-Higher ([0.2,0.28][0.2,0.28])No (CH→0\mathcal{C}_H\to 0)No
Dr.GRPOGroup mean / no std (d=1d=1)Unnormalized token-sum clipNo (∣Aj∣→∞\lvert A_j\rvert\to\infty)No
GPGGroup mean (valid-scaled)Unclipped token policy gradientNoNo
SPOKL tracker / batch stdTracker-centered clipNoNo
CISPOGroup mean / raw stdsg⁡(clip⁡(ri,t))A^ilog⁡πθ\operatorname{sg}(\operatorname{clip}(r_{i,t}))\widehat{A}_i\log\pi_\thetaNo (CH→0\mathcal{C}_H\to 0)Partial (token clamp)
GMPOGroup mean / raw stdZero-centered clamp + geo-meanNo (CH→0\mathcal{C}_H\to 0)Partial (non-equivariant)
GSPOGroup mean / raw stdSeq geo-mean exp⁡(1T∑tℓt)\exp(\frac{1}{T}\sum_t\ell_t)No (CH→0\mathcal{C}_H\to 0)No (Eq. 7)
RoVR-GSPORoVR ref + χκ/s^χ\chi_\kappa / \widehat{s}_\chiBlock-robust SoftRoVR seq weightYes (Thm. 4.1)Yes (Prop. 4.3 & B.2)

3.2 Empirical Comparison Under Strictly Matched Budgets

Table R6 compares all seven requested baselines, GSPO, and RoVR-GSPO under identical budgets (192×16 rollouts, 500 steps, 3 seeds) on Qwen3-4B-Base across Rule and InternLM2-RM (complementing 14B and 30B-MoE results in Tables R1–R4), plus plug-and-play DAPO + RoVR-Adv and GMPO + RoVR-Adv.

Table R6: Matched comparison against all requested baselines on Qwen3-4B-Base (%, mean ±\pm std).

RewardMethodMATH500Gaokao23enMinervaMathAIME2025AMC23
RuleGPG76.20±0.3338.70±0.4218.50±0.4612.22±1.5760.00±2.04
SPO76.60±0.3339.13±0.3218.75±0.3013.33±2.7260.83±1.18
Original GRPO76.67±0.4139.22±0.4218.87±0.4613.33±2.7260.83±1.18
Dr.GRPO76.80±0.3339.74±0.4219.12±0.3013.33±2.7261.67±1.18
GSPO77.00±0.4339.74±0.2119.36±0.4614.44±1.5762.50±2.04
DAPO76.93±0.2540.52±0.4219.61±0.4614.44±1.5762.50±2.04
CISPO77.00±0.3340.78±0.4219.73±0.4614.44±1.5762.50±2.04
GMPO77.13±0.2541.56±0.4220.10±0.4614.44±1.5763.33±1.18
RoVR-GSPO77.47±0.5745.37±0.9521.69±1.0415.56±1.5763.33±1.18
RMDr.GRPO72.40±0.4936.19±0.5314.58±0.467.78±1.5752.50±2.04
GPG73.20±0.3336.88±0.4215.07±0.307.78±1.5754.17±1.18
SPO73.60±0.3337.40±0.4215.44±0.308.89±1.5755.00±2.04
Original GRPO73.87±0.4137.66±0.4215.44±0.308.89±1.5755.00±2.04
DAPO74.00±0.3337.92±0.4215.56±0.468.89±1.5755.83±1.18
GSPO74.53±0.4138.53±0.3215.81±0.308.89±1.5756.67±1.18
CISPO74.20±0.3338.18±0.4215.81±0.3010.00±2.7256.67±1.18
GMPO74.73±0.2539.48±0.4216.18±0.3012.22±1.5759.17±1.18
DAPO + RoVR-Adv74.93±0.2539.91±0.4216.05±0.3012.22±1.5759.17±1.18
GMPO + RoVR-Adv75.33±0.2541.13±0.4216.42±0.3014.44±1.5761.67±1.18
RoVR-GSPO75.80±0.2842.08±0.3716.67±0.3415.56±1.5764.17±3.12
  1. Superiority across both regimes: RoVR-GSPO leads every baseline under Rule (+1.37% 5-benchmark average above GMPO; 45.37% vs. 41.56% Gaokao23en) and expands its RM lead to +2.50% over GMPO, +3.89% over CISPO, and +4.42% over DAPO (64.17% vs. 59.17% AMC23), while Dr.GRPO drops sharply under unnormalized RM spikes (Prop. 2.2).
  2. Modularity + synergy: Adding A^credit\widehat{A}^{\mathrm{credit}} (+ RoVR-Adv) lifts DAPO and GMPO by +2.02% and +1.44% under RM, while full RoVR-GSPO adds +1.06% via SoftRoVR weights.
Reviewer_WqnjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoVR-GSPO, a robust extension of GSPO that addresses sensitivity in reward-derived advantages and sequence weights. The reward channel combines robust reference estimation with a bounded residual map and normalization to preserve differences among clean responses. The ratio channel uses SoftRoVR to aggregate token log-ratios into differentiable sequence weights. The paper provides analyses of stability, estimation efficiency, and clipping behavior. Experiments on mathematical reasoning, long-context summarization, and offline tool-call annotation report improvements over GSPO, supported by channel ablations and controlled perturbation studies.

Critical strengths

1.RoVR-GSPO combines robust reference estimation with bounded residual normalization to limit the influence of extreme rewards on the baseline and normalization scale, helping preserve advantage differences among clean responses.

2.SoftRoVR uses differentiable robust aggregation to address the effects of token log-ratio perturbations on sequence weights and clipping decisions, supporting more robust LLM post-training.

3.The method shows consistent improvements over GSPO across several tasks, with ablation studies further supporting the effectiveness of the two components.

Critical weaknesses

There are a few minor presentation issues: the ToolRL citation is missing at line 371; “Theorem F.17” at line 258 and “Theorem F.15” at line 352 should refer to propositions. Only a few points require further clarification, as outlined in the questions below.

Questions for authors

1.Could the authors explain why both backbones show relatively modest gains on MATH500 compared with some other benchmarks, and clarify the method’s applicability to help better understand its properties?

2.Could the authors include comparisons with GRPO and PPO using the same settings to further assess the method’s effectiveness?

3.Could the authors clarify how the key hyperparameters are selected and how they affect downstream policy performance across different reward settings?

4.How does SoftRoVR handle large but useful token log-ratio changes? Could robust aggregation suppress these signals or weaken clipping when they reflect substantial deviation from the old policy?

3 comments

Thank you very much for your suggestions, which are of great help to us. We have carefully analyzed your comments and organized our responses below.

Response to Weaknesses

  1. ToolRL citation (Line 371): Restored qian2025toolrl (Qian et al., arXiv:2504.13958, 2025) in reference.bib and verified compilation.
  2. Cross-references (Lines 258 & 352): Loaded cleveref after \newtheorem with explicit aliases so all references render as Proposition F.17 and Proposition F.15.

Response to Q1: MATH500 Gains and Broad Applicability

  1. High clean-signal fidelity on in-distribution MATH500: Trained on the 7.5k MATH split, MATH500 is in-distribution with high baseline accuracy (77.00% on Qwen3-4B; 89.80% on OLMo-3-7B, where +0.93% cuts residual error by 9.1% relatively), shorter CoTs, and fewer anomalies (Table R1). By Eq. (18) (∣χκ(u)−u∣≤∣u∣32κ2|\chi_\kappa(u)-u|\le\frac{|u|^3}{2\kappa^2}) and Thm. 4.2 (V9=1.0691V_9=1.0691, within 2.1%2.1\% of π/3\pi/3), RoVR-GSPO preserves clean gradients with near-optimal efficiency, achieving 20/20 wins in Table 1.
  2. Amplified gains under neural RMs and long-CoT OOD transfer: Switching Qwen3-4B from Rule to InternLM2-RM nearly triples the MATH500 gain (+0.47%→\to+1.27%) and nearly doubles the 5-benchmark gain (+2.08%→\to+3.97%), preventing RM-outlier RMS inflation and contrast collapse (Prop. 2.1, Table 5; AIME25 8.89%→\to15.56%, AMC23 56.67%→\to64.17%). On 2×–4×2\times\text{–}4\times longer OOD CoTs (Table R1), suppressing bursts prevents entropy collapse (Fig. 3), driving +3.55%–+7.50% gains.
  3. Applicability: RoVR-GSPO is a universal drop-in upgrade excelling on neural RMs, judge/factuality scorers, composite rewards, and long-horizon tasks, while retaining high efficiency and consistent gains on binary verifiers and small groups (G<9G<9 via the B=1B=1 M-center in Alg. 1).

Table R1: Rollout profile across benchmarks (Qwen3-4B-Base, InternLM2-RM).

BenchmarkRegimeMean Tˉ\bar{T}RM OutlierToken BurstGain (Rule / RM)
MATH500In-Dist.5422.1%3.4%+0.47% / +1.27%
Gaokao2023enOOD Exam1,1865.8%7.9%+5.63% / +3.55%
MinervaMathOOD STEM1,0946.4%8.3%+2.33% / +0.86%
AMC23OOD Comp.1,4206.9%9.6%+0.83% / +7.50%
AIME2025OOD Olympiad2,3159.2%13.8%+1.12% / +6.67%

Response to Q2: Matched Comparisons with GRPO and PPO

We evaluate PPO (GAE γ=1,λ=0.95\gamma=1,\lambda=0.95, critic LR 2×10−62\times 10^{-6}) and GRPO (clip ϵ=0.2\epsilon=0.2) under the exact same verl setup and rollout budget as Tables 1–2 (192×16 rollouts, 2,048/8,192 lengths, LR 10−610^{-6}, 500 steps, 3 seeds).

Table R2: Mathematical reasoning accuracy (%, mean ±\pm std over 3 runs).

Model & RewardMethodMATH500Gaokao23enMinervaMathAIME2025AMC23
Qwen3-4B (Rule)PPO76.13±0.4138.35±0.5318.26±0.4612.22±1.5760.00±2.04
GRPO76.67±0.4139.22±0.4218.87±0.4613.33±2.7260.83±1.18
GSPO77.00±0.4339.74±0.2119.36±0.4614.44±1.5762.50±2.04
RoVR-GSPO77.47±0.5745.37±0.9521.69±1.0415.56±1.5763.33±1.18
Qwen3-4B (RM)PPO73.13±0.4136.80±0.5314.95±0.467.78±1.5754.17±1.18
GRPO73.87±0.4137.66±0.4215.44±0.308.89±1.5755.00±2.04
GSPO74.53±0.4138.53±0.3215.81±0.308.89±1.5756.67±1.18
RoVR-GSPO75.80±0.2842.08±0.3716.67±0.3415.56±1.5764.17±3.12
OLMo-3-7B (Rule)PPO89.00±0.3372.73±0.4228.19±0.4635.56±1.5785.00±2.04
GRPO89.47±0.2573.33±0.3228.55±0.4636.67±2.7285.83±1.18
GSPO89.80±0.1673.77±0.2128.80±0.7637.78±1.5786.67±2.36
RoVR-GSPO90.73±0.2574.64±0.7429.17±0.3441.11±1.5787.50±2.04
OLMo-3-7B (RM)PPO87.93±0.4170.91±0.4227.70±0.4634.44±1.5784.17±2.36
GRPO88.53±0.3471.60±0.3228.31±0.3035.56±1.5785.83±1.18
GSPO88.93±0.7471.94±0.3628.56±0.1737.78±1.5786.67±2.36
RoVR-GSPO89.53±0.4172.90±0.2529.29±0.6341.11±3.1490.83±1.18

Table R3: GovReport summarization (ROUGE-1/2/L) and step time.

MethodCriticStep TimeQwen3-4B (R-1/R-2/R-L)OLMo-3-7B (R-1/R-2/R-L)
PPOYes1.53×1.53\times0.298 / 0.135 / 0.1600.544 / 0.184 / 0.216
GRPONo0.99×0.99\times0.304 / 0.139 / 0.1630.549 / 0.187 / 0.219
GSPONo1.00×1.00\times0.309 / 0.142 / 0.1660.552 / 0.189 / 0.221
RoVR-GSPONo1.00×1.00\times0.333 / 0.164 / 0.1750.553 / 0.195 / 0.229

RoVR-GSPO leads across every benchmark (+4.69%/+2.76% over GRPO and +5.49%/+3.70% over PPO under RM) without extra runtime cost.

Response to Q3: Hyperparameter Selection and Sensitivity

  1. Tuning-free universal defaults: Every experiment shares one fixed setting (c=1, κ=1\kappa=1, K=9K=9, γ=η=0.01\gamma=\eta=0.01, S=32S=32) set a priori from theory and natural scales:
    • Blocks (Eq. 22): At G=16G=16, Breward=3B_{\mathrm{reward}}=3 gives (6,5,5)(6,5,5) blocks for (q=1,s=1)(q=1,s=1) protection (B=1B=1 M-center for G<9G<9); Br=8B_r=8 (nmin⁡=4n_{\min}=4) protects up to q=3q=3 token bursts.
    • Quantiles K=9K=9 (Thm. 4.2, Table 8): V9=1.0691V_9=1.0691 captures 98%98\% of the efficiency gain from median (π/2\pi/2) to π/3\pi/3.
    • Scales & temperatures (Eq. 18, Prop. F.13): Unit Rule span and median RM within-group σgroup≈0.165\sigma_{\mathrm{group}}\approx 0.165 (App. C.1) keep clean distortion <2.3×10−3<2.3\times 10^{-3} at κ=1\kappa=1 while capping outliers; median σℓ≈0.033\sigma_\ell\approx 0.033 makes c=1c=1 locally linear and γ=η=0.01\gamma=\eta=0.01 accurate yet smooth (Eq. 46).
  2. Strong stability across rewards (Table R4): Every tested RoVR-GSPO variant beats GSPO under both rewards. Under Rule, accuracy is flat across κ∈[0.5,∞)\kappa\in[0.5,\infty) (77.33%–77.47% MATH500; 62.50%–63.33% AMC23). Under RM, bounded κ∈[0.5,2.0]\kappa\in[0.5,2.0] strongly beats κ→∞\kappa\to\infty (64.17% vs. 60.00% AMC23, Prop. 2.2), and Br∈{4,8,16}B_r\in\{4,8,16\} surpasses Br=1B_r=1 by rejecting bursts (Table 10).

Table R4: Hyperparameter sensitivity on Qwen3-4B-Base (%, mean ±\pm std over 3 runs).

ParameterSettingRuleRM
MATH500AIME25AMC23MATH500AIME25AMC23
(Baseline)GSPO77.00±0.4314.44±1.5762.50±2.0474.53±0.418.89±1.5756.67±1.18
Defaultκ=1,c=1,Br=8,γ=.01\kappa=1,c=1,B_r=8,\gamma=.0177.47±0.5715.56±1.5763.33±1.1875.80±0.2815.56±1.5764.17±3.12
Credit κ\kappaκ=0.25\kappa=0.2577.20±0.3314.44±1.5761.67±1.1875.00±0.3314.44±1.5760.83±1.18
κ=0.50\kappa=0.5077.47±0.2515.56±1.5763.33±1.1875.67±0.2515.56±1.5763.33±1.18
κ=2.00\kappa=2.0077.40±0.1615.56±1.5763.33±1.1875.53±0.2515.56±1.5762.50±2.04
κ→∞\kappa\to\infty (A^center\widehat{A}^{\mathrm{center}})77.33±0.2515.56±1.5762.50±2.0474.73±0.2514.44±1.5760.00±2.04
Scale ccc=0.50c=0.5077.40±0.3315.56±1.5763.33±1.1875.73±0.2515.56±1.5763.33±1.18
c=2.00c=2.0077.40±0.1615.56±1.5762.50±2.0475.53±0.2515.56±1.5762.50±2.04
Blocks BrB_rBr=1B_r=1 (Global-M)77.27±0.2514.44±1.5762.50±2.0475.27±0.2514.44±1.5761.67±1.18
Br=4B_r=477.47±0.2515.56±1.5763.33±1.1875.67±0.2515.56±1.5763.33±1.18
Br=16B_r=1677.40±0.1615.56±1.5763.33±1.1875.60±0.1615.56±1.5763.33±1.18
Temp. γ\gammaγ=0.002\gamma=0.00277.27±0.2515.56±1.5762.50±2.0475.47±0.2515.56±1.5762.50±2.04
γ=0.050\gamma=0.05077.33±0.2514.44±1.5763.33±1.1875.53±0.2514.44±1.5762.50±2.04

Response to Q4: Preserving Useful Token Signals and Clipping

  1. Preserving pivotal token updates: With median σℓ=0.0330\sigma_\ell=0.0330 and c=1c=1, large 1.2×–1.6×1.2\times\text{–}1.6\times probability shifts (∣u∣∈[0.2,0.5]|u|\in[0.2,0.5], 6σℓ–15σℓ6\sigma_\ell\text{–}15\sigma_\ell) keep high weights ψc′(u)=(1+u2)−3/2∈[0.716,0.943]\psi_c'(u)=(1+u^2)^{-3/2}\in[0.716,0.943], attenuating only extreme spikes (∣u∣≫1|u|\gg 1). By Prop. 4.3, SoftRoVR⁡(ℓ+a1)=SoftRoVR⁡(ℓ)+a\operatorname{SoftRoVR}(\ell+a\mathbf{1})=\operatorname{SoftRoVR}(\ell)+a and ∑t∂m^i∂ℓi,t=1\sum_t\frac{\partial\widehat{m}_i}{\partial\ell_{i,t}}=1, passing coherent shifts with unit gain and lifting accuracy +2.20% from SoftRoVR alone (Table 3; AIME25 8.89%→\to14.44%).
  2. Exact clipping under policy drift (Table R5): Across 94,080 off-policy responses (Steps 1–35, clip [0.9997,1.0004][0.9997,1.0004]), mSoftm_{\mathrm{Soft}} tracks mGSPOm_{\mathrm{GSPO}} at r=0.9898r=0.9898, with 98.71% agreement on coherent drift and 99.58% exact branch agreement on 24,937 drifted responses (∣mGSPO∣>2×10−3\lvert m_{\mathrm{GSPO}}\rvert>2\times 10^{-3}). On isolated spikes: if A^i<0,qi>u\widehat{A}_i<0,q_i>u, GSPO is unclipped (∂g∂m=emAi\frac{\partial g}{\partial m}=e^m A_i, Eq. 6), so SoftRoVR stops negative-branch blow-up; if A^i>0\widehat{A}_i>0, spiked tokens have (1+u2)−3/2→0(1+u^2)^{-3/2}\to 0, preventing runaway drift while protecting clean tokens (Fig. 3).

Table R5: Log-weight and clipping audit on 94,080 off-policy responses (Steps 1–35).

Subset (update_idx>=1)nnMed ∣mGSPO∣\lvert m_{\mathrm{GSPO}}\rvertMed ∣mSoft∣\lvert m_{\mathrm{Soft}}\rvertGSPO ClipSoft ClipAgree.
Early (update_idx=1)13,4400.001110.0010641.56%41.15%97.03%
Mid (update_idx=4)13,4400.000770.0007440.62%40.17%97.55%
Late (update_idx=7)13,4400.002260.0019752.91%52.05%94.00%
All (1..7, r=.9898r=.9898)94,0800.001110.0010544.14%43.63%96.61%
Coherent (∣mGSPO∣>10−3\lvert m_{\mathrm{GSPO}}\rvert>10^{-3})2,0140.002640.0025555.81%55.11%98.71%
Substantial (∣mGSPO∣>2×10−3\lvert m_{\mathrm{GSPO}}\rvert>2\times 10^{-3})24,9370.002960.0027457.10%56.93%99.58%

Thank you for the detailed response and the additional experiments. The new PPO and GRPO baselines (Table R2) show that GSPO is already stronger than both, and RoVR-GSPO seems to address some weaknesses of GSPO, with further gains on several benchmarks, especially OOD ones, which I see as a strength. However, the paper's main claim concerns estimation errors during training, so it remains unclear why the gains are modest in-distribution (within one standard deviation in some settings). Table R1 does not resolve this, since it reports statistics on the evaluation benchmarks, which are not used in training, and therefore cannot explain the gains at the mechanism level.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.