- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a dual-channel optimizer, RoVR-GSPO, to mitigate the interference caused by potential reward perturbations in GSPO. The authors evaluate the proposed method on two models and demonstrate its effectiveness through experimental results.
Critical strengths
- This paper investigates and analyzes the issues arising from reward perturbations in GSPO, providing insights that help researchers better understand and apply the algorithm.
Critical weaknesses
-
The experimental results are not entirely convincing. As shown in Table 1, most performance gains over GSPO are less than two points. Such marginal improvements may result from hyperparameter tuning or random variation, making it difficult to establish the effectiveness of the proposed approach without additional evidence, such as results across multiple runs and statistical significance tests.
-
As shown in Table 2, the performance gains decrease as the model size increases from 4B to 7B. This trend raises concerns that the benefits of the proposed approach may diminish further on larger models, potentially limiting its practical value and scalability.
-
To better establish the effectiveness and generalizability of the proposed algorithmic improvement, the authors should evaluate it on a broader range of tasks and across models of different sizes and model families. Such experiments would help determine whether the observed benefits consistently extend beyond the current experimental settings.
Questions for authors
- There is a missing citation on line 371.
2 comments
3. Generalizability Across Model Families, Sizes, and Broader Task Domains
To establish broad algorithmic generality, we evaluate RoVR-GSPO across three distinct model families (Qwen3, OLMo-3, Llama-3.1), five model scales and architectures (4B, 7B, 8B, 14.8B Dense, and 30.5B-A3B MoE), and five diverse post-training domains: Mathematical Reasoning (Table R3), Long-Context Summarization (Table R2), Execution-Verified Code Generation, Single-Turn Function Calling, and Multi-Turn Interactive Agents (Table R4)—all using the exact same default RoVR-GSPO parameters.
Table R3: Cross-scale and cross-family mathematical reasoning (%, 3-run means; Rel. Error Red. denotes five-benchmark relative error reduction).
| Family & Model | Params | Reward | GSPO: MATH500 / AIME25 / AMC23 | RoVR-GSPO: MATH500 / AIME25 / AMC23 | Rel. Error Red. |
|---|---|---|---|---|---|
| Qwen3-4B-Base | 4B | Rule | 77.00 / 14.44 / 62.50 | 77.47 / 15.56 / 63.33 | 3.6% |
| Qwen3-8B-Base | 8B | Rule | 82.20 / 22.22 / 70.83 | 83.00 / 25.56 / 74.17 | 5.6% |
| Qwen3-30B-A3B (MoE) | 30.5B | Rule | 88.27 / 38.89 / 84.17 | 89.33 / 44.44 / 88.33 | 8.4% |
| Qwen3-4B-Base | 4B | RM | 74.53 / 8.89 / 56.67 | 75.80 / 15.56 / 64.17 | 6.5% |
| Qwen3-8B-Base | 8B | RM | 80.40 / 18.89 / 66.67 | 81.80 / 24.44 / 72.50 | 7.1% |
| Qwen3-14B-Base | 14.8B | RM | 84.80 / 30.00 / 77.50 | 86.20 / 34.44 / 81.67 | 7.2% |
| Qwen3-30B-A3B (MoE) | 30.5B | RM | 87.00 / 35.56 / 81.67 | 88.73 / 43.33 / 87.50 | 11.2% |
| OLMo-3-7B-Instruct | 7B | Rule | 89.80 / 37.78 / 86.67 | 90.73 / 41.11 / 87.50 | 3.5% |
| OLMo-3-7B-Instruct | 7B | RM | 88.93 / 37.78 / 86.67 | 89.53 / 41.11 / 90.83 | 5.3% |
| Llama-3.1-8B-Instruct | 8B | Rule | 54.13 / 8.89 / 44.17 | 55.60 / 12.22 / 48.33 | 4.1% |
| Llama-3.1-8B-Instruct | 8B | RM | 51.73 / 7.78 / 40.00 | 54.27 / 11.11 / 46.67 | 5.4% |
For broader task domains (Table R4):
- Execution-verified coding (
HumanEval+,MBPP+,LiveCodeBench v5): Trained with sandbox unit-test pass-rate rewards and syntax/timeout penalties, where RoVR-GSPO protects partial-pass credit contrast and stabilizes branching-token bursts. - Function calling & multi-turn interactive agents: Evaluated on full 500-step ToolRL (
RLLA-4K), zero-shot BFCL-v3 AST accuracy, 8-turn Search-R1 retrieval (HotpotQAEM), and ReTool Python execution (AIME2024), where dual-channel estimation handles both tool-parse cliff penalties and post-observation log-ratio spikes.
Table R4: Generalization to Execution-Verified Coding (Pass@1 %) and Agentic Tasks.
| Family & Backbone | Method | Coding: HumanEval+ / MBPP+ / LiveCodeBench (%) | Tool-Call: ToolRL (500-step) / BFCL-v3 (%) | Interactive Agent: Search-R1 HotpotQA / ReTool AIME24 (%) |
|---|---|---|---|---|
| Qwen3-4B-Base | GSPO | 70.12 / 67.20 / 26.16 | 2.294 / 75.60 | 43.20 / 38.89 |
| (4B Dense) | RoVR-GSPO | 71.95 / 68.78 / 28.67 | 2.478 / 78.10 | 45.80 / 42.22 |
| Llama-3.1-8B-Instruct | GSPO | 67.68 / 65.08 / 21.86 | 2.215 / 73.40 | 41.80 / 23.33 |
| (8B Meta Family) | RoVR-GSPO | 70.12 / 66.93 / 24.37 | 2.406 / 76.20 | 44.60 / 26.67 |
| Qwen3-14B-Base | GSPO | 79.27 / 75.13 / 39.78 | 2.615 / 83.70 | 51.00 / 52.22 |
| (14.8B Dense) | RoVR-GSPO | 81.10 / 76.72 / 42.29 | 2.784 / 85.90 | 53.60 / 55.56 |
| Qwen3-30B-A3B-Base | GSPO | 81.71 / 76.98 / 43.37 | — | — |
| (30.5B MoE) | RoVR-GSPO | 83.54 / 78.57 / 46.24 | — | — |
Across all three model families, all scales (4B to 30.5B MoE), and all five task domains, RoVR-GSPO consistently outperforms GSPO—achieving +1.97% to +2.27% average gains on Coding (+2.51% to +2.87% on LiveCodeBench v5), +2.20% to +2.80% on BFCL-v3, and +2.60% to +3.33% on multi-turn Search-R1 and ReTool agents.
4. Response to Question
Thank you for catching this. The missing BibTeX entry qian2025toolrl (Qian et al., "ToolRL: Reward is All Tool Learning Needs," arXiv:2504.13958, 2025) has been restored, and we have verified clean compilation across the manuscript.
Thank you very much for your suggestions. We have carefully analyzed your comments, organized them and supplemented the corresponding experiments, hoping to address your concerns. We summarize the issues you raised as follows:
1. Statistical Significance, Multi-Seed Evidence, and Hyperparameter Neutrality
1.1 Zero hyperparameter tuning and universal fixed defaults
192/24, G=16, clip [0.9997,1.0004], length 8,192,500steps) are inherited verbatim from the parent GSPO baseline.c=1.0, κ=1.0, K=9, Bratio=8, γ=η=0.01) were fixed a priori from theoretical efficiency (V9=1.0691, within 2.1% of π/3, Table 8) and natural scales, and applied identically without per-task, per-model, or per-reward tuning. Sweeps over κ∈[0.5,2.0], c∈[0.5,2.0], Bratio∈{4,8,16}, and γ∈[0.002,0.05] confirm every tested variant beats GSPO (+1.65%–+2.08%on Rule;+3.21%–+3.97%on RM).1.2 Multi-run expansion (n=5 seeds) and formal significance tests
In Table 1 (3 runs), RoVR-GSPO achieves 20/20 consistent wins, cutting residual error by
9.1%relatively even on near-saturatedOLMo-3-7BMATH500 (89.80%→90.73%) while delivering+3.55%–+7.50%absolute gains on harder OOD and RM benchmarks. Expanding all four(backbone × reward)pairs ton = 5independent seeds (Table R1), we compute two-sided Welch's t-tests across independent runs and paired stratified bootstrap 95% CIs (10,000replicates) across all1,227test problems.Table R1: Extended
n = 5independent-seed evaluation and significance tests.All four gains are highly statistically significant (p≤0.0014) with 95% CIs strictly above zero.
2. Controlled Scaling Study: Strong and Growing Gains on Larger Models
In Table 2, comparing
Qwen3-4B-Base(GSPO R-1=0.309) withOLMo-3-7B-Instruct(GSPO R-1=0.552) simultaneously changed model family, size, and alignment stage (Basevs. SFT-alignedInstructnear the~0.56GovReport R-1 ceiling). Holding family andBaseinitialization fixed acrossQwen3-4B-Base,8B-Base,14B-Base, and30B-A3B-Base(MoE) overn=3seeds (Tables R2–R3) isolates true scaling:Table R2: Multi-seed (
n = 3) GovReport scaling study (ROUGE-1 / ROUGE-2 / ROUGE-L, mean ± std).4B→8B→14B, ROUGE gains hold steady (+.023–.025R-1,+.019–.021R-2) while R-L gain increases from+.009to+.013(p≤0.0084);Instructgains are also significant (p<0.009).+3.97%(4B),+3.68%(8B),+3.06%(14B), and+4.33%(30.5B MoE)—a monotonically increasing relative error reduction from6.5%(4B) →7.1%(8B) →7.2%(14B) →11.2%(30.5B MoE).