- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a novel reinforcement learning method that leverages lightweight offline dataset profiling to enable efficient rollout allocation during training, achieving substantial compute savings compared to reinforcement learning baselines.
Critical strengths
SGPO enables superior compute allocation via compute-efficient offline difficulty assessment, cutting total FLOPs and GPU hours by 2.5–2.9× over baselines.
Critical weaknesses
-
The mathematical evaluation is limited exclusively to competition-level benchmarks (AIME and HMMT), whereas mainstream studies (e.g., GRESO) routinely evaluate across benchmarks with diverse difficulty levels. For a 1.5B model, evaluating on such challenging datasets yields evaluation variance and noise that likely dominate the actual score, offering little meaningful signal.
-
The evaluated baselines all incorporate dynamic sampling, which inherently introduces substantial computational overhead in FLOPs. In fact, standard GRPO on smaller models can already achieve competitive performance with significantly lower FLOPs compared to these dynamic sampling baselines. The omission of vanilla GRPO is a critical flaw. Furthermore, the paper should compare against methods that achieve data filtering and acceleration without relying on dynamic sampling, such as GRESO without dynamic sampling, as well as zero-overhead allocation methods like VIP and VIGOR.
Questions for authors
- I have reservations regarding the theoretical grounding of sGPO's rollout allocation rule, which appears to conflict with established conclusions in works such as VIP and XRPO. While existing studies emphasize minimizing gradient estimation variance or exploring uncertainty, sGPO instead targets maximizing the probability of exactly one successful rollout per group—a heuristic that could induce substantial gradient variance. Although the authors prove optimality under this specific surrogate criterion, they fail to adequately reconcile it with prior theoretical findings and offer little formal analysis on the resulting gradient quality or actual compute efficiency. I encourage the authors to clarify this theoretical discrepancy and provide more rigorous justification or empirical evidence to support their allocation objective.
2 comments
Analysis
Q1 (reconciling sGPO with VIP and XRPO)
-
VIP and XRPO (and sGPO) optimize for different compute regimes. Assuming a large budget and unnormalized advantages (), VIP focuses on "minimizing gradient variance": because peaks at and vanishes as , VIP concentrates rollouts on medium-difficulty prompts and treats low- prompts - whose groups mostly return zero updates (, ) - as already low-variance rather than wasted compute. Under our tighter budget, however, the primary bottleneck on hard prompts () is
learning signal discovery- getting at least one passing rollout so the group does not produce a zero update. Scaling rollouts inversely with pass rate (, so at and at ) lowers the probability of an all-fail group at from down to . This echoes XRPO's finding that uncertainty-based allocation overlooks sparsely rewarded hard prompts (quoting the XRPO's arxiv paper: "However, simply allocating more rollout budgets to the uncertain prompts will overlook the sparsely sampled and hard prompts."). -
While VIP's "Inverse-Accuracy" experiment underperformed, this was likely because they did not filter out unsolvable () prompts, which wasted compute on of DAPO-Math-17k; once we filter out of prompts upfront and apply only to solvable prompts (), we have shown that we outperform a compute-matched uniform baseline on 7B (31.0 vs. 29.5 on AIME24; 82.6 vs. 80.9 on MATH500).
-
1-of-8 groups
do not blow up gradient variancein normalized GRPO. Dividing rewards by the group standard deviation mathematically forces the mean squared advantage to equal () for any group with at least one pass and one fail. For example, a 1-of-8 group produces one large positive advantage () and seven small negative ones (), whose squares average to - the exact same average squared weight as a balanced 4-of-8 group (). Under equal rollout variance (as assumed in VIP), 1-of-8 and 4-of-8 groups inject the exact same gradient variance into Eq. (4). Finally, capping naturally bounds the maximum advantage at and preserves exact zero-sum updates (). This avoids extreme spikes seen in Knapsack RL, which allocates up to ). We will add this discussion to Section 3.2 and Appendix B.
We thank the reviewer again and would be glad to discuss further.
References:
- XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation - https://arxiv.org/abs/2510.06672
- Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards (VIP) - https://arxiv.org/abs/2602.01601
- Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning (VIGOR) - https://arxiv.org/abs/2607.22002
Response to Reviewer WW5n
We thank the reviewer for the review and for recognizing the compute savings of offline difficulty assessment. Our central claim is training efficiency: reaching the baselines' accuracy with substantially less compute. We address each point below.
We agree that competition math at 1.5B (especially HMMT25) sits near the floor with high relative noise, which is why
Section 4.2 interprets 1.5B Table 2 strictly as preserving aggregate accuracy at 2.9x lower compute rather than claiming accuracy gains. Away from the competition floor, sGPO's efficiency holds cleanly across broader difficulty levels:SciKnowEval science across model families (Tables 3 & 10): Unlike recent RLVR rollout allocators evaluated only on math, code, or retrieval (GRESO, Knapsack RL, XRPO, VIP, VIGOR),
sGPO transfers the exact same default configuration(N=8, t=0.75, alpha=10%) unchanged toSciKnowEval-L3across four science domains—matching DAPO on Qwen3-4B (58.0 vs. 57.9 Avg; 62.5 vs. 62.2 wAvg) and matching DAPO, GRESO, and Knapsack RL on Llama-3.2-3B using2.4-2.8x less compute.MATH500 across Levels 1–5 (new): we added MATH500, which spans a broad difficulty distribution (Levels 1–5) well away from the competition floor. On Qwen2.5-Math-7B,
sGPO reaches 82.6 at 8.9 EF - outperforming compute-matched uniform G=4(80.9 at 8.5 EF), uniform G=2 (79.0 at 5.1 EF), and no-curriculum (81.0 at 8.9 EF), while matching unfiltered sGPO (82.3 at 16.7 EF) at nearly half the compute (see our response to ReviewernjLRfor the full table). We will add MATH500 across all methods and both model scales to the main results table in the revised paper.Thank you for this important point. We ran standard GRPO without dynamic sampling (G∈{2,4,8}, KL loss, symmetric clipping, sequence-level loss). As shown below,
removing dynamic sampling does not make the baseline more efficient. At matched compute sGPO is clearly ahead (30.6 vs. 25.0 for the best GRPO run within 8 EF), and it reaches DAPO's best accuracy at 7.8 EF, where DAPO needs $\sim$18 EF. Shrinking the uniform group does not recover efficiency either: smaller groups are cheaper but plateau at lower accuracy (see also the matched-compute uniform G=4 ablation in our response to ReviewernjLR).Removing dynamic sampling from GRESO likewise leaves uniform G = 8 throughout its cold-start phase (before reward history accumulates) and uniform G = 8 on all retained queries, whereas sGPO filters from step 1 and uses G∈2,4 on easier queries.
Online allocators like VIP and VIGOR excel on single-pass streaming or long, non-stationary runs by updating difficulty estimates online without an upfront pass. In our budget-constrained regime, however, sGPO offers two key advantages:
Upfront filtering vs. full 8|D| rollouts: Neither VIP nor VIGOR pre-filters prompts—both spend the full 8|D| rollouts/epoch (VIP requires a MiniLM GP with floor G≥3; VIGOR runs 4 sequential generation barriers/step). By contrast, sGPO useszero auxiliary models, assigns G=2 to 27.3% of queries, and uses one cheap inference pass to drop p^>0.75 and 90% of p^=0 upfront -cutting training FLOPs by 2.5x while preserving single-pass parallel generation.Better hard-prompt coverage: because both VIP and VIGOR rank prompts by p(1−p), they concentrate most rollouts on medium difficulty prompts (p≈0.5) while under-sampling hard ones. We will continue this argument in the following paragraph.