acceptodds
Under review as a conference paper at ICLR 2027

DeliberateRL: Shaping the Practice Distribution for Compute-Efficient RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) typically assigns a fixed rollout budget to every query, even though success probabilities under the policy vary widely. As a result, many response groups are all-correct or all-incorrect: they consume generation compute but yield no group-relative learning signal. We frame allocation as shaping the practice distribution - how training effort is distributed across queries of different policy competence over the course of training. Inspired by deliberate practice, DeliberateRL is a competence-aware offline framework that estimates each query's success rate once, with a verified inference-only profiling pass, and uses this estimate to decide what to practice, how much to practice each query, and when. We instantiate it with sorted-Group Policy Optimization (sGPO), which filters high-success queries, sizes rollout groups toward roughly one expected success, orders training from easier to harder queries, and, following spaced repetition, revisits a small fixed subset of zero-observed-success queries throughout the curriculum. Across mathematical and scientific reasoning tasks with Qwen- and Llama-family models, sGPO matches the accuracy of uniform and online-adaptive baselines while using 2.5x less compute than uniform allocation over a two-epoch horizon, including the one-time profiling cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
deliberate-practice, efficient-rlvr, curriculum-learning, adaptive-compute, inference-scaling, offline-profiling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WW5nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a novel reinforcement learning method that leverages lightweight offline dataset profiling to enable efficient rollout allocation during training, achieving substantial compute savings compared to reinforcement learning baselines.

Critical strengths

SGPO enables superior compute allocation via compute-efficient offline difficulty assessment, cutting total FLOPs and GPU hours by 2.5–2.9× over baselines.

Critical weaknesses
  • The mathematical evaluation is limited exclusively to competition-level benchmarks (AIME and HMMT), whereas mainstream studies (e.g., GRESO) routinely evaluate across benchmarks with diverse difficulty levels. For a 1.5B model, evaluating on such challenging datasets yields evaluation variance and noise that likely dominate the actual score, offering little meaningful signal.

  • The evaluated baselines all incorporate dynamic sampling, which inherently introduces substantial computational overhead in FLOPs. In fact, standard GRPO on smaller models can already achieve competitive performance with significantly lower FLOPs compared to these dynamic sampling baselines. The omission of vanilla GRPO is a critical flaw. Furthermore, the paper should compare against methods that achieve data filtering and acceleration without relying on dynamic sampling, such as GRESO without dynamic sampling, as well as zero-overhead allocation methods like VIP and VIGOR.

Questions for authors
  • I have reservations regarding the theoretical grounding of sGPO's rollout allocation rule, which appears to conflict with established conclusions in works such as VIP and XRPO. While existing studies emphasize minimizing gradient estimation variance or exploring uncertainty, sGPO instead targets maximizing the probability of exactly one successful rollout per group—a heuristic that could induce substantial gradient variance. Although the authors prove optimality under this specific surrogate criterion, they fail to adequately reconcile it with prior theoretical findings and offer little formal analysis on the resulting gradient quality or actual compute efficiency. I encourage the authors to clarify this theoretical discrepancy and provide more rigorous justification or empirical evidence to support their allocation objective.
2 comments
AuthorsOpenReview ↗

Response to Reviewer WW5n

We thank the reviewer for the review and for recognizing the compute savings of offline difficulty assessment. Our central claim is training efficiency: reaching the baselines' accuracy with substantially less compute. We address each point below.

W1 (benchmarks beyond competitions).

We agree that competition math at 1.5B (especially HMMT25) sits near the floor with high relative noise, which is why Section 4.2 interprets 1.5B Table 2 strictly as preserving aggregate accuracy at 2.9x lower compute rather than claiming accuracy gains. Away from the competition floor, sGPO's efficiency holds cleanly across broader difficulty levels:

  1. SciKnowEval science across model families (Tables 3 & 10): Unlike recent RLVR rollout allocators evaluated only on math, code, or retrieval (GRESO, Knapsack RL, XRPO, VIP, VIGOR), sGPO transfers the exact same default configuration (N=8, t=0.75, alpha=10%) unchanged to SciKnowEval-L3 across four science domains—matching DAPO on Qwen3-4B (58.0 vs. 57.9 Avg; 62.5 vs. 62.2 wAvg) and matching DAPO, GRESO, and Knapsack RL on Llama-3.2-3B using 2.4-2.8x less compute.

  2. MATH500 across Levels 1–5 (new): we added MATH500, which spans a broad difficulty distribution (Levels 1–5) well away from the competition floor. On Qwen2.5-Math-7B, sGPO reaches 82.6 at 8.9 EF - outperforming compute-matched uniform G=4 (80.9 at 8.5 EF), uniform G=2 (79.0 at 5.1 EF), and no-curriculum (81.0 at 8.9 EF), while matching unfiltered sGPO (82.3 at 16.7 EF) at nearly half the compute (see our response to Reviewer njLR for the full table). We will add MATH500 across all methods and both model scales to the main results table in the revised paper.


W2 (vanilla GRPO and dynamic-sampling FLOPs).

Thank you for this important point. We ran standard GRPO without dynamic sampling (G∈{2,4,8}G \in \{2, 4, 8\}, KL loss, symmetric clipping, sequence-level loss). As shown below, removing dynamic sampling does not make the baseline more efficient. At matched compute sGPO is clearly ahead (30.6 vs. 25.0 for the best GRPO run within 8 EF), and it reaches DAPO's best accuracy at 7.8 EF, where DAPO needs $\sim$18 EF. Shrinking the uniform group does not recover efficiency either: smaller groups are cheaper but plateau at lower accuracy (see also the matched-compute uniform G=4G=4 ablation in our response to Reviewer njLR).

Method≤ 4 EF≤ 8 EF≤ 16 EF≤ 32 EF
GRPO, G=2G=221.023.023.324.0
GRPO, G=4G=421.524.025.526.7
GRPO, G=8G=822.525.027.528.5
sGPO27.830.631.331.6

Removing dynamic sampling from GRESO likewise leaves uniform G = 8 throughout its cold-start phase (before reward history accumulates) and uniform G = 8 on all retained queries, whereas sGPO filters from step 1 and uses G∈2,4G \in {2, 4} on easier queries.

Online allocators like VIP and VIGOR excel on single-pass streaming or long, non-stationary runs by updating difficulty estimates online without an upfront pass. In our budget-constrained regime, however, sGPO offers two key advantages:

  • Upfront filtering vs. full 8|D| rollouts: Neither VIP nor VIGOR pre-filters prompts—both spend the full 8|D| rollouts/epoch (VIP requires a MiniLM GP with floor G≥3G \ge 3; VIGOR runs 4 sequential generation barriers/step). By contrast, sGPO uses zero auxiliary models, assigns G=2G=2 to 27.3% of queries, and uses one cheap inference pass to drop p^>0.75\hat{p} > 0.75 and 90% of p^=0\hat{p} = 0 upfront - cutting training FLOPs by 2.5x while preserving single-pass parallel generation.

  • Better hard-prompt coverage: because both VIP and VIGOR rank prompts by p(1−p)p(1-p), they concentrate most rollouts on medium difficulty prompts (p≈0.5p \approx 0.5) while under-sampling hard ones. We will continue this argument in the following paragraph.

Q1 (reconciling sGPO with VIP and XRPO)

  • VIP and XRPO (and sGPO) optimize for different compute regimes. Assuming a large 8∣D∣8|D| budget and unnormalized advantages (ai=ri−rˉa_i = r_i - \bar{r}), VIP focuses on "minimizing gradient variance": because p(1−p)p(1-p) peaks at p=0.5p=0.5 and vanishes as p→0p \to 0, VIP concentrates rollouts on medium-difficulty prompts and treats low-pp prompts - whose groups mostly return zero updates (0/G0/G, σr=0\sigma_r = 0) - as already low-variance rather than wasted compute. Under our 2.5x2.5x tighter budget, however, the primary bottleneck on hard prompts (p≈1/8p \approx 1/8) is learning signal discovery - getting at least one passing rollout so the group does not produce a zero update. Scaling rollouts inversely with pass rate (G≈1/pG \approx 1/p, so G=2G=2 at p=0.5p=0.5 and G=8G=8 at p=1/8p=1/8) lowers the probability of an all-fail group at p=1/8p=1/8 from (7/8)2≈0.77(7/8)^2 \approx 0.77 down to (7/8)8≈0.34(7/8)^8 \approx 0.34. This echoes XRPO's finding that uncertainty-based allocation overlooks sparsely rewarded hard prompts (quoting the XRPO's arxiv paper: "However, simply allocating more rollout budgets to the uncertain prompts will overlook the sparsely sampled and hard prompts.").

  • While VIP's "Inverse-Accuracy" experiment underperformed, this was likely because they did not filter out unsolvable (p=0p=0) prompts, which wasted compute on 37.6%37.6\% of DAPO-Math-17k; once we filter out 90%90\% of p^=0\hat{p}=0 prompts upfront and apply G≈1/p^G \approx 1/\hat{p} only to solvable prompts (p^>0\hat{p}>0), we have shown that we outperform a compute-matched uniform G=4G=4 baseline on 7B (31.0 vs. 29.5 on AIME24; 82.6 vs. 80.9 on MATH500).

  • 1-of-8 groups do not blow up gradient variance in normalized GRPO. Dividing rewards by the group standard deviation σr=rˉ(1−rˉ)\sigma_r = \sqrt{\bar{r}(1-\bar{r})} mathematically forces the mean squared advantage to equal 11 (1G∑i=1Gai2=1\frac{1}{G}\sum_{i=1}^G a_i^2 = 1) for any group with at least one pass and one fail. For example, a 1-of-8 group produces one large positive advantage (+7≈+2.65+\sqrt{7} \approx +2.65) and seven small negative ones (−1/7≈−0.38-1/\sqrt{7} \approx -0.38), whose squares average to 18[7+7(17)]=1\frac{1}{8}[7 + 7(\frac{1}{7})] = 1 - the exact same average squared weight as a balanced 4-of-8 group (18[4(1)2+4(−1)2]=1\frac{1}{8}[4(1)^2 + 4(-1)^2] = 1). Under equal rollout variance (as assumed in VIP), 1-of-8 and 4-of-8 groups inject the exact same gradient variance into Eq. (4). Finally, capping G≤8G \le 8 naturally bounds the maximum advantage at ∣ai∣≤7≈2.65|a_i| \le \sqrt{7} \approx 2.65 and preserves exact zero-sum updates (∑ai=0\sum a_i = 0). This avoids extreme spikes seen in Knapsack RL, which allocates up to G≈93−128G \approx 93-128). We will add this discussion to Section 3.2 and Appendix B.


We thank the reviewer again and would be glad to discuss further.

References:

Reviewer_njLROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DeliberateRL and its SGPO implementation for compute-efficient RLVR. It uses an offline estimate of the model's competence on each query to guide query selection, rollout group sizes, and curriculum ordering. The resulting schedule allocates different generation budgets to different queries and organizes training around the model's estimated ability to solve them. The main contribution is combining these decisions into a simple training procedure. Experiments on mathematical and scientific reasoning tasks evaluate its accuracy and computational cost, including the upfront profiling work.

Critical strengths
  1. I find it interesting that the method uses the model's competence on each query to choose the rollout group size. This connects the difficulty of a training example to the amount of compute spent on it and gives a straightforward way to tailor the generation budget.
  2. The GPU-hour comparisons include the time spent on offline profiling. I appreciate this, since profiling is a necessary part of the method and could otherwise make the reported training savings look larger than they are. Including it makes the comparison fairer.
Critical weaknesses
  1. Profiling requires generating and verifying multiple responses for every query before training starts. Although this cost is included in the reported comparisons, its overhead on substantially larger datasets or longer responses remains unclear. This upfront work may be difficult to amortize under a limited training budget, especially if only part of the profiled dataset is eventually used. More discussion of when the subsequent savings cover this cost would help clarify the method's practical scope.
  2. The component ablations in Table 4 are limited to one model and AIME24, and remove the components cumulatively. They do not fully separate the contributions of query selection, adaptive group sizing, and curriculum ordering. Uniform G=2 and G=4 controls would also help determine how much of the saving comes from simply using smaller groups, rather than the proposed allocation rule.
  3. The initial competence profile becomes outdated as the policy improves. Appendix J usefully shows that re-profiling after the first epoch performs worse than reusing the initial profile, but it changes query selection, group sizes, and curriculum ordering together. This leaves open whether the advantage persists over longer training, when a refresh would be useful, and whether updating only part of the profile could justify the extra cost.
Questions for authors
  1. How does the profiling overhead change with dataset size and response length, and under what training budgets do the later savings outweigh it? Could profiling only a subset retain most of the benefit?
  2. Could you add independent component ablations and uniform G=2/G=4 controls, ideally confirming the main findings on another benchmark?
  3. Beyond the epoch-boundary experiment in Appendix J, what criterion would you use to refresh the profile? Would a partial or less frequent refresh improve the accuracy-compute trade-off over longer training, including its additional cost?
2 comments
AuthorsOpenReview ↗

Response to Reviewer njLR

We thank the reviewer for the constructive review, and for noting that counting the profiling cost makes the comparison fairer. Our central claim is training efficiency, and we answer each point with that in mind.

W1, Q1 (profiling cost).

Thank you for this question; our cost accounting gives a clear answer. In our runs, profiling costs 1.6 of 8.9 EF (18% of wall-clock time, Appendix I) and is repaid within $\sim$0.2 epochs (Table 11). In general, profiling and training both scale linearly with dataset size and response length, so the FLOP accounting in Appendix M.5 gives a simple rule. Profiling the full dataset beats uniform G=8G=8 training once training covers at least a fraction N6(G−Gˉ)\frac{N}{6(G-\bar{G})} of its queries: about 35% here (N=8N=8, Gˉ=4.2\bar{G}=4.2). Below that, one should profile a random subset sized to the budget, as DOTS [1] does with a reference set; well-chosen subsets suffice for RL [2]. A subset loses nothing in allocation: as shown below, even 10% reproduces the full profile's bucket mix to within about a point. Our runs train for two full epochs, well above the threshold, so we profile everything, which also keeps the candidate pool identical to the baselines'. We will add this rule to Appendix E.

ProfiledUnsolved (p^=0\hat{p}=0)G=8G=8G=4G=4G=2G=2Trivial (p^>0.75\hat{p}>0.75)Gˉ\bar{G}
Full set37.6%14.7%11.3%27.3%9.1%4.18
50% random37.6±0.337.6 \pm 0.314.7±0.314.7 \pm 0.311.3±0.311.3 \pm 0.327.3±0.327.3 \pm 0.39.2±0.39.2 \pm 0.34.18±0.024.18 \pm 0.02
10% random37.7±1.137.7 \pm 1.114.7±0.714.7 \pm 0.711.2±0.911.2 \pm 0.927.2±0.827.2 \pm 0.89.1±0.99.1 \pm 0.94.18±0.054.18 \pm 0.05

W2, Q2: (independent ablations)

We appreciate this suggestion. We now remove each component individually from full sGPO, keeping the other two, and report AIME24 and MATH500, which spans a wider range of difficulty. Uniform G=4G=4 on the filtered set matches sGPO's compute (8.5 vs. 8.9 EF) but scores lower, so the gain comes from the allocation rule, not from smaller groups. Without filtering, compute nearly doubles for no gain in accuracy, and without the curriculum, accuracy drops at the same cost. Only all three together reach DAPO-level accuracy at 2.5×2.5\times less compute. We will add these ablations alongside Table 4.

ConfigurationTotal EFAIME24MATH500
sGPO (filtering + adaptive GG + curriculum)8.931.082.6
− adaptive GG: uniform G=2G=25.128.479.0
− adaptive GG: uniform G=4G=48.529.580.9
− filtering16.730.682.3
− curriculum8.929.681.0

W3, Q3 (static profile).

Thank you for this question. Keeping the profile fixed is a deliberate design choice for the short-horizon, budget-constrained regime sGPO targets, and our results support it. Appendix J shows that the static profile outperforms naive re-profiling. Re-applying the filter after an epoch removes exactly the queries the policy has just learned, which shrinks and narrows the training set, so a fresher estimate does not by itself yield a better practice distribution. The static profile also holds up well beyond two epochs: training stays stable for about 8 epochs (32 EF) with no collapse, and the queries it excluded are not forgotten (see our response to Reviewer sYrC).

For longer horizons, the lesson from Appendix J is that refresh should be local and non-ejecting rather than global. Because every decision in sGPO comes from p^\hat{p}, this needs no extra profiling: training rollouts already give each query a running success rate, so a query can step down a bucket only when its running rate crosses a bucket boundary (for example, the G=8G=8 phase already averages 0.26–0.400.26\text{–}0.40 reward during training in Appendix L, Fig. 9 - above the p^≤1/8\hat{p} \le 1/8 boundary for G=8G=8, so its core queries can step down to G=4G=4 or G=2G=2). We will add this discussion to Appendix J, and we plan a dedicated follow-up study of adaptive refresh for long-horizon training, covering when to refresh, which queries to re-estimate, and how often.


We thank the reviewer again; these suggestions substantially sharpened the paper, and we would be glad to discuss further.

References

[1] Sun, Y., Shen, J., Wang, Y., Chen, T., Wang, Z., Zhou, M., and Zhang, H. Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

[2] Li, X., Zou, H., and Liu, P. LIMR: Less is More for RL Scaling. arXiv preprint arXiv:2502.11886, 2025.

Thank you for the response. It addresses some of my concerns, and I will increase the rating.

Reviewer_sYrCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper describes a prioritization method for training queries. The key contribution is an extension of GRPO, called DeliberateRL with sorted-Group Policy Optimization (sGPO), where the group size GG is calculated separately for each query qq. In particular, the authors empirically calculate the success probability p^(q)\hat{p}(q) of policy pθ0p_{\theta_0} given query qq and pick group size to be G(q):=max⁡(2,⌊1/p^(q)⌋)G(q) := \max\left({2,\lfloor 1/\hat{p}(q) \rfloor}\right). The group size is theoretically justified using direct derivation from the probability improvement Pr(n=1∣G+1,p)/Pr(n=1∣G,p)\mathrm{Pr}(n=1\vert G+1,p) / \mathrm{Pr}(n=1\vert G,p). Additionally, the queries which are too easy are excluded from the training dataset, making the training focus on more difficult queries DlearnD_{\mathrm{learn}} (and a fraction of D0D_0) instead. Finally, the authors describe easy-to-hard curriculum and demonstrate measurable efficiency improvement (measured in GPUh) without adversely affecting the performance (measured on AIME, HMMT, etc).

Critical strengths
  • Filtering out queries which already have large success rate reduces redundant training steps on these problems over short training horizons.
  • By separately calculating the group size G(q)G(q) for each query qq, the number of rollouts is adjusted depending on the estimated difficulty of the problem/query. Therefore, easier problems receive smaller rollout budgets, while more challenging ones receive larger ones.
  • Empirical evidence from two-epoch training (Table 2) suggests that over short training horizon the performance (as measured by accuracy on AIME and HMMT benchmarks) remains unaffected, while substantially improving computational efficiency (as measured in GPUh & FLOPS).
Critical weaknesses
  • The training horizon for DeliberateRL/sGPO is limited to only two epochs, whereas the baseline methods (Knapsack RL, DAPO, GRESO, etc) typically have significantly longer training runs. My main concern is that the current experiments do not necessarily establish that the approach remains stable over longer training runs. For example, it is unclear whether the performance remains stable on easier problems (which have been excluded from the training dataset) after training the model on more difficult ones, particularly after ≫2\gg 2 epochs.
  • The empirical success rate p^(q)\hat{p}(q) is estimated using the initial policy pθ0p_{\theta_0} and remains frozen throughout the training. While some limited experiments, where p^(q)\hat{p}(q) is re-profiled at the end of each epoch, are reported in Appendix J (Fig. 7), the training is limited to only two epochs, which limits any conclusions about the stability of the approach over a longer training horizon.
Questions for authors
  1. Could the authors provide experimental evidence of DeliberateRL/sGPO's stability over training runs which are substantially longer than two epochs?
  2. Could the authors provide difficulty-conditioned evaluation of the performance metrics reported in Table 2? In particular, it is unclear whether the observed improvements come from solving previously difficult (or unsolved D0D_0) problems or from the problems that the initial policy could already solve.
1 comment
AuthorsOpenReview ↗

Response to Reviewer sYrC

We thank the reviewer for the thoughtful review and the clear questions.

W1, W2, Q1 (training horizon).

Thank you for raising this point.

First, to clarify how our horizon compares with prior work: while it is true that R1-Zero-style runs from 32B+ general base models train over many passes as long-CoT emerges from near-zero initial accuracy, 1–3 epochs is in fact the standard horizon across recent RLVR rollout-allocation studies on 1.5B–7B models - VIP trains all models on DAPO-Math-17k for 2 epochs; XRPO trains for 1-3 epochs; and VIGOR trains for 2 epochs at 7B (and 3 epochs at 1.5B/3B) after showing across a 7-epoch study on Qwen2.5-1.5B/3B/7B that "the first epoch alone can already account for around 80% of the final validation-score gain" at 1.5B/3B and matches "the total gain from the following six epochs" at 7B before validation gains flatten; similarly, Knapsack RL captures most of its gains within the first 3–5 epochs.

Second, our claim is about training efficiency under a constrained compute budget, which the paper scopes explicitly ("a short-horizon strategy", Section 1). The appropriate test is therefore accuracy at equal compute, so we compare the best AIME24 accuracy each method reaches within a fixed budget, counting sGPO's profiling. To address stability directly, we also continue sGPO training. As shown below, sGPO reaches DAPO's best accuracy (29.8) at 7.8 EF, where DAPO needs ~18 EF, and Knapsack RL's best (31.1) at 8.3 EF, where Knapsack RL needs 19 EF: about 2.3× less compute in both cases. Over the longer horizon its accuracy remains stable, with no collapse. We will state this budget-constrained scope more prominently in the introduction.

Method≤ 4 EF≤ 8 EF≤ 16 EF≤ 32 EF
DAPO23.726.028.929.8
Knapsack RL23.828.430.331.1
sGPO (ours)27.830.631.331.6

W1: For example, it is unclear whether the performance remains stable on easier problems...

Thank you for this question. sGPO never trains on the 1,285 queries it profiles as high-success, so we re-profiled exactly those queries with later checkpoints. As shown below, competence on them does not degrade. Their mean success rate goes from approximately 0.80 at profiling to 0.92–0.95, and at least 93.5% remain high-success across the full compute range (4–32 EF), with at most 1% falling to p̂ ≤ 0.25. Excluding easy queries therefore saves compute without costing performance on them. We will add this analysis to Appendix F.

Compute4 EF8 EF16 EF32 EF
Mean p̂0.920.930.940.95
Still high-success93.5%94.0%94.0%95.0%
Dropped to p̂ ≤ 0.250.3%0.8%1.0%1.0%

Q2: Could the authors provide difficulty-conditioned evaluation of the performance metrics reported in Table 2?

Thank you for suggesting this breakdown. We bin the 153 Table 2 test problems by the base model's own accuracy and report each method within each bin. As shown below, the gains come mainly from problems the base model can already partly solve, for all methods. This is where sGPO concentrates its rollout budget: it trains on queries with 0 < p̂ ≤ 0.75 and sizes each group for about one expected success. sGPO matches or exceeds the baselines in every bin while using 2.3–2.5× less compute; its clearest difference is on the medium/easy bin. On problems the base model never solves (76% of the test set), accuracy stays low for every method. sGPO nonetheless solves at least once 13 of these 117 problems, against 9 for DAPO and 10 for Knapsack RL. Overall accuracy is comparable across methods; the difference is the compute needed to reach it. We will add this difficulty-conditioned version of Table 2 to the appendix.

Bin (base avg@16)# problemsBaseDAPOKnapsack RLsGPO
Unsolved (0)1170.01.51.51.8
↳ solved at least once (of 117)091013
Hard (0, 0.25]179.927.527.928.5
Medium/Easy (> 0.25)1943.869.470.573.7
All (Table 2 w Avg)1536.512.813.013.7
Total compute (EF)–22.320.18.9

We hope these additions address the reviewer's concerns.

References:

Comments on the paper

1 comment

We thank the Area Chair and all three reviewers for their constructive feedback and for recognizing sGPO's 2.5-2.9x FLOP and 3.0-3.7x GPU-hour reductions across mathematical and scientific reasoning. Below is a brief summary of our new experiments and analyses:


(i) Extended 32-EF (~8-epoch) horizon and equal-compute speedup (R1, R2): Across recent 1.5B–7B RLVR rollout-allocation studies (VIP, XRPO, VIGOR), 1–3 epochs is the standard horizon because gains are heavily front-loaded. Counting its 1.6 EF profiling cost, sGPO already leads inside Epoch 1 (≤4\le 4 EF: 27.8 vs. 23.7 DAPO / 23.8 Knapsack RL on AIME24), reaches DAPO's peak performance at 7.8 EF (vs. 18 EF) and Knapsack RL's peak (31.1) at 8.3 EF (vs. 19 EF; ~2.3x less compute). sGPO stays stable out to 32 EF (31.6) with no collapse under the frozen profile.


(ii) No forgetting on excluded easy queries (R1, R2): Re-evaluating the 1,285 excluded high-success queries (p^>0.75\hat{p} > 0.75) up to 32 EF shows zero degradation - their mean pass rate rises from ~0.8 to 0.92–0.95, with 93.5–95.0% remaining at p^>0.75\hat{p} > 0.75.


(iii) Difficulty-conditioned evaluation & MATH500 (R1, R2, R3): Across all 153 Table 2 problems binned by base avg@16, sGPO matches or exceeds DAPO and Knapsack RL in every difficulty bin at 2.3-2.5x less compute (solving 13/117 base-unsolved problems vs. 9 and 10). On MATH500 (Levels 1–5), sGPO reaches 82.6 at 8.9 EF, outperforming compute-matched uniform G=4G=4 (80.9, 8.5 EF), uniform G=2G=2 (79.0, 5.1 EF), and no-curriculum (81.0, 8.9 EF).


(iv) Leave-one-out ablations and Vanilla GRPO without dynamic sampling (R2, R3): Removing adaptive GG (uniform G=4: 29.5 AIME24 / 80.9 MATH500; uniform G=2: 28.4 / 79.0) or the curriculum (29.6 / 81.0) lowers accuracy, while removing filtering nearly doubles cost (16.7 EF) for no gain (30.6 / 82.3). New Vanilla GRPO runs (G∈{2,4,8}G \in \{2, 4, 8\}, no dynamic sampling) trail sGPO at every budget (25.0 best GRPO vs. 30.6 sGPO within ≤8\le 8 EF; G=2,4G=2, 4 plateau at 24.0 and 26.7 even at 32 EF).


(v) Theoretical reconciliation (VIP, VIGOR, XRPO) (R3): Whereas VIP minimizes variance on unnormalized advantages (ai=ri−rˉa_i = r_i - \bar{r}) at full 8∣D∣8|D| budget, under a 2.5x tighter budget G≈1/pG \approx 1/p cuts all-fail (0/G0/G) waste on hard prompts (77% to 34% at p=1/8p=1/8).


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.