- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies zero-variance prompts in GRPO, where all-correct or all-incorrect groups yield no group-relative advantage. The authors propose OAGS, which tracks per-prompt pass rates, samples candidates from a Gaussian curriculum around a moving target pass rate, and retains mixed-outcome groups for standard GRPO updates. On Qwen3-8B and Qwen3-8B-Base trained on DAPO-Math, OAGS improves macro-average Acc@8/Pass@8 over GRPO, Dynamic Sampling, GRESO, and DEPO.
Critical strengths
-
Zero-variance groups are a real efficiency bottleneck in GRPO, and decoupling candidate probing from update-batch selection so that discarded rollouts still refresh pass-rate estimates is a sensible and practical design.
-
The paper reports both rollout counts and wall-clock runtime, and compares against Dynamic Sampling, GRESO, and DEPO across two model initializations rather than only vanilla GRPO.
Critical weaknesses
-
According to Figure 3b and 3c, OAGS generates as many rollouts and needs more training time than GRPO (48.6 h vs. 33.9 h) at step 267. Thus, it is unfair to compare their performance in Table 1 and Table 2 at the same update step without a rollout- or time-matched GRPO baseline. Besides, the experiments are only conducted on one training dataset (DAPO-Math) and one model scale (8B), which is insufficient for demonstrating generalization.
-
Several core design choices in the Gaussian curriculum appear heuristic and are not well justified theoretically or empirically. In particular, Table 2 shows that removing tail reservation ("w/o Tail Reservation") outperforms the full OAGS model on both metrics (49.2 / 65.1 vs. 48.6 / 64.5), which contradicts the paper's core--tail design in Section 3.2. Meanwhile, rather than relying on hand-crafted Gaussian proposals and square-root pacing rules ( in Eq. 12), recent studies [1-6] have formulated online difficulty tracking, dynamic budget allocation, and active prompt selection through more principled predictive, bandit, or optimization frameworks. The paper neither discusses nor compares against these closely related formulations, leaving the conceptual advantage of a heuristic Gaussian curriculum unclear.
[1] Small generalizable prompt predictive models can steer efficient RL post-training of large reasoning models
[2] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
[3] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
[4] BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
[5] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
[6] Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
Questions for authors
-
Figure 3b and 3c show that OAGS uses 2x the rollouts and 48.6 hours versus 33.9 hours for GRPO at step 267. Can the authors report GRPO at matched wall-clock time or matched total rollouts in Table 1 and Table 2?
-
In Table 2, why does removing the dedicated tail exploration quota ("w/o Tail Reservation") outperform full OAGS by +0.6 Acc@8 and +0.6 Pass@8? If the core-tail split hurts performance, why is it kept as a core component of OAGS?
-
In Eq. (9), retaining candidate groups ranked by realized outcome variance conditions the training batch on the sampled outcomes rather than only the prompt . For a prompt with a low true pass rate, this selectively over-samples "lucky" groups containing rare successes, making the empirical group baseline systematically higher than the true expected reward under the behavior policy . Does this post-rollout outcome selection introduce bias into the GRPO gradient when no importance correction is applied?
-
Can the authors discuss the conceptual advantages of their heuristic Gaussian proposal and schedule over more principled predictive or bandit rollout-allocation methods [1-6], and empirically compare with at least some of them?
7 comments
Tail Ablation and Motivation for Gaussian Curriculum Sampling
We appreciate the concern that the Gaussian and core–tail choices appear heuristic. We agree that the fixed 31.73% tail quota is not empirically optimal and provide a new ablation together with a more precise conceptual justification.
1. Strict Core experiment: tail exploration vs. fixed reservation. We report complete Qwen3-8B-Base results below. Each entry denotes Acc/Pass@8.
| Method | Minerva | AMC23 | MATH500 | AIME24 | AIME25 | Olympiad | AVG |
|---|---|---|---|---|---|---|---|
| Strict Core (Tail-0) | 41.6/52.9 | 68.4/90.0 | 85.5/94.6 | 21.7/36.7 | 17.9/33.3 | 52.1/66.2 | 47.9/62.3 |
| OAGS | 42.2/54.0 | 68.4/92.5 | 86.1/94.0 | 25.0/43.3 | 18.8/36.7 | 50.9/66.4 | 48.6/64.5 |
| w/o Tail Reservation | 41.2/53.7 | 70.3/92.5 | 86.2/94.0 | 24.2/50.0 | 22.5/33.3 | 50.8/67.0 | 49.2/65.1 |
Importantly, Strict Core and w/o Tail Reservation represent different sampling mechanisms:
- OAGS Full: The fixed oversampling budget is nominally divided into 175 Gaussian-core and 81 tail candidates (68.27%/31.73%), with capacity-based cross-filling when needed.
- w/o Tail Reservation (manuscript): Removes the fixed 31.73% tail quota. Instead, all candidate positions are allocated across eligible pass-rate buckets according to Gaussian weights. Consequently, tail prompts remain eligible and can still be sampled. This is flexible Gaussian allocation, not the removal of tail exploration.
- Strict Core (Tail-0): Sets the target tail budget to zero and concentrates all candidates within the Gaussian core whenever sufficient core prompts are available. Tail candidates are considered only if the number of core candidates is insufficient to reach .
The results show that Full OAGS outperforms Strict Core by +0.7 Acc@8 and +2.2 Pass@8 points. The no-reservation variant further outperforms Strict Core by +1.3/+2.8 points and Full OAGS by +0.6/+0.6 points.
These findings support distinguishing tail coverage from fixed tail reservation. Restricting candidate allocation to the Gaussian core reduces performance, whereas allowing Gaussian-weighted sampling across the full eligible capability space performs best in our experiments. Thus, exploration outside the current target region is useful, but the originally prescribed 31.73% reservation is not necessarily optimal.
The nominal 31.7% tail reservation is motivated by the standard Gaussian distribution, where approximately 68.3% of the probability mass lies within one standard deviation of the mean and 31.7% lies in the tails. This property remains invariant as the Gaussian center and bandwidth evolve during training. We adopt this ratio as a simple, distribution-motivated heuristic for allocating the candidate budget between core exploitation and tail exploration, rather than claiming it is optimal or reflects the actual mass of the discrete prompt distribution. The stronger performance without explicit tail reservation suggests that this fixed allocation may not achieve the best exploration–exploitation balance. We will clarify this distinction in the revised manuscript and leave adaptive core–tail budget allocation to future work.
This motivation is further supported by our tracker analysis: we observed 10,188 non-ZVP-to-ZVP transitions across successive prompt observations, while 1,563 initially ZVP prompts later became non-ZVP before returning to ZVP. Although finite-group sampling contributes to these changes, they illustrate why permanently excluding historically uninformative prompts may be undesirable.
2. Why Gaussian proposal and square-root pacing?
Under conditionally independent binary outcomes with success probability , the predicted zero-variance probability is where the probability of obtaining a mixed-outcome group is maximized at . This motivates prioritizing intermediate capability levels, consistent with the findings of Gao et al. (Prompt Curriculum Learning). The Gaussian proposal provides a simple, interpretable mechanism, where determines the target capability region and controls its coverage.
The schedules and jointly control the curriculum's focus and coverage. The concave square-root schedule accelerates progression toward harder prompts early in training while slowing it later, whereas the feedback controller adapts based on observed capability changes. Ablations yield 48.2/63.2 without the controller and 48.3/62.6 with linear pacing, compared with 48.6/64.5 for Full OAGS (Acc@8/Pass@8).
We do not claim theoretical optimality for either Gaussian allocation or square-root pacing. The uniform-proposal ablation (47.8/61.4 versus 48.6/64.5) provides empirical support for capability-guided allocation, rather than a guarantee of optimality.
Clarification of Outcome-Conditioned Selection and GRPO Gradient Bias
We thank the reviewer for highlighting the selection-bias issue in Eq. (9). We acknowledge that selecting groups based on realized outcome variance introduces outcome-dependent selection bias relative to the original unfiltered, uniform-prompt GRPO objective. We do not claim otherwise.
1. Why selection bias arises. Let denote the number of successful responses, assuming conditionally independent Bernoulli outcomes with true success probability . For a simplified selector retaining only mixed-outcome groups, , we have which generally differs from . For example, when and , the conditional mean increases to approximately 0.149, compared with the true success probability of 0.05. Although OAGS employs a more complex Top- selection over candidates, this example illustrates the reviewer's concern that rare successful outcomes on difficult prompts may be disproportionately retained.
2. What OAGS actually optimizes. As explicitly defined in Section 3.4 and Eq. (14), OAGS optimizes the standard GRPO clipped surrogate under the selection-induced distribution , rather than claiming an unbiased estimate of the original uniform-prompt objective. All responses are freshly generated by the current behavior policy, without historical trajectory replay. However, on-policy generation does not eliminate outcome-dependent selection bias. The standard PPO/GRPO likelihood ratio accounts for policy probability changes but does not correct for the additional group-selection probability. We therefore acknowledge that OAGS changes the effective training distribution without selection-specific importance correction.
3. Why outcome-based selection remains useful. Our goal is to allocate a fixed optimization budget toward groups providing nonzero group-relative reward signals. As defined in Eq. (9), the empirical variance is zero for all-correct or all-incorrect groups and maximized when , thereby prioritizing mixed-outcome groups with more balanced success–failure compositions. This design is motivated by ICLR26 paper, Prompt Curriculum Learning for Efficient LLM Post-Training, which empirically identifies intermediate-difficulty prompts as particularly favorable for training efficiency and gradient quality. Accordingly, OAGS uses realized outcome variance to prioritize informative groups among already-generated candidates, rather than relying solely on historical difficulty estimates. This trades fidelity to the original sampling distribution for a higher concentration of informative training groups. Our empirical results support this efficiency-oriented design, but do not establish that the resulting selection bias is universally harmless or theoretically optimal.
We will clarify the empirical motivation for variance-based selection, the selection-induced distribution shift, its distinction from policy importance weighting, and the corresponding limitations in the revised manuscript.
4. Recent studies [1–6]. We agree these works deserve discussion and have initiated matched-setting reproductions of DPS [2], BOTS [4], and AR3PO [5]. Currently, these reproductions are still running: DPS and BOTS are at step 50, while AR3PO is at step 144 . Given the computational time required, we may not obtain final results for DPS and BOTS before the rebuttal deadline. We expect to provide completed results for AR3PO by the deadline. We will expand the related work to discuss references [1–6], and we will include completed, comparable evaluations for DPS [2], BOTS [4], and AR3PO [5] in the revised manuscript.
[2] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
[4] BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
[5] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
Rollout-Matched GRPO Results and Cross-Scale Evaluation
We thank the reviewer for the detailed assessment and apologize for responding close to the deadline.
1. New rollout-matched GRPO. We agree that matching only policy updates cannot establish computational efficiency. We therefore trained Vanilla GRPO for 536 updates on Qwen3-8B, nearly matching OAGS's training rollout budget. Entries below are Acc@8 / Pass@8 (%).
At nearly equal training rollout budgets (GRPO uses 0.37% more), OAGS remains ahead by +1.9 Acc@8 and +3.3 Pass@8 points. Regarding wall-clock time, although OAGS takes longer (48.6 h) than GRPO (33.9 h) at the same 267 updates, the rollout-matched GRPO requires 70 h in total. Since gradient update time outweighs rollout time, OAGS is actually more time-efficient when computational budgets are aligned.
2. Initialization-inclusive accounting. Each target model initializes its tracker using its own initial checkpoint and four responses per prompt, adding 4∣D∣≈68K responses. OAGS therefore uses approximately 614.8K total responses (546.8K training + 68K initialization), compared with 826.4K training responses for Dynamic Sampling and 628.9K for GRESO at 267 updates.
We will distinguish update-matched, training-rollout-matched, and initialization-inclusive budgets in the revision.
3. New cross-scale evidence. We also evaluated Qwen3-1.7B on the same DAPO-Math-17K training data and six benchmarks (Acc@8/Pass@8):
OAGS improves over GRPO by +1.7/+2.7 points on Qwen3-1.7B, supporting cross-scale effectiveness within Qwen3. Because GRPO incurs significant training overhead, we will add the cross-dataset experiment results (using MATH-lighteval as the training set) in the revised manuscript.
4. Additional training- and evaluation-seed robustness.
We independently trained Qwen3-8B OAGS with two additional seeds. Each entry denotes Acc@8 / Pass@8 (%).
We additionally evaluated with different sampling seeds:
Across the three independent training runs, OAGS achieves 61.30 ± 0.20 Acc@8 and 74.97 ± 0.32 Pass@8 (mean ± sample standard deviation). Two additional evaluation-seed results are 61.2/75.0 and 61.4/75.3. These provide complementary evidence of evaluation robustness.
Overall, the new evidence supports capability-aware Gaussian candidate allocation and continued tail coverage, while revealing that the original fixed tail allocation may be improved. We thank the reviewer for helping clarify these distinctions. Across three independent Qwen3-8B training runs, OAGS averages 61.30 ± 0.20 Acc@8 and 74.97 ± 0.32 Pass@8 (sample SD), providing initial robustness evidence.