acceptodds
Under review as a conference paper at ICLR 2027

Objective-Aware Gaussian Sampling for Zero-Variance Prompts in LLM Reinforcement Learning

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an efficient reinforcement learning paradigm for fine-tuning large language models, yet its effectiveness can be hindered by zero-variance prompts—cases where all sampled responses yield identical rewards, producing zero group-relative advantages while still incurring generation costs. We introduce Objective-Aware Gaussian Sampling (OAGS), a prompt-level curriculum framework that dynamically reallocates the rollout budget toward prompts in an evolving target pass-rate region. OAGS estimates prompt-specific pass rates from observed outcome rewards, constructs Gaussian candidate proposals centered on the current curriculum target, and adaptively adjusts the curriculum via a progress-controlled pacing mechanism. By concentrating training on informative prompts while preserving controlled exploration of harder and easier regions, OAGS prioritizes non-zero-variance response groups within a fixed candidate budget. Our approach retains the GRPO advantage estimator and clipped surrogate while adapting prompt sampling and post-rollout group selection. Experimental results show improved macro-average performance across six mathematical reasoning benchmarks compared to standard GRPO and the evaluated data-selection baselines, with fewer training rollouts than Dynamic Sampling and GRESO at matched policy-update counts.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Data Selection, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XWh4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies zero-variance prompts in GRPO, where all-correct or all-incorrect groups yield no group-relative advantage. The authors propose OAGS, which tracks per-prompt pass rates, samples candidates from a Gaussian curriculum around a moving target pass rate, and retains mixed-outcome groups for standard GRPO updates. On Qwen3-8B and Qwen3-8B-Base trained on DAPO-Math, OAGS improves macro-average Acc@8/Pass@8 over GRPO, Dynamic Sampling, GRESO, and DEPO.

Critical strengths
  1. Zero-variance groups are a real efficiency bottleneck in GRPO, and decoupling candidate probing from update-batch selection so that discarded rollouts still refresh pass-rate estimates is a sensible and practical design.

  2. The paper reports both rollout counts and wall-clock runtime, and compares against Dynamic Sampling, GRESO, and DEPO across two model initializations rather than only vanilla GRPO.

Critical weaknesses
  1. According to Figure 3b and 3c, OAGS generates 2×2\times as many rollouts and needs 1.43×1.43\times more training time than GRPO (48.6 h vs. 33.9 h) at step 267. Thus, it is unfair to compare their performance in Table 1 and Table 2 at the same update step without a rollout- or time-matched GRPO baseline. Besides, the experiments are only conducted on one training dataset (DAPO-Math) and one model scale (8B), which is insufficient for demonstrating generalization.

  2. Several core design choices in the Gaussian curriculum appear heuristic and are not well justified theoretically or empirically. In particular, Table 2 shows that removing tail reservation ("w/o Tail Reservation") outperforms the full OAGS model on both metrics (49.2 / 65.1 vs. 48.6 / 64.5), which contradicts the paper's core--tail design in Section 3.2. Meanwhile, rather than relying on hand-crafted Gaussian proposals and square-root pacing rules (μt\mu_t in Eq. 12), recent studies [1-6] have formulated online difficulty tracking, dynamic budget allocation, and active prompt selection through more principled predictive, bandit, or optimization frameworks. The paper neither discusses nor compares against these closely related formulations, leaving the conceptual advantage of a heuristic Gaussian curriculum unclear.

[1] Small generalizable prompt predictive models can steer efficient RL post-training of large reasoning models

[2] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

[3] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

[4] BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning

[5] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse

[6] Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

Questions for authors
  1. Figure 3b and 3c show that OAGS uses 2x the rollouts and 48.6 hours versus 33.9 hours for GRPO at step 267. Can the authors report GRPO at matched wall-clock time or matched total rollouts in Table 1 and Table 2?

  2. In Table 2, why does removing the dedicated tail exploration quota ("w/o Tail Reservation") outperform full OAGS by +0.6 Acc@8 and +0.6 Pass@8? If the core-tail split hurts performance, why is it kept as a core component of OAGS?

  3. In Eq. (9), retaining candidate groups ranked by realized outcome variance V(x)=P(x)(1−P(x))V(x) = P(x)(1 - P(x)) conditions the training batch on the sampled outcomes rather than only the prompt xx. For a prompt with a low true pass rate, this selectively over-samples "lucky" groups containing rare successes, making the empirical group baseline Rˉ\bar{R} systematically higher than the true expected reward under the behavior policy πold\pi_{\text{old}}. Does this post-rollout outcome selection introduce bias into the GRPO gradient when no importance correction is applied?

  4. Can the authors discuss the conceptual advantages of their heuristic Gaussian proposal and ct\sqrt{c_t} schedule over more principled predictive or bandit rollout-allocation methods [1-6], and empirically compare with at least some of them?

7 comments
AuthorsOpenReview ↗

Rollout-Matched GRPO Results and Cross-Scale Evaluation

We thank the reviewer for the detailed assessment and apologize for responding close to the deadline.

1. New rollout-matched GRPO. We agree that matching only policy updates cannot establish computational efficiency. We therefore trained Vanilla GRPO for 536 updates on Qwen3-8B, nearly matching OAGS's training rollout budget. Entries below are Acc@8 / Pass@8 (%).

MethodUpdatesTrain rolloutsMinervaAMC23MATH500AIME24AIME25OlympiadAVG
GRPO267273,40845.8/56.684.1/95.091.0/95.840.0/60.033.3/53.358.3/69.658.8/71.7
GRPO (rollout matched)536548,86446.3/56.680.9/95.089.7/96.042.5/60.036.2/53.359.5/70.759.2/71.9
OAGS267546,81647.8/58.586.9/97.590.5/95.451.3/76.731.7/53.358.4/70.161.1/75.2

At nearly equal training rollout budgets (GRPO uses 0.37% more), OAGS remains ahead by +1.9 Acc@8 and +3.3 Pass@8 points. Regarding wall-clock time, although OAGS takes longer (48.6 h) than GRPO (33.9 h) at the same 267 updates, the rollout-matched GRPO requires 70 h in total. Since gradient update time outweighs rollout time, OAGS is actually more time-efficient when computational budgets are aligned.

2. Initialization-inclusive accounting. Each target model initializes its tracker using its own initial checkpoint and four responses per prompt, adding 4∣D∣≈68K4|\mathcal D|\approx68\mathrm K responses. OAGS therefore uses approximately 614.8K total responses (546.8K training + 68K initialization), compared with 826.4K training responses for Dynamic Sampling and 628.9K for GRESO at 267 updates.

We will distinguish update-matched, training-rollout-matched, and initialization-inclusive budgets in the revision.

3. New cross-scale evidence. We also evaluated Qwen3-1.7B on the same DAPO-Math-17K training data and six benchmarks (Acc@8/Pass@8):

MethodMinervaAMC23MATH500AIME24AIME25OlympiadAverage
GRPO34.0/47.460.0/90.079.3/89.619.2/30.019.6/33.342.7/59.342.5/58.3
DS34.1/48.552.8/85.076.2/89.615.8/30.012.9/33.339.2/60.338.5/57.8
GRESO34.6/48.555.3/75.079.6/90.019.2/40.015.8/23.342.6/58.841.2/55.9
OAGS35.8/48.560.9/90.082.4/92.020.8/40.020.4/33.345.1/62.144.2/61.0

OAGS improves over GRPO by +1.7/+2.7 points on Qwen3-1.7B, supporting cross-scale effectiveness within Qwen3. Because GRPO incurs significant training overhead, we will add the cross-dataset experiment results (using MATH-lighteval as the training set) in the revised manuscript.

4. Additional training- and evaluation-seed robustness.

We independently trained Qwen3-8B OAGS with two additional seeds. Each entry denotes Acc@8 / Pass@8 (%).

Training runMinervaAMC23MATH500AIME24AIME25OlympiadAverage
Original47.8/58.586.9/97.590.5/95.451.3/76.731.7/53.358.4/70.161.1/75.2
Seed 4047.7/59.286.9/97.590.8/95.451.3/73.333.8/56.758.4/68.761.5/75.1
Seed 4148.0/59.686.2/95.090.8/96.051.3/73.332.9/53.358.3/70.261.3/74.6

We additionally evaluated with different sampling seeds:

Evaluation seedMinervaAMC23MATH500AIME24AIME25OlympiadAVG
Seed 4047.6/58.585.3/95.091.0/96.251.3/76.733.8/53.758.4/69.961.2/75.0
Seed 4147.8/58.886.2/97.590.9/95.650.8/73.334.2/56.758.5/70.161.4/75.3

Across the three independent training runs, OAGS achieves 61.30 ± 0.20 Acc@8 and 74.97 ± 0.32 Pass@8 (mean ± sample standard deviation). Two additional evaluation-seed results are 61.2/75.0 and 61.4/75.3. These provide complementary evidence of evaluation robustness.

Overall, the new evidence supports capability-aware Gaussian candidate allocation and continued tail coverage, while revealing that the original fixed tail allocation may be improved. We thank the reviewer for helping clarify these distinctions. Across three independent Qwen3-8B training runs, OAGS averages 61.30 ± 0.20 Acc@8 and 74.97 ± 0.32 Pass@8 (sample SD), providing initial robustness evidence.

AuthorsOpenReview ↗

Tail Ablation and Motivation for Gaussian Curriculum Sampling

We appreciate the concern that the Gaussian and core–tail choices appear heuristic. We agree that the fixed 31.73% tail quota is not empirically optimal and provide a new ablation together with a more precise conceptual justification.

1. Strict Core experiment: tail exploration vs. fixed reservation. We report complete Qwen3-8B-Base results below. Each entry denotes Acc/Pass@8.

MethodMinervaAMC23MATH500AIME24AIME25OlympiadAVG
Strict Core (Tail-0)41.6/52.968.4/90.085.5/94.621.7/36.717.9/33.352.1/66.247.9/62.3
OAGS42.2/54.068.4/92.586.1/94.025.0/43.318.8/36.750.9/66.448.6/64.5
w/o Tail Reservation41.2/53.770.3/92.586.2/94.024.2/50.022.5/33.350.8/67.049.2/65.1

Importantly, Strict Core and w/o Tail Reservation represent different sampling mechanisms:

  • OAGS Full: The fixed oversampling budget M=256M=256 is nominally divided into 175 Gaussian-core and 81 tail candidates (68.27%/31.73%), with capacity-based cross-filling when needed.
  • w/o Tail Reservation (manuscript): Removes the fixed 31.73% tail quota. Instead, all MM candidate positions are allocated across eligible pass-rate buckets according to Gaussian weights. Consequently, tail prompts remain eligible and can still be sampled. This is flexible Gaussian allocation, not the removal of tail exploration.
  • Strict Core (Tail-0): Sets the target tail budget to zero and concentrates all MM candidates within the Gaussian core whenever sufficient core prompts are available. Tail candidates are considered only if the number of core candidates is insufficient to reach MM.

The results show that Full OAGS outperforms Strict Core by +0.7 Acc@8 and +2.2 Pass@8 points. The no-reservation variant further outperforms Strict Core by +1.3/+2.8 points and Full OAGS by +0.6/+0.6 points.

These findings support distinguishing tail coverage from fixed tail reservation. Restricting candidate allocation to the Gaussian core reduces performance, whereas allowing Gaussian-weighted sampling across the full eligible capability space performs best in our experiments. Thus, exploration outside the current target region is useful, but the originally prescribed 31.73% reservation is not necessarily optimal.

The nominal 31.7% tail reservation is motivated by the standard Gaussian distribution, where approximately 68.3% of the probability mass lies within one standard deviation of the mean and 31.7% lies in the tails. This property remains invariant as the Gaussian center μt\mu_t and bandwidth σt\sigma_t evolve during training. We adopt this ratio as a simple, distribution-motivated heuristic for allocating the candidate budget between core exploitation and tail exploration, rather than claiming it is optimal or reflects the actual mass of the discrete prompt distribution. The stronger performance without explicit tail reservation suggests that this fixed allocation may not achieve the best exploration–exploitation balance. We will clarify this distinction in the revised manuscript and leave adaptive core–tail budget allocation to future work.

This motivation is further supported by our tracker analysis: we observed 10,188 non-ZVP-to-ZVP transitions across successive prompt observations, while 1,563 initially ZVP prompts later became non-ZVP before returning to ZVP. Although finite-group sampling contributes to these changes, they illustrate why permanently excluding historically uninformative prompts may be undesirable.

2. Why Gaussian proposal and square-root pacing?

Under conditionally independent binary outcomes with success probability pp, the predicted zero-variance probability is z(p)=pG+(1−p)G,1−z(p)=Pr⁡(mixed group),z(p)=p^G+(1-p)^G, 1-z(p)=\Pr(\mathrm{mixed\ group}), where the probability of obtaining a mixed-outcome group is maximized at p=0.5p=0.5. This motivates prioritizing intermediate capability levels, consistent with the findings of Gao et al. (Prompt Curriculum Learning). The Gaussian proposal wt(x)∝exp⁡[−(Pt−1(x)−μt)2/(2σt2)]w_t(x)\propto\exp[-(P_{t-1}(x)-\mu_t)^2/(2\sigma_t^2)] provides a simple, interpretable mechanism, where μt\mu_t determines the target capability region and σt\sigma_t controls its coverage.

The schedules μt=μstart+(μend−μstart)ct\mu_t=\mu_{\rm start}+(\mu_{\rm end}-\mu_{\rm start})\sqrt{c_t} and σt=σstart+(σend−σstart)ct\sigma_t=\sigma_{\rm start}+(\sigma_{\rm end}-\sigma_{\rm start})c_t jointly control the curriculum's focus and coverage. The concave square-root schedule accelerates progression toward harder prompts early in training while slowing it later, whereas the feedback controller adapts ctc_t based on observed capability changes. Ablations yield 48.2/63.2 without the controller and 48.3/62.6 with linear pacing, compared with 48.6/64.5 for Full OAGS (Acc@8/Pass@8).

We do not claim theoretical optimality for either Gaussian allocation or square-root pacing. The uniform-proposal ablation (47.8/61.4 versus 48.6/64.5) provides empirical support for capability-guided allocation, rather than a guarantee of optimality.

AuthorsOpenReview ↗

Clarification of Outcome-Conditioned Selection and GRPO Gradient Bias

We thank the reviewer for highlighting the selection-bias issue in Eq. (9). We acknowledge that selecting groups based on realized outcome variance introduces outcome-dependent selection bias relative to the original unfiltered, uniform-prompt GRPO objective. We do not claim otherwise.

1. Why selection bias arises. Let K=∑i=1GRiK=\sum_{i=1}^{G}R_i denote the number of successful responses, assuming conditionally independent Bernoulli outcomes with true success probability pp. For a simplified selector retaining only mixed-outcome groups, S={1≤K≤G−1}S=\{1\leq K\leq G-1\}, we have E[K/G∣S]=p−pG1−pG−(1−p)G,\mathbb{E}[K/G\mid S]=\frac{p-p^G}{1-p^G-(1-p)^G}, which generally differs from pp. For example, when p=0.05p=0.05 and G=8G=8, the conditional mean increases to approximately 0.149, compared with the true success probability of 0.05. Although OAGS employs a more complex Top-BB selection over MM candidates, this example illustrates the reviewer's concern that rare successful outcomes on difficult prompts may be disproportionately retained.

2. What OAGS actually optimizes. As explicitly defined in Section 3.4 and Eq. (14), OAGS optimizes the standard GRPO clipped surrogate under the selection-induced distribution ωt\omega_t, rather than claiming an unbiased estimate of the original uniform-prompt objective. All responses are freshly generated by the current behavior policy, without historical trajectory replay. However, on-policy generation does not eliminate outcome-dependent selection bias. The standard PPO/GRPO likelihood ratio accounts for policy probability changes but does not correct for the additional group-selection probability. We therefore acknowledge that OAGS changes the effective training distribution without selection-specific importance correction.

3. Why outcome-based selection remains useful. Our goal is to allocate a fixed optimization budget toward groups providing nonzero group-relative reward signals. As defined in Eq. (9), the empirical variance Vt(x)=Pt(x)(1−Pt(x))V_t(x)=P_t(x)(1-P_t(x)) is zero for all-correct or all-incorrect groups and maximized when Pt(x)=0.5P_t(x)=0.5, thereby prioritizing mixed-outcome groups with more balanced success–failure compositions. This design is motivated by ICLR26 paper, Prompt Curriculum Learning for Efficient LLM Post-Training, which empirically identifies intermediate-difficulty prompts as particularly favorable for training efficiency and gradient quality. Accordingly, OAGS uses realized outcome variance to prioritize informative groups among already-generated candidates, rather than relying solely on historical difficulty estimates. This trades fidelity to the original sampling distribution for a higher concentration of informative training groups. Our empirical results support this efficiency-oriented design, but do not establish that the resulting selection bias is universally harmless or theoretically optimal.

We will clarify the empirical motivation for variance-based selection, the selection-induced distribution shift, its distinction from policy importance weighting, and the corresponding limitations in the revised manuscript.

4. Recent studies [1–6]. We agree these works deserve discussion and have initiated matched-setting reproductions of DPS [2], BOTS [4], and AR3PO [5]. Currently, these reproductions are still running: DPS and BOTS are at step 50, while AR3PO is at step 144 . Given the computational time required, we may not obtain final results for DPS and BOTS before the rebuttal deadline. We expect to provide completed results for AR3PO by the deadline. We will expand the related work to discuss references [1–6], and we will include completed, comparable evaluations for DPS [2], BOTS [4], and AR3PO [5] in the revised manuscript.

[2] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

[4] BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning

[5] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse

Reviewer_u56JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OAGS, an adaptive prompt-sampling method for GRPO. It tracks empirical pass rates and uses a Gaussian curriculum to allocate rollouts around a changing target difficulty. Fresh outcomes control the curriculum and refresh the tracker, while response groups with higher reward variance are prioritized for policy updates. Experiments compare OAGS with GRPO and data-selection methods on mathematical reasoning benchmarks.

Critical strengths
  1. Practical motivation: Avoiding uninformative rollouts is useful for GRPO training. Updating the tracker with discarded groups also makes use of observations that provide no immediate policy gradient.
  2. Clear presentation and useful comparisons: The method and implementation are generally clear. The paper evaluates both model initializations and reports performance, rollout usage, runtime, and component ablations.
Critical weaknesses
  1. The added complexity is not fully justified: The controller brings only modest gains over simpler schedules, and the variant without a dedicated tail quota performs better on average (Table 2).
  2. The efficiency comparison is incomplete: OAGS uses more rollout computation than vanilla GRPO, but an equally budgeted GRPO comparison is missing. Training-cost comparisons also exclude tracker initialization, which is reported separately. Results lack variation across independent training runs, making the smaller gains difficult to assess.
Questions for authors
  1. Please analyze OAGS against DAPO's Dynamic Sampling, separating pre-rollout allocation from post-rollout filtering. Do its gains over DS and vanilla GRPO persist under a matched total compute budget, including tracker initialization?
  2. Are the controller's gains stable across independent runs, and why does removing tail reservation improve the averages?
3 comments
AuthorsOpenReview ↗

Pre-Rollout Gaussian Allocation vs. Dynamic Sampling: Ablation and Compute Comparisons

We thank the reviewer for the constructive assessment and apologize for the late response. We prioritized completing additional computationally intensive RL experiments to provide measured evidence. We address the distinction between pre-rollout allocation and post-rollout filtering, followed by new budget-matched comparisons.

1. Pre-rollout allocation versus DAPO Dynamic Sampling.

Although both OAGS and DAPO's Dynamic Sampling (DS) prioritize non-zero-variance groups, their rollout allocation mechanisms differ.

DS repeatedly generates B=128B=128 candidate groups, discards zero-variance groups, and continues resampling until BB non-ZVP groups have been collected for a policy update. Its total candidate budget is therefore determined by the resampling rounds .

OAGS fixes its candidate budget before generation. At each iteration, a Gaussian proposal selects M=256M=256 distinct prompts using historical capability estimates. All candidates generate G=8G=8 responses and refresh the tracker, including discarded groups. An outcome-based selector then retains B=128B=128 groups, prioritizing non-ZVP groups and their empirical reward variance. If fewer than BB non-ZVP groups are available, remaining positions are filled from the MM candidate pool.

Thus, OAGS does not eliminate post-rollout filtering; its distinction is capability-aware pre-rollout allocation with a bounded candidate budget, avoiding iterative generation until a fully non-ZVP update batch is obtained.

2. Isolating the contribution of Gaussian allocation.

Our existing ablation (Table 2) replaces Gaussian candidate allocation with uniform sampling while preserving the same oversampling budget, eligibility constraints, and downstream outcome-based selector.

Qwen3-8B-BaseAcc@8Pass@8
Uniform Proposal + Online Selection47.861.4
Gaussian Proposal + Online Selection (OAGS)48.664.5

Gaussian allocation improves Acc@8 by 0.8 points and Pass@8 by 3.1 points, even when post-rollout selection is held constant. This supports its contribution beyond oversampling and outcome filtering alone.

3. Rollout-matched Vanilla GRPO comparison.

We additionally trained Vanilla GRPO for 536 policy-update steps, approximately matching OAGS's training rollout budget on Qwen3-8B. The following table reports complete results across six mathematical reasoning benchmarks.

MethodUpdatesTraining RolloutsMinerva Acc@8Minerva Pass@8AMC23 Acc@8AMC23 Pass@8MATH500 Acc@8MATH500 Pass@8AIME24 Acc@8AIME24 Pass@8AIME25 Acc@8AIME25 Pass@8Olympiad Acc@8Olympiad Pass@8AVG Acc@8AVG Pass@8
GRPO (original)267273,40845.856.684.195.091.095.840.060.033.353.358.369.658.871.7
GRPO (rollout-matched)536548,86446.356.680.995.089.796.042.560.036.253.359.570.759.271.9
OAGS267546,81647.858.586.997.590.595.451.376.731.753.358.470.161.175.2

At approximately equal training rollout budgets (GRPO generates 0.37% more responses), OAGS outperforms GRPO by +1.9 Acc@8 and +3.3 Pass@8 percentage points. Doubling Vanilla GRPO's update count yields only modest average improvements (+0.4 Acc@8 and +0.2 Pass@8). Furthermore, the original manuscript reports DS-r at 56.1/67.0 and DS-g at 58.2/69.9 in average Acc@8/Pass@8 on Qwen3-8B, both below OAGS's 61.1/75.2. Here, DS-r and DS-g denote training-rollout-matched and policy-update-matched comparisons, respectively.

These results support the benefit of Gaussian pre-rollout allocation combined with bounded candidate selection, rather than merely increasing the number of generated responses or optimizer updates.

4. Accounting for initialization and computational cost.

We agree that initialization should be included in cost accounting. Stage-1 tracker initialization generates 4∣D∣≈68K4|\mathcal D|\approx68\mathrm K responses, using the target model's initial checkpoint. Therefore, OAGS requires approximately 614.8K total generated responses, including initialization. At 267 policy-update steps, DS generates 826.4K training responses, while OAGS achieves higher accuracy with approximately 25.6% fewer generated responses, even after charging its initialization cost.

In summary, the evidence supports OAGS's contribution from two complementary aspects: Gaussian pre-rollout allocation improves the quality of a fixed candidate pool, while bounded candidate generation avoids the potentially larger rollout costs of repeated post-rollout rejection. We will make the mechanisms and budget definitions more explicit in the revision.

AuthorsOpenReview ↗

New Strict-Core Ablation, Tail Exploration Analysis, and Robustness Evidence

We appreciate the reviewer's concerns regarding the modest controller gains, the stronger no-tail-reservation variant, and the lack of training-run variability. We clarify the distinction between tail exploration and fixed tail reservation, supported by a new Strict Core experiment and additional random-seed evaluations.

1. Curriculum controller. Table 2 reports the following average Acc@8/Pass@8 results on Qwen3-8B-Base:

ConfigurationAcc@8Pass@8
OAGS Full48.664.5
w/o Controller48.263.2
Linear Curriculum48.362.6

Removing feedback control while retaining the square-root trajectory reduces Pass@8 by 1.3 points. Replacing the square-root trajectory with linear progression reduces Pass@8 by another 0.6 points, although Acc@8 slightly increases.

The controller is a lightweight supporting component, not an independent optimization objective. It regulates the Gaussian center and bandwidth based on observed capability changes and their directional consistency, allowing the curriculum to advance, retreat, or progress conservatively.

2. Strict Core experiment: tail exploration vs. fixed reservation. We report complete Qwen3-8B-Base results below. Each entry denotes Acc/Pass@8.

MethodMinervaAMC23MATH500AIME24AIME25OlympiadAVG
Strict Core (Tail-0)41.6/52.968.4/90.085.5/94.621.7/36.717.9/33.352.1/66.247.9/62.3
OAGS42.2/54.068.4/92.586.1/94.025.0/43.318.8/36.750.9/66.448.6/64.5
w/o Tail Reservation41.2/53.770.3/92.586.2/94.024.2/50.022.5/33.350.8/67.049.2/65.1

Importantly, Strict Core and w/o Tail Reservation represent different sampling mechanisms:

  • OAGS Full: The fixed oversampling budget M=256M=256 is nominally divided into 175 Gaussian-core and 81 tail candidates (68.27%/31.73%), with capacity-based cross-filling when needed.
  • w/o Tail Reservation (manuscript): Removes the fixed 31.73% tail quota. Instead, all MM candidate positions are allocated across eligible pass-rate buckets according to Gaussian weights. Consequently, tail prompts remain eligible and can still be sampled. This is flexible Gaussian allocation, not the removal of tail exploration.
  • Strict Core (Tail-0): Sets the target tail budget to zero and concentrates all MM candidates within the Gaussian core whenever sufficient core prompts are available. Tail candidates are considered only if the number of core candidates is insufficient to reach MM.

The results show that Full OAGS outperforms Strict Core by +0.7 Acc@8 and +2.2 Pass@8 points. The no-reservation variant further outperforms Strict Core by +1.3/+2.8 points and Full OAGS by +0.6/+0.6 points.

These findings support distinguishing tail coverage from fixed tail reservation. Restricting candidate allocation to the Gaussian core reduces performance, whereas allowing Gaussian-weighted sampling across the full eligible capability space performs best in our experiments. Thus, exploration outside the current target region is useful, but the originally prescribed 31.73% reservation is not necessarily optimal.

The nominal 31.7% tail reservation is motivated by the standard Gaussian distribution, where approximately 68.3% of the probability mass lies within one standard deviation of the mean and 31.7% lies in the tails. This property remains invariant as the Gaussian center μt\mu_t and bandwidth σt\sigma_t evolve during training. We adopt this ratio as a simple, distribution-motivated heuristic for allocating the candidate budget between core exploitation and tail exploration, rather than claiming it is optimal or reflects the actual mass of the discrete prompt distribution. The stronger performance without explicit tail reservation suggests that this fixed allocation may not achieve the best exploration–exploitation balance. We will clarify this distinction in the revised manuscript and leave adaptive core–tail budget allocation to future work.

This motivation is further supported by our tracker analysis: we observed 10,188 non-ZVP-to-ZVP transitions across successive prompt observations, while 1,563 initially ZVP prompts later became non-ZVP before returning to ZVP. Although finite-group sampling contributes to these changes, they illustrate why permanently excluding historically uninformative prompts may be undesirable.

Statistical MetricCountProportion
First observation is ZVP-07,37342.80%
First observation is ZVP-13121.81%
First observation is ZVP7,68544.61%
First observation is Non-ZVP9,54355.39%
Samples with at least one Non-ZVP →\rightarrow ZVP transition in history9,34254.23%

3. Additional training- and evaluation-seed robustness.

We independently trained Qwen3-8B OAGS with two additional seeds. Each entry denotes Acc@8 / Pass@8 (%).

Training runMinervaAMC23MATH500AIME24AIME25OlympiadAverage
Original47.8/58.586.9/97.590.5/95.451.3/76.731.7/53.358.4/70.161.1/75.2
Seed 4047.7/59.286.9/97.590.8/95.451.3/73.333.8/56.758.4/68.761.5/75.1
Seed 4148.0/59.686.2/95.090.8/96.051.3/73.332.9/53.358.3/70.261.3/74.6

We additionally evaluated with different sampling seeds:

Evaluation seedMinervaAMC23MATH500AIME24AIME25OlympiadAVG
Seed 4047.6/58.585.3/95.091.0/96.251.3/76.733.8/53.758.4/69.961.2/75.0
Seed 4147.8/58.886.2/97.590.9/95.650.8/73.334.2/56.758.5/70.161.4/75.3

Across the three independent training runs, OAGS achieves 61.30 ± 0.20 Acc@8 and 74.97 ± 0.32 Pass@8 (mean ± sample standard deviation). Two additional evaluation-seed results are 61.2/75.0 and 61.4/75.3. These provide complementary evidence of evaluation robustness.

Overall, the new evidence supports capability-aware Gaussian candidate allocation and continued tail coverage, while revealing that the original fixed tail allocation may be improved. We thank the reviewer for helping clarify these distinctions.

Reviewer_KCAaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Objective-Aware Gaussian Sampling (OAGS), a prompt curriculum for GRPO with binary outcome rewards. It maintains empirical prompt pass rates, proposes a fixed oversampled candidate batch using a Gaussian core and exploratory tail, refreshes estimates from all candidate outcomes, and retains a fixed number of response groups for optimization. A feedback controller adjusts the proposal center and bandwidth. Experiments on two Qwen3-8B initializations and six mathematical benchmarks report higher macro-average accuracy and pass@8 than the evaluated baselines, with lower training rollout cost than Dynamic Sampling and GRESO at matched update counts.

Critical strengths
  1. Separating candidate observation from training selection is a useful practical design. Outcomes from discarded groups can still refresh stale prompt estimates, while the fixed candidate budget limits the repeated-generation overhead of strict zero-variance filtering. The paper also explicitly states that outcome-dependent selection changes the training distribution, despite retaining the GRPO surrogate.

  2. The comparison includes both rollout-matched and update-matched DS/GRESO variants. OAGS achieves the best reported macro-average scores for both model initializations, and the Gaussian-proposal ablation supports capability-guided allocation beyond oversampling and post-rollout selection alone.

Critical weaknesses
  1. The evidence does not yet establish the overall performance–cost advantage relative to vanilla GRPO. OAGS generates twice as many training responses and takes 48.6 rather than 33.9 hours at the reported update count. The rollout-matched comparisons address DS/GRESO, but an analogous comparison with vanilla GRPO is missing. Moreover, the one-time initialization cost of 4|D| responses is excluded from the reported training rollout budget. This matters because the practical contribution depends on whether the additional allocation machinery improves the trade-off available at the same total budget.

  2. The incremental contribution of the full curriculum design is insufficiently established. On the Base initialization, removing the controller changes macro-average Acc@8/Pass@8 from 48.6/64.5 to 48.2/63.2, while removing fixed tail reservation improves both to 49.2/65.1 (Table 2). The latter still explores the tail, so it specifically questions the prescribed allocation rather than exploration itself. Without training-run variability or uncertainty for these differences, it is unclear whether the controller is reliably beneficial and why the full configuration should be preferred over the simpler reported variant.

Overall assessment: 2 — Weak rejection. The allocation framework is useful and the aggregate results are encouraging, but the current evidence does not sufficiently establish the added value of the complete curriculum or its trade-off against vanilla GRPO at equal total cost.

Questions for authors
  1. At the same total rollout or wall-clock budget, including tracker initialization, how does vanilla GRPO compare with OAGS? Please distinguish any available measured comparison from an extrapolation of the existing curves.

  2. Are the controller and tail-reservation differences reproducible across training seeds? If only the existing runs are available, can you provide paired evaluation uncertainty and explain why the full configuration is preferred despite the stronger no-tail-reservation result?

  3. Which policy actually initialized the tracker in each reported experiment, and what exact training-usage quotas and reset rules were used? These settings determine prompt eligibility and early controller feedback.

2 comments
AuthorsOpenReview ↗

Additional Rollout-Matched GRPO Results and Implementation Clarifications

We thank the reviewer for the constructive feedback and for recognizing the value of decoupling candidate observation from training selection. We also apologize for posting our response close to the rebuttal deadline, as we prioritized completing computationally intensive RL experiments to provide measured empirical evidence rather than preliminary estimates.

1. Rollout-matched GRPO comparison. We agree that equal policy-update counts alone do not establish computational efficiency, since OAGS evaluates additional candidate groups before each update. To address this concern, we conducted a new Vanilla GRPO experiment with 536 policy-update steps, approximately matching OAGS's training rollout budget on Qwen3-8B.

MethodUpdatesTraining RolloutsMinerva Acc@8Minerva Pass@8AMC23 Acc@8AMC23 Pass@8MATH500 Acc@8MATH500 Pass@8AIME24 Acc@8AIME24 Pass@8AIME25 Acc@8AIME25 Pass@8Olympiad Acc@8Olympiad Pass@8AVG Acc@8AVG Pass@8
GRPO (original)267273,40845.856.684.195.091.095.840.060.033.353.358.369.658.871.7
GRPO (rollout-matched)536548,86446.356.680.995.089.796.042.560.036.253.359.570.759.271.9
OAGS267546,81647.858.586.997.590.595.451.376.731.753.358.470.161.175.2

At nearly identical training rollout budgets (GRPO uses 0.37% more generated responses), OAGS still outperforms GRPO by +1.9 Acc@8 and +3.3 Pass@8 percentage points. Doubling Vanilla GRPO's update count produces only modest improvements (+0.4 Acc@8 and +0.2 Pass@8). This provides additional evidence that OAGS's gains are not solely attributable to its larger training rollout budget.

The two comparisons address complementary questions: update-matched evaluation isolates performance at equal numbers of policy updates, whereas rollout-matched evaluation tests whether the benefits persist when Vanilla GRPO receives additional updates under comparable generation budgets.

2. Initialization-inclusive cost accounting. We agree that tracker initialization must be counted. Stage 1 evaluates each training prompt using four responses, requiring 4∣D∣4|\mathcal D|, approximately 4$\times$17K=68K additional generations. Thus, OAGS's initialization-inclusive rollout count is approximately 614.8K, compared with the reported training rollout counts of 628.9K for GRESO and 826.4K for Dynamic Sampling at equal update counts.

In the revised manuscript, we will make these distinctions explicit by reporting the complete rollout counts for both the initialization-inclusive OAGS and the training-rollout-matched GRPO.

3. Tracker initialization and training quotas. We clarify the implementation details requested by the reviewer.

Each target model uses its own initial checkpoint for Stage-1 tracker initialization. Specifically, Qwen3-8B and Qwen3-8B-Base use their corresponding initial policies, rather than a shared cross-model prior. Each prompt receives four initial rollouts, and the mean binary outcome reward determines its initial pass-rate estimate.

During training, all M=256M=256 evaluated candidate groups refresh their prompt-specific capability estimates, including groups excluded from the policy update. The tracker retains historical observations but uses the latest observed pass rate for subsequent Gaussian allocation, consistent with Eq. (4).

OAGS maintains separate probe and training-selection counters. Only the B=128B=128 groups retained for GRPO optimization consume training quotas. Under the reported configuration, the base selection quota is 2 (number of epochs), increased to five for prompts satisfying the current Gaussian-core membership. The counters accumulate across epochs without epoch-boundary resets. Independent training runs initialize their trackers and counters separately.

We appreciate the reviewer's suggestions, which have helped us clarify the distinction between rollout efficiency, optimization efficiency, and the reproducibility of the sampling mechanism.

AuthorsOpenReview ↗

Additional Tail Ablations, Curriculum Analysis, and Multi-Seed Robustness

1. Curriculum controller. Table 2 compares Full OAGS (48.6/64.5), w/o Controller (48.2/63.2), and Linear Curriculum (48.3/62.6) in average Acc@8/Pass@8. Removing feedback control while retaining the square-root schedule reduces Pass@8 by 1.3 points; replacing the square-root trajectory with a linear schedule further reduces Pass@8 by 0.6 points. The controller is a lightweight pacing mechanism rather than an independent optimization objective. It adapts curriculum progression using the magnitude and directional consistency of fresh capability feedback.

2. Strict Core experiment: tail exploration vs. fixed reservation. We report complete Qwen3-8B-Base results below. Each entry denotes Acc/Pass@8.

MethodMinervaAMC23MATH500AIME24AIME25OlympiadAVG
Strict Core (Tail-0)41.6/52.968.4/90.085.5/94.621.7/36.717.9/33.352.1/66.247.9/62.3
OAGS42.2/54.068.4/92.586.1/94.025.0/43.318.8/36.750.9/66.448.6/64.5
w/o Tail Reservation41.2/53.770.3/92.586.2/94.024.2/50.022.5/33.350.8/67.049.2/65.1

Importantly, Strict Core and w/o Tail Reservation represent different sampling mechanisms:

  • OAGS Full: The fixed oversampling budget M=256M=256 is nominally divided into 175 Gaussian-core and 81 tail candidates (68.27%/31.73%), with capacity-based cross-filling when needed.
  • w/o Tail Reservation (manuscript): Removes the fixed 31.73% tail quota. Instead, all MM candidate positions are allocated across eligible pass-rate buckets according to Gaussian weights. Consequently, tail prompts remain eligible and can still be sampled. This is flexible Gaussian allocation, not the removal of tail exploration.
  • Strict Core (Tail-0): Sets the target tail budget to zero and concentrates all MM candidates within the Gaussian core whenever sufficient core prompts are available. Tail candidates are considered only if the number of core candidates is insufficient to reach MM.

The results show that Full OAGS outperforms Strict Core by +0.7 Acc@8 and +2.2 Pass@8 points. The no-reservation variant further outperforms Strict Core by +1.3/+2.8 points and Full OAGS by +0.6/+0.6 points.

These findings support distinguishing tail coverage from fixed tail reservation. Restricting candidate allocation to the Gaussian core reduces performance, whereas allowing Gaussian-weighted sampling across the full eligible capability space performs best in our experiments. Thus, exploration outside the current target region is useful, but the originally prescribed 31.73% reservation is not necessarily optimal.

The nominal 31.7% tail reservation is motivated by the standard Gaussian distribution, where approximately 68.3% of the probability mass lies within one standard deviation of the mean and 31.7% lies in the tails. This property remains invariant as the Gaussian center μt\mu_t and bandwidth σt\sigma_t evolve during training. We adopt this ratio as a simple, distribution-motivated heuristic for allocating the candidate budget between core exploitation and tail exploration, rather than claiming it is optimal or reflects the actual mass of the discrete prompt distribution. The stronger performance without explicit tail reservation suggests that this fixed allocation may not achieve the best exploration–exploitation balance. We will clarify this distinction in the revised manuscript and leave adaptive core–tail budget allocation to future work.

This motivation is further supported by our tracker analysis: we observed 10,188 non-ZVP-to-ZVP transitions across successive prompt observations, while 1,563 initially ZVP prompts later became non-ZVP before returning to ZVP. Although finite-group sampling contributes to these changes, they illustrate why permanently excluding historically uninformative prompts may be undesirable.

3. Additional training- and evaluation-seed robustness. We independently trained Qwen3-8B OAGS with two additional seeds. Complete results (Acc@8 / Pass@8, %) are:

Training runMinervaAMC23MATH500AIME24AIME25OlympiadAverage
Original47.8/58.586.9/97.590.5/95.451.3/76.731.7/53.358.4/70.161.1/75.2
Seed 4047.7/59.286.9/97.590.8/95.451.3/73.333.8/56.758.4/68.761.5/75.1
Seed 4148.0/59.686.2/95.090.8/96.051.3/73.332.9/53.358.3/70.261.3/74.6

We additionally evaluated with different sampling seeds:

Evaluation seedMinervaAMC23MATH500AIME24AIME25OlympiadAVG
Seed 4047.6/58.585.3/95.091.0/96.251.3/76.733.8/53.758.4/69.961.2/75.0
Seed 4147.8/58.886.2/97.590.9/95.650.8/73.334.2/56.758.5/70.161.4/75.3

Across three independent training runs, OAGS achieves 61.30 ± 0.20 Acc@8 and 74.97 ± 0.32 Pass@8, with consistent results across additional evaluation seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.