acceptodds
Under review as a conference paper at ICLR 2027

SpinningRL: Boosting RLVR Efficiency with Theoretical Reward Gain Modeling and Adaptive UCB Sampling

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of Large Language Models (LLMs), yet it remains painfully inefficient. Uniform sampling wastes computation on non-informative prompts and aggravates the zero-advantage problem in GRPO-like methods. Prior schedulers often select medium-success-rate prompts via learnability-style metrics (e.g., or ), which optimize for **gradient magnitude** rather than **policy improvement**. We instead propose a more direct metric: the KL-constrained **potential reward gain** that is estimable from existing rollouts, and introduce SpinningRL, a non-stationary bandit framework that continuously prioritizes prompts with the highest expected gain under the current policy. Concentrating sampling in this subspace improves sample efficiency and reduces baseline variance, mitigating zero-advantage issues. Experiments show consistent gains on mathematics, logical reasoning and coding tasks under the same runtime budget, with 17%-50% fewer rollouts than uniform sampling at matched performance. The code is anonymously available at [this URL](https://anonymous.4open.science/r/SpinningRL-B666/README.md).

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Data-efficient Training, RLVR, Sample Efficiency

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_aqgwOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper argues that RLVR algorithms such as GRPO devote substantial computational resources to rollouts but use them inefficiently.
  • To address this issue, the paper defines reward gain under a KL constraint as G(q_i). The authors argue that training should prioritize queries expected to yield the largest reward gain after a policy update.
  • The paper reformulates RLVR as a non-stationary bandit problem and introduces SW-UCB to schedule training queries. This ensures a degree of exploration when selecting queries for training.
  • The paper introduces EMA to incorporate historical rewards into advantage computation.
  • The authors report improved performance and sample efficiency over baselines across model sizes (1.7B–7B), model families (Qwen2.5 and Qwen3), and domains (mathematics, coding, and logical reasoning).
Critical strengths
  • The zero-variance and low sample efficiency problems in RLVR addressed by this paper are important issues that warrant resolution.
  • The reward gain metric defined in the paper, is clearly an appropriate target metric for assessing the training value of individual queries.
  • Prior RLVR studies have also argued that queries in the “difficult-yet-solvable” region are effective for training. This observation is consistent with Figure 4. Equations (6) and (7), derived from reward gain, provide a compelling theoretical explanation of this phenomenon.
  • The proposed method demonstrates improved effectiveness and efficiency over baselines across model sizes (1.7B–7B), model families (Qwen2.5 and Qwen3), and domains (mathematics, coding, and logical reasoning).
Critical weaknesses
  1. Reward gain is defined as the difference between the expected rewards of the optimal policy and the current policy. However, the implementation computes it using historical rewards generated by previous policies rather than the current policy. These rewards may originate from multiple policies at different training steps rather than a single previous policy. No empirical or theoretical guarantee is provided that these historical samples adequately represent current samples.

  2. Equation (1) formulates the problem under a KL constraint. However, the experiments set the KL loss coefficient to zero, which implies the absence of a KL constraint. The authors explain that GRPO’s clipping mechanism serves the role of the KL term, and I agree with this explanation to some extent. Nevertheless, I am not fully convinced, on theoretical grounds, that the main experiments operate within the KL-regularized setting assumed in the paper.

  3. The paper introduces two components: (1) reward-gain-based training sample scheduling and (2) history-aware reward normalization. It is therefore unclear whether the reported improvements arise from better scheduling or improved reward estimation using historical rewards (see this study). The appendix provides a comparison with US that partially addresses this issue. However, the paper should clarify whether EMA was applied to the other baselines. Further ablation studies would also be valuable.

  4. The paper does not provide an ablation study on the window size used in SW-UCB. Window size is a hyperparameter related to the extent of policy shift, which is affected by the training batch size. Practical guidelines for selecting these hyperparameters would therefore be important for broader adoption of the method.

Questions for authors
  1. As discussed in Weakness 1, could you provide empirical or theoretical evidence that reward gain estimated from historical samples adequately represents the gain estimated from current samples?

  2. As discussed in Weakness 2, it remains unclear whether GRPO without a KL loss term operates within the KL-constrained setting assumed in Equation (1). Could you provide a theoretical justification showing that clipping alone places GRPO within this setting? Such an explanation would help address my concern. At a minimum, this justification should cover specific clipping configurations, such as the commonly used lower and upper clipping coefficients of (0.2,0.28)(0.2, 0.28).

  3. As discussed in Weakness 3, was EMA applied to the baselines other than US? If not, could you report results with EMA applied to those baselines?

  4. Given studies reporting improvements over EMA, such as this work, could you explain your choice of EMA? If incorporating historical rewards is also a main contribution of this paper, I would appreciate a comparison with this approach.

  5. As discussed in Weakness 4, could you provide practical guidelines for selecting the window size and training batch size?

9 comments
AuthorsOpenReview ↗

Authors' Response (Part 1)

We thank Reviewer aqgw for the thoughtful and constructive feedback. We address the raised concerns point by point below. We also uploaded a new pdf version, with the revised part highlighted in blue. We greatly appreciate the reviewer's time and questions. Please let us know if you have any remaining questions or concerns.

W1 & Q1. Empirical guarantee of history-based gain estimation

We thank the reviewer for this thoughtful question which helps us to evaluate the validity of our core consumption. To assess the discrepancy introduced by using historical rewards to compute the current reward-gain signal, we collect 1000 recent samples at training steps 50, 100, 150 and 200, and estimate both mean accuracy and G^\widehat{G} using (i) historical rewards and (ii) N=8 on-policy current rollouts. We treat N=32 current on-policy rollout estimates as the ground truth, and report Pearson correlations of each estimator with this ground truth in the table below.

stepcurrent Acc Pearsonhistory Acc Pearsoncurrent G^\hat{G} Pearsonhistory G^\hat{G} Pearson
500.9470.8560.8210.650
1000.9660.8960.8590.713
1500.9800.9310.8750.730
2000.9790.9410.8680.722

We observe that the prompt difficulty is well preserved under history, with Acc correlations above 0.85 and increasing as training proceeds. The correlation for G^\hat G is moderate-to-strong (0.65-0.75) using historical rewards. Since SW-UCB scheduling primarily needs relative ordering across prompts rather than calibrated absolute values, this level of agreement retains ranking information sufficient for our use case.

AuthorsOpenReview ↗

Authors' Response (Part 2)

W2 & Q2. Theoretical and empirical justification for the KL-constrained interpretation of GRPO-Clip

We thank the reviewer for pressing on this important point. We clarify both the theoretical status of GRPO clipping relative to Eq.1 and the empirical evidence under our main-experiment configuration (ϵl,ϵh)=(0.2,0.28)(\epsilon_l,\epsilon_h)=(0.2,0.28).

Clipping as an incentive-level soft trust region.

For a prompt qq and response tokens yty_t, define the per-token importance ratio

=\frac{\pi_\theta(y_t\mid q,y_{<t})} {\pi_{\mathrm{old}}(y_t\mid q,y_{<t})}$$ The GRPO clipped surrogate is $$L_{\mathrm{clip}}(\theta) = \mathbb{E}_{(q,y)\sim\pi_{\mathrm{old}}} \left[ \sum_{t} \min\left( r_{t}(\theta) A_{t}, \operatorname{clip}(r_{t}(\theta),1-\epsilon_{l},1+\epsilon_{h})A_{t} \right) \right]$$ For $A_t>0$, increasing the ratio above $1+\epsilon_h$ no longer improves this sample’s clipped objective. For $A_t<0$, decreasing it below $1-\epsilon_l$ likewise yields no further gain. Thus clipping serves as an **incentive-level soft constraint** that limits the surrogate incentive for sampled token ratios to leave $[1-\epsilon_l,1+\epsilon_h]$. **Conditional KL implication under a hard ratio bound.** Assuming a token-level KL constraint regulated by the clipping mechanism $$1-\epsilon_l \\;\leq\\; \frac{\pi_{\mathrm{new}}(a\mid h)}{\pi_{\mathrm{old}}(a\mid h)} \\;\leq\\; 1+\epsilon_h$$ then the conditional KL would satisfy the loose bound $$D_{\mathrm{KL}}\\!\bigl( \pi_{\mathrm{new}}(\cdot\mid h) \\,\Vert\\, \pi_{\mathrm{old}}(\cdot\mid h) \bigr) \\;\leq\\; \log(1+\epsilon_h),$$ and likewise $D_{\mathrm{KL}}(\pi_{\mathrm{old}}(\cdot\mid h)\Vert\pi_{\mathrm{new}}(\cdot\mid h))\le -\log(1-\epsilon_l)$. For the standard configuration $(\epsilon_l,\epsilon_h)=(0.2,0.28$) used in our experiments, this gives $$\log(1.28)\approx 0.247, \qquad -\log(0.8)\approx 0.223.$$ The clipping mechanism could therefore be interpreted as a local soft trust region whose natural KL scale is on the order of $10^{-1}$ per token under a hard-bound idealization, while in practice the updates show much smaller change and remain near the clip boundaries. **Empirical training KL divergence distribution.** To assess whether our main GRPO-Clip runs operate in a KL-constrained regime compatible with Eq.1, we measure the **same-direction** per-token update KL $D_{\mathrm{KL}}(\pi_{\theta^{*}}\Vert\pi_\theta)$ on selected prompts under the exact main-experiment settings $(\epsilon_l=0.2, \epsilon_h=0.28)$, and compare it with the theoretical radius $\hat\delta_q(\beta)$ induced by our reward-gain temperature $\beta=0.4$. The resulting distributions over the first 100 training steps are shown in Figure 11 in Appendix M. The two distributions show **close token-level dynamics** in terms of **order of magnitude**, **p10/p90/mean/median value distribution** and **trend** (early rise then decline), supporting that Eq.1’s KL-constrained modeling is a faithful description of the effective update budget under GRPO-Clip, even without an explicit KL loss term.
AuthorsOpenReview ↗

Authors' Response (Part 3)

W3 & Q3. Ablation study of EMA

We ablated the history-aware baseline estimator over GRPO baseline in Appendix K of the original submission, with results show that the benefit of EMA-baseline alone is limited in our training setting, so our main contribution focuses on sample selection rather than advantage normalization. We would make this clear in the final paper.

HistoryAcc
USw/o46.2
USw/47.2
SpinningRLw/o49.2
SpinningRLw/49.9

Q4. Comparison with DBB's baseline estimation

DBB [1] models each prompt's binary reward as Bernoulli(pp) with a Beta prior, and updates the posterior with a discount factor λ\lambda over historical visits (α=λα\alpha = \lambda \alpha for successes and β=λβ\beta = \lambda \beta for failures). The GRPO baseline is then the posterior mean μ=α/(α+β)\mu = \alpha / (\alpha + \beta) with scale σ=p(1−p)\sigma = \sqrt{p(1-p)}.

Our history-aware EMA baseline shares the same high-level design by tracking a discounted running estimate of (μ,σ)(\mu, \sigma) over past rewards. The main difference is that our EMA estimator is non-parametric. Our design tracks the empirical mean and second moment without assuming a Bernoulli-Beta generative model. The empirical numerical differences between the two methods are as follows.

Mean. DBB assumes a medium difficulty prior Beta($\alpha=\beta =1 $) for each sample, which injects pseudocounts that pull μ\mu toward 0.5. Without such a prior, our EMA estimator has no such shrinkage behavior. This prior also leads to behavior divergence when the reward history is all 0 or all 1 under a binary reward case. DBB’s prior keeps μ\mu slightly interior and σ>0\sigma \gt 0, so every sample still receives a small nonzero advantage, whereas our EMA would assign the samples zero advantage without such a middle-difficulty prior.

Variance. With the prior removed, the variance of the two methods differ only by the Bessel factor N/(N−1)N/(N-1). DBB uses the plug in p(1−p)p(1-p), whereas our second moment based method uses the unbiased sample variance.

We would like to empirically compare against DBB in terms of learning behavior and performance when time permits.

W4 & Q5. Ablation of hyper-parameters

Window size. We ablated the window size WW on Qwen3-1.7B Math at 150 training steps. The results are shown in the table below. The results indicated that enlarging the window size leads to better performance, while also bringing slightly higher computation and storage overhead.

MATHAMCAIME24AIME25AIME26AvgTraining hours
US83.858.227.320.415.040.915.6
window 2580.954.021.720.022.539.818.6
window 50 (Default)83.558.729.224.621.543.516.9
window 7583.960.531.923.319.243.817.6

Candidate batch size. We also ablated the selection of different candidate batch size on Qwen3-1.7B Math at 150 training steps. Results below showed that larger candidate set size also encourages faster convergence with a slighly higher efficiency trade-off.

MATHAMCAIME24AIME25AIME26AvgTraining hours
US83.858.227.320.415.040.915.6
∣D∣/8\lvert\mathcal{D}\rvert/883.157.227.920.421.542.017.6
∣D∣/32\lvert\mathcal{D}\rvert/32 (Default)83.558.729.224.621.543.516.9
∣D∣\lvert\mathcal{D}\rvert84.361.531.021.024.844.518.4

We thank the reviewer for the constructive questions, and would include the results in the final paper.

[1] Discounted Beta–Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards. Haechan Kim, Soohyun Ryu et al.

Reviewer_v7KWOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Authors propose a method of adaptively sampling rollouts in GRPO-style RL with Monte Carlo advantage estimation. Authors propose an estimator for expected reward gain. This is done by approximating the policy with Monte Carlo samples and applying the optimal policy update. This results in a simple, practical estimator. Authors characterize how KL regularization parameter shapes the expected gain, and also establishes a connection to learnability metrics. Empirical estimates also validate the estimator. Based on this estimator, authors propose Sliding-Window UCB-based adaptive GRPO algorithm. The proposed algorithm shows a consistent improvement across multiple domains (math, logical reasoning, coding).

Critical strengths

The proposed method has a principled foundation on multi-arm bandit and the optimal policy characterization. As these are very well-studied areas of study, the proposed method is both theoretically and practically well founded. This also establishes the connection between GRPO, multi-arm bandits, and optimal policy characterization. This would also lead to a productive research at the intersection of these areas, which will in turn make the impact of the paper stronger.

Authors also provide an insightful characterization of the proposed estimator by 1) making a connection to prior practices (learnability based methods), 2) analyze the closed-form function and its limits, 3) compare against empirical estimation across training runs. Such a strong characterization enables readers to have more confidence adopting the method, thereby increasing the opportunity to have impact to practitioners.

Critical weaknesses

The comparison against DAPO, the most popular baseline, could've been made clearer with an ablation study. DAPO fixes the prompt set and then adaptively samples rollouts; SpinningRL could similarly fix the prompt set and run UCB sampling to do adaptive sampling. In this case, baseline estimator in Section 2.4 is not needed, and a regular GRPO-style advantage estimate shall be used. This would be a cleaner comparison against DAPO to identify the benefit of UCB sampling.

Learning dynamics is the area that is not sufficiently discussed and compared against. Learning dynamics also study the expected improvement from updates, hence it can provide complementary view on the proposed method and possibly alternative methods.

Questions for authors

Can we ablate the baseline estimator by applying the estimator in Section 2.4 to GRPO baseline? It wasn't clear to me how much gain is coming from this baseline estimator vs. full SpinningRL.

How does the window size impact the effectiveness of the method?

Where does 8 in |Q_t| = |D|/8 come from? How does the candidate set size impact the effectiveness of the method- should practitioners favor larger |Q_t|, smaller |Q_t|, or would the optimum be in the middle?

3 comments
AuthorsOpenReview ↗

Authors' Response

We thank Reviewer v7KW for the thoughtful and constructive feedback. We address the raised concerns point by point below. We also uploaded a new pdf version, with the revised part highlighted in blue. We greatly appreciate the reviewer's time and questions. Please let us know if you have any remaining questions or concerns.

W1. Use UCB as sampling criteria within DAPO's Dynamic Sampling

We thank the reviewer for this constructive question. The sampling criteria of DAPO is to randomly select samples with non-zero advantages. Replace this criteria with our reward gain estimtor G^\hat G and the UCB score selector is an important and meaningful ablation. We conduct experiment on Qwen3-1.7B Math by combining SpinningRL's UCB selector with DAPO's Dynamic Sampling and compare with the DAPO baseline. Results at training step 75 are summarized below. This ablation study verified the benefit of SpinningRL's UCB selector alone. We will complete the observation to more training steps and report the results in the final paper.

MATHAMCAIME24AIME25AIME26AvgTraining hours
DAPO81.347.621.517.316.036.714.2
DAPO w/ UCB81.555.925.817.516.039.313.5

Q1. Ablation of the baseline estimator

We already conducted this ablation in the original submission (Appendix K). As shown by Qwen3-1.7B Math at 200 training steps, the sample selection mechanism of SpinningRL brings the largest improvement over GRPO baseline, while the baseline estimator alone also slightly improves the baseline performance.

HistoryAcc
USw/o46.2
USw/47.2
SpinningRLw/o49.2
SpinningRLw/49.9

Q2. Ablation of window size

We ablated the window size WW on Qwen3-1.7B Math at 150 training steps. The results are shown in the table below. The results indicated that enlarging the window size leads to better performance, while also bringing slightly higher computation and storage overhead.

MATHAMCAIME24AIME25AIME26AvgTraining hours
US83.858.227.320.415.040.915.6
window 2580.954.021.720.022.539.818.6
window 50 (Default)83.558.729.224.621.543.516.9
window 7583.960.531.923.319.243.817.6

Q3. Ablation of candidate set size

We ablated the selection of different candidate batch size on Qwen3-1.7B Math at 150 training steps. Results below showed that larger candidate set size also encourages faster convergence with a slighly higher efficiency trade-off.

MATHAMCAIME24AIME25AIME26AvgTraining hours
US83.858.227.320.415.040.915.6
∣D∣/8\lvert\mathcal{D}\rvert/883.157.227.920.421.542.017.6
∣D∣/32\lvert\mathcal{D}\rvert/32 (Default)83.558.729.224.621.543.516.9
∣D∣\lvert\mathcal{D}\rvert84.361.531.021.024.844.518.4

We thank the reviewer for the constructive questions, and would include the ablation results in the final paper.

Thanks authors for running these ablation studies. They demonstrate the reliability of the findings. I will update the score accordingly.

Thank you for your recognition and for the constructive suggestions. We will include these analyses in the final version of the paper. We sincerely appreciate your thoughtful evaluation and encouraging feedback.

Reviewer_ubGfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SPINNINGRL, which employs the latent reward gain—subject to a KL constraint—as a metric for prompt selection. By integrating a sliding-window Upper Confidence Bound (UCB) scheduling strategy with historical baseline smoothing, the method enhances the training efficiency of RLVR (Reinforcement Learning from Verifiable Rewards). The authors validate the approach across mathematical, logical, and coding tasks, further substantiating its effectiveness through ablation studies on metrics and baselines, experiments using multiple random seeds, and an analysis of the correlation between prompt selection and future reward gains.

Critical strengths

1.Efficiency improvements with practical value. The method achieves superior results across various models and tasks—given comparable training budgets—while incurring minimal scheduling overhead. 2. A design that is both interpretable and mechanistically validated. The performance metrics are supported by clear mathematical derivations, while targeted ablation studies and predictive validity analyses provide evidence that they outperform traditional learnability metrics.

Critical weaknesses

The connection between the theoretical proxy and the actual update process is insufficient. The metric corresponds to an idealized reward-reweighting strategy rather than the actual GRPO update. In the case of binary rewards, it effectively functions as a non-linear transformation of the pass rate, failing to adequately distinguish whether performance gains stem from reward modeling or from the model’s ability to handle more challenging problems. This limits the persuasiveness of the core contribution.

The applicability of the regret guarantees to practical settings remains unclear. Factors such as stochastic candidate sets, batch selection, and estimation bias from finite samples have not been sufficiently incorporated into the analysis; given that the actual candidate coverage is only 1/8, the provided upper bound on subsampling error cannot simply be dismissed as negligible. This undermines the reliability of the theoretical contribution.

Questions for authors

What is the measured per-step KL divergence distribution during training, and what \delta in Eq.~1 does it correspond to? Is the choice of \beta = 0.4 quantitatively consistent with this measurement?

How do performance and overhead change if the candidate set is enlarged from |\mathcal{Q}_t| = |\mathcal{D}|/8 to the full dataset?

What is the signal-to-noise ratio of the \hat{G} estimate under N = 8 rollouts? Have you considered applying shrinkage to \hat{G}, or explicitly separating estimation noise from the exploration bonus in the UCB score?

2 comments
AuthorsOpenReview ↗

Authors' response (Part 1)

We thank reviewer ubGf for the thoughtful and constructive feedback. We address the raised concerns point by point below. We also uploaded a new pdf version, with the revised part highlighted in blue. We greatly appreciate the reviewer's time and questions. Please let us know if you have any remaining questions or concerns.

Q1. Empirical training KL divergence distribution & Consistency with the theoretical modeling

We thank the reviewer for this constructive question. Under the trust-region problem in Eq.1 and the Lagrange multiplier solution in Eq.2, the KL radius induced by a temperature β\beta for a prompt qq and current policy πθ\pi_\theta is

=D_{\mathrm{KL}}\big(q_{\beta}(\cdot\mid q)\\,\\|\\,\pi_{\theta}(\cdot\mid q)\big) =\frac{\mathbb{E}_{q\beta}(r(q))}{\beta}-\log Z_{q}(\beta)$$ where $q_{\beta}(y)\propto \pi_{\theta}(y\mid q)\exp(r(y)/\beta)$ and $Z_{q}(\beta)=\mathbb{E}_{\pi\theta}(\exp(r/\beta))$. $\delta_{q}(\beta)$ represents the **trust-region radius corresponding to a chosen $\beta$**. We could further derive that $\mathrm{d}\delta_q/\mathrm{d}\beta\le 0$, meaning that a smaller $\beta$ induces a stronger reward tilt and a larger KL radius. In practice, we estimate this theoretical radius with the same finite-rollout Monte Carlo construction used by our training diagnostic. With $N{=}8$ responses from $\pi_\theta$, $$\hat\delta_q(\beta) =\sum_{i=1}^{N}\tilde w_i(\beta)\,\frac{r_i}{\beta}-\log\hat Z_q(\beta), \quad \tilde w=\mathrm{softmax}(r/\beta)$$ We then compare it with the **same-direction empirical update KL** on the same selected prompts $$D_{\mathrm{KL}}\big(\pi_{\theta^{\ast}}(\cdot\mid q)\\,\\|\\,\pi_{\theta}(\cdot\mid q)\big) =\sum_{t=1}^{|o|} \mathbb{E}_{o_{< t}\sim\pi_{\theta^{\ast}}} \Big[ D_{\mathrm{KL}}\big( \pi_{\theta^{\ast}}(\cdot\mid q,o_{< t}) \\,\\|\\, \pi_{\theta}(\cdot\mid q,o_{< t}) \big) \Big]$$ With the main-experiment choice $\beta=0.4$, we plot the theoretical $\hat\delta_q(\beta)$ and empirical update-KL distributions over the first 100 training steps, both distributions are normalized by the response length, representing token-level training dynamics. The results are plotted in Figure 11 in Appendix M. We observe that 1. The two distributions show clear **scale consistency** and **trend consistency**. The two distributions fall in roughly the same value ranges. Both distributions rise early and then decline as training proceeds, with comparable p10-p90 spread. 2. After roughly mid-training, while remaining on the same scale, the theoretical $\delta_{q}(0.4)$ becomes **slightly lower** than the empirical update KL. This suggests that using **adaptive $\beta$ value across different training stages**, such as using smaller $\beta$ toward later training could further align the theoretical trust-region budget with the measured update size. We view adaptive $\beta$ as a promising refinement and plan to conduct further ablation to verify its effectiveness. We thank the reviewer for prompting this diagnostic, which strengthens our understanding of the reward-gain modeling. We will include the plots and discussion in the final version of the paper. ### **Q2. Performance and overhead ablation of candidate set size** We conduct ablation study on the candidate set size with Qwen3-1.7B, Math. The average test accuracy (Mean@32) and total training hours are reported in the Table below. | | MATH | AMC | AIME24 | AIME25 | AIME26 | Avg | Training hours | | ------------------ | -------- | -------- | -------- | -------- | -------- | -------- | -------------- | | US | 83.8 | 58.2 | 27.3 | 20.4 | 15.0 | 40.9 | 15.6 | | $\lvert\mathcal{D}\rvert/8$ | 83.1 | 57.2 | 27.9 | 20.4 | 21.5 | 42.0 | 16.6 | | $\lvert\mathcal{D}\rvert/32$ | 83.5 | 58.7 | 29.2 | **24.6** | 21.5 | 43.5 | 16.9 | | $\lvert\mathcal{D}\rvert$ | **84.3** | **61.5** | **31.0** | 21.0 | **24.8** | **44.5** | 17.4 | Results show that enlarging the candidate set size enables faster convergence in the early training stages while also introducing larger scheduling overhead. We will complete the comparison with longer training steps and report the suggested hyper-parameter choice under this performance-efficiency trade-off.
AuthorsOpenReview ↗

Authors' Response (Part 2)

Q3. Signal-to-noise ratio of G^\hat{G} estimation & Effect of applying shrinkage

Following the empirical-Bayes SNR construction of [1], we measure the signal-to-noise ratio of G^\widehat{G} and ablate shrinkage within the candidate batch.

SNR definition. For each prompt qiq_i, we collect 32 on-policy rollouts and draw R=32 subsets of size N=8 without replacement. The per-prompt noise power is the variance of G^\widehat{G} across these subsets,

\qquad \overline{\hat\sigma_{n}^{2}}=\mathrm{mean}_{i}(\hat\sigma_{n,i}^{2}).$$ The signal power and SNR are $$\hat\sigma_{s}^{2} = \max\\!\Bigl( \mathrm{Var}_{i}(\widehat{G}_{i})-\overline{\hat\sigma_{n}^{2}},\\,0 \Bigr), \qquad \mathrm{SNR} = \frac{\hat\sigma_{s}^{2}}{\overline{\hat\sigma_{n}^{2}}}.$$ **Empirical-Bayes shrinkage.** On the candidate batch $\\{\widehat{G}_{i},\hat\sigma_{n,i}^{2}\\}$ we form an empirical-Bayes prior and shrink: $$\overline{G} = \mathrm{mean}_{i}(\widehat{G}_{i}), \qquad \hat\sigma_{s}^{2} = \max\\!\Bigl( \mathrm{Var}_{i}(\widehat{G}_{i})-\overline{\hat\sigma_{n}^{2}},\\,0 \Bigr), \\\\ \omega_{i} = \frac{\hat\sigma_{s}^{2}}{\hat\sigma_{s}^{2}+\hat\sigma_{n,i}^{2}}, \qquad \widetilde{G}_{i} = \overline{G}+\omega_{i}\\,(\widehat{G}_{i}-\overline{G}).$$ Prompts with larger $\hat\sigma_{n,i}^{2}$ receive smaller $\omega_i$ and are pulled more toward $\overline{G}$. **Offline evaluation.** We collect 1000 samples at steps 50, 100, 150 and 200 and evaluate SNR offline. Results show that shrinkage slightly improves SNR, but the gains are modest. | step | SNR raw | SNR post | improvement ratio | | ---: | ------: | -------: | ----------------: | | 50 | 2.33 | 2.71 | ×1.16 | | 100 | 3.22 | 3.64 | ×1.13 | | 150 | 3.65 | 4.06 | ×1.11 | | 200 | 3.43 | 3.90 | ×1.14 | **Training ablation.** When applying shrinkage to the training process of Qwen3-1.7B Math, we observed no significant benefit in the first 150 training steps. | | MATH | AMC | AIME24 | AIME25 | AIME26 | Avg | Training hours | | ------------- | ---- | ---- | ------ | ------ | ------ | -------- | -------------- | | US | 83.8 | 58.2 | 27.3 | 20.4 | 15.0 | 40.9 | 15.6 | | w/o shrinkage | 83.5 | 58.7 | 29.2 | 24.6 | 21.5 | **43.5** | 16.9 | | w/ shrinkage | 82.8 | 57.8 | 27.1 | 23.8 | 22.1 | 42.7 | 19.2 | As shown in Figure 12 in Appendix O, shrinkage changes the $\widehat{G}$ distribution by **suppressing large gains, making the estimate more conservative**. This redistribution may require retuning other hyperparameters (e.g., the exploration coefficient $\lambda$), and additional bootstrapping also increases training cost. How to reduce the possible noise of the limited N=8 rollout estimation is a topic that deserves further exploration, and we thank reviewer ubGf for this thoughtful question. [1] Stein's Estimation Rule and its Competitors—An Empirical Bayes Approach, Bradley Efron and Carl Morris.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.