- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper argues that RLVR algorithms such as GRPO devote substantial computational resources to rollouts but use them inefficiently.
- To address this issue, the paper defines reward gain under a KL constraint as G(q_i). The authors argue that training should prioritize queries expected to yield the largest reward gain after a policy update.
- The paper reformulates RLVR as a non-stationary bandit problem and introduces SW-UCB to schedule training queries. This ensures a degree of exploration when selecting queries for training.
- The paper introduces EMA to incorporate historical rewards into advantage computation.
- The authors report improved performance and sample efficiency over baselines across model sizes (1.7B–7B), model families (Qwen2.5 and Qwen3), and domains (mathematics, coding, and logical reasoning).
Critical strengths
- The zero-variance and low sample efficiency problems in RLVR addressed by this paper are important issues that warrant resolution.
- The reward gain metric defined in the paper, is clearly an appropriate target metric for assessing the training value of individual queries.
- Prior RLVR studies have also argued that queries in the “difficult-yet-solvable” region are effective for training. This observation is consistent with Figure 4. Equations (6) and (7), derived from reward gain, provide a compelling theoretical explanation of this phenomenon.
- The proposed method demonstrates improved effectiveness and efficiency over baselines across model sizes (1.7B–7B), model families (Qwen2.5 and Qwen3), and domains (mathematics, coding, and logical reasoning).
Critical weaknesses
-
Reward gain is defined as the difference between the expected rewards of the optimal policy and the current policy. However, the implementation computes it using historical rewards generated by previous policies rather than the current policy. These rewards may originate from multiple policies at different training steps rather than a single previous policy. No empirical or theoretical guarantee is provided that these historical samples adequately represent current samples.
-
Equation (1) formulates the problem under a KL constraint. However, the experiments set the KL loss coefficient to zero, which implies the absence of a KL constraint. The authors explain that GRPO’s clipping mechanism serves the role of the KL term, and I agree with this explanation to some extent. Nevertheless, I am not fully convinced, on theoretical grounds, that the main experiments operate within the KL-regularized setting assumed in the paper.
-
The paper introduces two components: (1) reward-gain-based training sample scheduling and (2) history-aware reward normalization. It is therefore unclear whether the reported improvements arise from better scheduling or improved reward estimation using historical rewards (see this study). The appendix provides a comparison with US that partially addresses this issue. However, the paper should clarify whether EMA was applied to the other baselines. Further ablation studies would also be valuable.
-
The paper does not provide an ablation study on the window size used in SW-UCB. Window size is a hyperparameter related to the extent of policy shift, which is affected by the training batch size. Practical guidelines for selecting these hyperparameters would therefore be important for broader adoption of the method.
Questions for authors
-
As discussed in Weakness 1, could you provide empirical or theoretical evidence that reward gain estimated from historical samples adequately represents the gain estimated from current samples?
-
As discussed in Weakness 2, it remains unclear whether GRPO without a KL loss term operates within the KL-constrained setting assumed in Equation (1). Could you provide a theoretical justification showing that clipping alone places GRPO within this setting? Such an explanation would help address my concern. At a minimum, this justification should cover specific clipping configurations, such as the commonly used lower and upper clipping coefficients of .
-
As discussed in Weakness 3, was EMA applied to the baselines other than US? If not, could you report results with EMA applied to those baselines?
-
Given studies reporting improvements over EMA, such as this work, could you explain your choice of EMA? If incorporating historical rewards is also a main contribution of this paper, I would appreciate a comparison with this approach.
-
As discussed in Weakness 4, could you provide practical guidelines for selecting the window size and training batch size?
9 comments
Authors' Response (Part 2)
W2 & Q2. Theoretical and empirical justification for the KL-constrained interpretation of GRPO-Clip
We thank the reviewer for pressing on this important point. We clarify both the theoretical status of GRPO clipping relative to Eq.1 and the empirical evidence under our main-experiment configuration .
Clipping as an incentive-level soft trust region.
For a prompt and response tokens , define the per-token importance ratio
=\frac{\pi_\theta(y_t\mid q,y_{<t})} {\pi_{\mathrm{old}}(y_t\mid q,y_{<t})}$$ The GRPO clipped surrogate is $$L_{\mathrm{clip}}(\theta) = \mathbb{E}_{(q,y)\sim\pi_{\mathrm{old}}} \left[ \sum_{t} \min\left( r_{t}(\theta) A_{t}, \operatorname{clip}(r_{t}(\theta),1-\epsilon_{l},1+\epsilon_{h})A_{t} \right) \right]$$ For $A_t>0$, increasing the ratio above $1+\epsilon_h$ no longer improves this sample’s clipped objective. For $A_t<0$, decreasing it below $1-\epsilon_l$ likewise yields no further gain. Thus clipping serves as an **incentive-level soft constraint** that limits the surrogate incentive for sampled token ratios to leave $[1-\epsilon_l,1+\epsilon_h]$. **Conditional KL implication under a hard ratio bound.** Assuming a token-level KL constraint regulated by the clipping mechanism $$1-\epsilon_l \\;\leq\\; \frac{\pi_{\mathrm{new}}(a\mid h)}{\pi_{\mathrm{old}}(a\mid h)} \\;\leq\\; 1+\epsilon_h$$ then the conditional KL would satisfy the loose bound $$D_{\mathrm{KL}}\\!\bigl( \pi_{\mathrm{new}}(\cdot\mid h) \\,\Vert\\, \pi_{\mathrm{old}}(\cdot\mid h) \bigr) \\;\leq\\; \log(1+\epsilon_h),$$ and likewise $D_{\mathrm{KL}}(\pi_{\mathrm{old}}(\cdot\mid h)\Vert\pi_{\mathrm{new}}(\cdot\mid h))\le -\log(1-\epsilon_l)$. For the standard configuration $(\epsilon_l,\epsilon_h)=(0.2,0.28$) used in our experiments, this gives $$\log(1.28)\approx 0.247, \qquad -\log(0.8)\approx 0.223.$$ The clipping mechanism could therefore be interpreted as a local soft trust region whose natural KL scale is on the order of $10^{-1}$ per token under a hard-bound idealization, while in practice the updates show much smaller change and remain near the clip boundaries. **Empirical training KL divergence distribution.** To assess whether our main GRPO-Clip runs operate in a KL-constrained regime compatible with Eq.1, we measure the **same-direction** per-token update KL $D_{\mathrm{KL}}(\pi_{\theta^{*}}\Vert\pi_\theta)$ on selected prompts under the exact main-experiment settings $(\epsilon_l=0.2, \epsilon_h=0.28)$, and compare it with the theoretical radius $\hat\delta_q(\beta)$ induced by our reward-gain temperature $\beta=0.4$. The resulting distributions over the first 100 training steps are shown in Figure 11 in Appendix M. The two distributions show **close token-level dynamics** in terms of **order of magnitude**, **p10/p90/mean/median value distribution** and **trend** (early rise then decline), supporting that Eq.1’s KL-constrained modeling is a faithful description of the effective update budget under GRPO-Clip, even without an explicit KL loss term.Authors' Response (Part 3)
W3 & Q3. Ablation study of EMA
We ablated the history-aware baseline estimator over GRPO baseline in Appendix K of the original submission, with results show that the benefit of EMA-baseline alone is limited in our training setting, so our main contribution focuses on sample selection rather than advantage normalization. We would make this clear in the final paper.
| History | Acc | |
|---|---|---|
| US | w/o | 46.2 |
| US | w/ | 47.2 |
| SpinningRL | w/o | 49.2 |
| SpinningRL | w/ | 49.9 |
Q4. Comparison with DBB's baseline estimation
DBB [1] models each prompt's binary reward as Bernoulli() with a Beta prior, and updates the posterior with a discount factor over historical visits ( for successes and for failures). The GRPO baseline is then the posterior mean with scale .
Our history-aware EMA baseline shares the same high-level design by tracking a discounted running estimate of over past rewards. The main difference is that our EMA estimator is non-parametric. Our design tracks the empirical mean and second moment without assuming a Bernoulli-Beta generative model. The empirical numerical differences between the two methods are as follows.
Mean. DBB assumes a medium difficulty prior Beta($\alpha=\beta =1 $) for each sample, which injects pseudocounts that pull toward 0.5. Without such a prior, our EMA estimator has no such shrinkage behavior. This prior also leads to behavior divergence when the reward history is all 0 or all 1 under a binary reward case. DBB’s prior keeps slightly interior and , so every sample still receives a small nonzero advantage, whereas our EMA would assign the samples zero advantage without such a middle-difficulty prior.
Variance. With the prior removed, the variance of the two methods differ only by the Bessel factor . DBB uses the plug in , whereas our second moment based method uses the unbiased sample variance.
We would like to empirically compare against DBB in terms of learning behavior and performance when time permits.
W4 & Q5. Ablation of hyper-parameters
Window size. We ablated the window size on Qwen3-1.7B Math at 150 training steps. The results are shown in the table below. The results indicated that enlarging the window size leads to better performance, while also bringing slightly higher computation and storage overhead.
| MATH | AMC | AIME24 | AIME25 | AIME26 | Avg | Training hours | |
|---|---|---|---|---|---|---|---|
| US | 83.8 | 58.2 | 27.3 | 20.4 | 15.0 | 40.9 | 15.6 |
| window 25 | 80.9 | 54.0 | 21.7 | 20.0 | 22.5 | 39.8 | 18.6 |
| window 50 (Default) | 83.5 | 58.7 | 29.2 | 24.6 | 21.5 | 43.5 | 16.9 |
| window 75 | 83.9 | 60.5 | 31.9 | 23.3 | 19.2 | 43.8 | 17.6 |
Candidate batch size. We also ablated the selection of different candidate batch size on Qwen3-1.7B Math at 150 training steps. Results below showed that larger candidate set size also encourages faster convergence with a slighly higher efficiency trade-off.
| MATH | AMC | AIME24 | AIME25 | AIME26 | Avg | Training hours | |
|---|---|---|---|---|---|---|---|
| US | 83.8 | 58.2 | 27.3 | 20.4 | 15.0 | 40.9 | 15.6 |
| 83.1 | 57.2 | 27.9 | 20.4 | 21.5 | 42.0 | 17.6 | |
| (Default) | 83.5 | 58.7 | 29.2 | 24.6 | 21.5 | 43.5 | 16.9 |
| 84.3 | 61.5 | 31.0 | 21.0 | 24.8 | 44.5 | 18.4 |
We thank the reviewer for the constructive questions, and would include the results in the final paper.
[1] Discounted Beta–Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards. Haechan Kim, Soohyun Ryu et al.
Authors' Response (Part 1)
We thank Reviewer aqgw for the thoughtful and constructive feedback. We address the raised concerns point by point below. We also uploaded a new pdf version, with the revised part highlighted in blue. We greatly appreciate the reviewer's time and questions. Please let us know if you have any remaining questions or concerns.
W1 & Q1. Empirical guarantee of history-based gain estimation
We thank the reviewer for this thoughtful question which helps us to evaluate the validity of our core consumption. To assess the discrepancy introduced by using historical rewards to compute the current reward-gain signal, we collect 1000 recent samples at training steps 50, 100, 150 and 200, and estimate both mean accuracy and G using (i) historical rewards and (ii) N=8 on-policy current rollouts. We treat N=32 current on-policy rollout estimates as the ground truth, and report Pearson correlations of each estimator with this ground truth in the table below.
We observe that the prompt difficulty is well preserved under history, with Acc correlations above 0.85 and increasing as training proceeds. The correlation for G^ is moderate-to-strong (0.65-0.75) using historical rewards. Since SW-UCB scheduling primarily needs relative ordering across prompts rather than calibrated absolute values, this level of agreement retains ranking information sufficient for our use case.