acceptodds
Under review as a conference paper at ICLR 2027

GRPODropout: Less is More for Online Reinforcement Learning Rollouts

Abstract

Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at https://anonymous.4open.science/r/GRPODropout/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Reinforcement Learning, GRPO, Policy Entropy Collapse, Rollout Selection, Data Efficiency, Reasoning Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Py8hOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method to mitigate entropy collapse by changing which generated rollouts contribute positively to an update. To do this, this paper proposes GRPODROPOUT to remove a small number of high-probability, positive-advantage rollouts. The main idea is to optimize task return while mitigating policy entropy (Shannon entropy) collapse. Main results show that GRPODroupout improves accuracy and mitigates entropy collapse. GRPODropout achieves much higher entropy at the end of training as well, with 12% less rollouts.

Critical strengths

The main strength of this paper is the simplicity of the method and the results that are quite transparent. As mentioned in the paper, GRPODropout doesn't perform well in all cases. The ablation studies are also rather comprehensive and the experiments also show both mathematical and general QA settings as well.

Critical weaknesses

The main concern is that GRPODropout doesn't perform well in majority of the cases compared to other methods. Given such results, I am not sure if it is valid to say removing rollouts is actually effective in general. It actually may be an random artifact of the experiment setting since rollouts are sampled at 1.0 temperature and top-p of 1.0, as it will basically select from any tokens. Also there is another work that is quite similar to this "Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning", which was not cited in this work.

Misc. Some part of the text was unorganized and hard to read. For example, problem formulation is not well organized. The work only was performed on Qwen family of models, so the generalizability of this work is questionable.

Questions for authors

How do we ensure that we get a consistent improvement? For the experiment settings, how are the hyperparameters chosen? Is there a reason for these values in B.1?

4 comments
AuthorsOpenReview ↗

Response to reviewer Py8h (1/2)

W1 (first half): The main concern is that GRPODropout doesn't perform well in majority of the cases compared to other methods. Given such results, I am not sure if it is valid to say removing rollouts is actually effective in general. It actually may be an random artifact of the experiment setting since rollouts are sampled at 1.0 temperature and top-p of 1.0, as it will basically select from any tokens.

Minor W2: The work only was performed on Qwen family of models, so the generalizability of this work is questionable.

Thank you for the comments. Both points test the robustness of the method and are answered in "Significance of the Gains" in the above Official Comment, where you can find the detailed results. What we aim to show is: GRPODropout is significantly better than the original GRPO with less data, surpasses existing baselines in entropy and is on par with them in accuracy; and it mitigates entropy collapse at the rollout level rather than the token or algorithm level, a new perspective on data selection for reinforcement learning. Since there have recently been no good small reasoning models other than Qwen, we test generalizability at the algorithm level, extending the method to PPO.

W1 (second half): Also there is another work that is quite similar to this "Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning", which was not cited in this work.

Thank you for pointing out this work; we will cite and discuss PODS [1] in the related work. Both first generate a group of rollouts and then select a subset for the update; they differ mainly in two points.

First, the purpose of deletion differs. PODS targets efficiency: it exploits the asymmetry that generation parallelizes easily while updates are expensive, generating more rollouts and updating on only a subset, to reach the same accuracy faster. GRPODropout targets entropy collapse: it deletes rollouts that reduce entropy, and deletion itself is how the method works, not a way to save compute.

Second, which and how many to delete differ. PODS uses only rewards and keeps the subset with the largest reward variance, i.e., it aims to keep the most diverse data for training and tends to keep the highest-reward data. GRPODropout does the opposite: using both advantages and old-policy probabilities, it shows that high-probability positive-advantage rollouts are counterproductive and accelerate entropy collapse, so it deletes only such rollouts, stopping once the within-group covariance is non-negative.

Therefore, although both work at the rollout level, the target problems, methods, and conclusions are entirely different. The two may even be complementary, and combining them may yield better results. Thank you for the suggestion; we will discuss this fully in a later version.

References

  • [1] Yixuan Even Xu, Yash Savani, Fei Fang, and J. Zico Kolter. Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning. Transactions on Machine Learning Research, 2026.

Minor W1: Some part of the text was unorganized and hard to read. For example, problem formulation is not well organized.

Thank you for the suggestion. We will split the problem formulation into three explicit parts: notation for the prompt, the full sequence space, and policy entropy; the sampled group and each rollout's sgs_g and A(g)A^{(g)}; and the objective. We will move the subscript conventions to a notation table in the appendix, and add an overview at the start of Section 3 explaining that Propositions 1–2 correspond to deleting positive-advantage rollouts by sgs_g until Cov(A,s)≥0\mathrm{Cov}(A,s)\ge 0, Proposition 3 to deleting only positive-advantage rollouts, and Proposition 4 to probability-weighted recentering, so that theory and algorithm steps correspond one to one.

AuthorsOpenReview ↗

Response to reviewer Py8h (2/2)

W1 (sampling setting): It actually may be an random artifact of the experiment setting since rollouts are sampled at 1.0 temperature and top-p of 1.0, as it will basically select from any tokens.

Q1: How do we ensure that we get a consistent improvement? For the experiment settings, how are the hyperparameters chosen? Is there a reason for these values in B.1?

Thank you for the questions. The choice of sampling temperature and of the other hyperparameters is one question, so we answer them together, then summarize the results on consistency from "Significance of the Gains".

(1) Training and evaluation hyperparameters follow established settings from prior work, are fixed in advance, and are identical for all six methods. For training, we follow SCOPE-RL [2] and EasyR1 [3]: training data DAPO-17K, 8 rollouts per prompt, AdamW with learning rate 1e-6, weight decay 0.01, clip ratio 0.2, maximum response length 8,192, and rollout sampling temperature 1.0 and top-p 1.0. We made only two changes: batch from 512 to 128 prompts and mini-batch from 128 to 64, and training length of 1 epoch, i.e., 110 steps, to fit three model scales, six methods, and all ablations in the compute budget. Training sampling with temperature 1.0 and top-p 1.0 is the established standard on-policy setting in this direction; DAPO's [4] official training script also uses temperature 1.0, top-p 1.0, and learning rate 1e-6, and its evaluation directly uses temperature 1.0; this setting is identical for all six methods, causes no difference between them, and is also what our method requires, since the surprisal sgs_g and the importance ratio are both defined relative to the untruncated old policy.

For evaluation, we use the thinking-mode settings officially recommended by Qwen3 [5]: temperature 0.6, top-p 0.95, top-k 20, maximum generation length 8,192, with the same protocol and the same step-110 checkpoint for all methods; that is, evaluation is not at temperature 1.0. Moreover, GRPODropout has no hyperparameters of its own to tune, as Cov(A,s)≥0\mathrm{Cov}(A,s)\ge 0 determines the deletion amount; all baselines follow their authors' recommended recipes.

(2) Consistency of the gains is answered in "Significance of the Gains" above; summary below. Our perspective: with less data, GRPODropout consistently surpasses existing baselines in entropy, consistently beats the original GRPO in accuracy, and is on par with the other baselines. The entropy gain is consistent: GRPODropout's entropy is above GRPO's at all ten checkpoints, its final-step entropy is the highest of the six methods, and under PPO + GAE it is also higher at all three scales. In accuracy, GRPODropout beats the original GRPO at all three scales; the question-level bootstrap on the 4B competition sets gives +5.42 points, with a 95% CI of [+2.66, +8.33], and the pooled gain across scales over six decoding seeds under PPO is +0.3168 points; it is on par with existing entropy-aware baselines.

References

  • [2] Chen Wang, Zhaochun Li, Jionghao Bai, Hexuan Deng, Guanting Dong, Ge Lan, and Yue Wang. SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training. In Advances in Neural Information Processing Systems (NeurIPS), 2026.
  • [3] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework. https://github.com/hiyouga/EasyR1, 2025.
  • [4] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. CoRR, abs/2503.14476, 2025.
  • [5] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report. CoRR, abs/2505.09388, 2025.

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Thank you for the comments. Initial concerns are still valid as far as value of contribution, the lack of experiments on other families of models (which do exist despite the authors claims), and the overall organization of the paper, I will keep my score.

Reviewer_9Sk7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRPODropout studies whether reinforcing some high-probability positive responses can hinder exploration. An entropy analysis motivates removing selected positive-advantage rollouts and recentering the remaining advantages. Filtering and weighting use length-normalized response scores. All rollouts are still generated; fewer are used for updates. Experiments cover three Qwen3 model sizes and ten math and general-QA benchmarks, with ablations and analyses of entropy and reasoning-path diversity.

Critical strengths
  1. Selecting whole rollouts is a useful and simple intervention. The method reuses rewards and log-probabilities without adding a value network. It improves average accuracy over GRPO at all three model sizes while using fewer rollouts for updates.

  2. The ablations establish the importance of recentering. Table 2 shows severe performance loss when positive rollouts are deleted without recentering. The paper also acknowledges that its theory motivates the design but does not guarantee entropy preservation in neural-network training.

Critical weaknesses
  1. The benefit of the selection rule remains unclear. Table 2 supports deletion with recentering, but lacks two controls: random positive deletion with the same recentering, and weighted recentering without deletion. These would show whether the proposed criterion adds value beyond simpler changes to sample usage and advantage weights.

  2. The diversity metric does not measure successful paths separately. Appendix B.2 uses all 32 responses, mixing correct and incorrect solutions. Section 4.3 nevertheless claims to preserve correct-answer path diversity. Neither this metric nor the overall rank establishes that claim; diversity among correct solutions needs to be measured directly.

Questions for authors
  1. On one model, could you test random positive deletion with the same deletion count and recentering, plus weighted recentering without deletion?

  2. Could you measure diversity among correct solutions at matched sample counts, including how you handle too few correct samples? Please also report overall path diversity across all 130 questions, including the 28 omitted from the split analysis.

  3. The average accuracy gaps to strong baselines are small. Could you report independent training-seed results for one representative comparison?

5 comments
AuthorsOpenReview ↗

Response to reviewer 9Sk7 (1/3)

W1: The benefit of the selection rule remains unclear. Table 2 supports deletion with recentering, but lacks two controls: random positive deletion with the same recentering, and weighted recentering without deletion. These would show whether the proposed criterion adds value beyond simpler changes to sample usage and advantage weights.

Q1: On one model, could you test random positive deletion with the same deletion count and recentering, plus weighted recentering without deletion?

Thank you for the comments, which help us show the effectiveness of our method more fully. Details follow:

(1) Count-matched random positive deletion: random deletion is also effective, but our criterion is better in both entropy and accuracy. Setup: for each group, we first compute the number kk of positive-advantage rollouts the main method would delete, then delete kk of the group's positive-advantage rollouts uniformly at random; rollouts with non-positive advantages follow the main method's decisions. Every group thus keeps exactly the same number and positive/negative composition as the main method, differing only in which ones are deleted, which isolates the selection criterion as the single factor; all other settings also match the main method item by item. We train Qwen3-4B for 110 steps; the table gives ten-benchmark results in the format of Table 1, evaluation entropy is the token-averaged entropy on the four competition sets as in Table 3, and the highest value per column is in bold:

MethodGSM8kMath500AMC23OlympiadAIME24AIME25HMMT25
GRPO94.7791.8089.4558.3742.6033.2318.96
Random positive deletion94.5490.6088.9859.1148.6542.6025.63
GRPODropout94.7791.8088.2858.9651.9843.0225.94
MethodARC-CMMLUPSGPQAAvg.Eval. entropy
GRPO94.1168.3538.6463.030.0448
Random positive deletion94.0368.9240.3465.340.1855
GRPODropout94.1169.3540.2565.850.2291

Our criterion is better in both entropy and accuracy: evaluation entropy 0.2291 vs. 0.1855, step-110 training entropy 0.3980 vs. 0.3429, ten-benchmark average 65.85 vs. 65.34, and four-competition-set average 52.31 vs. 51.46. Random deletion keeps most of the effect because count matching makes it delete only 0.9 rollouts per group on average, and deleting the same number of positive rollouts in the same rounds makes the two overlap heavily: 51.6% of the rollouts it deletes are ones our criterion would also delete. With such a small candidate set, random selection has a considerable chance of hitting the criterion's choices, so this control is effectively a diluted version of our method, and the gap is correspondingly small. In a later version, we will add results for more random deletion strategies to further test our method.

(2) Probability-weighted recentering without deletion is almost identical to the original GRPO. In our method, recentering corrects the distribution shift caused by deletion. Without deletion, the 8 rollouts in a group are still random samples of the policy, and GRPO's standard equal-weight group-mean centering is already unbiased; replacing it with probability weighting is only a very small correction on top: as described in Section 3.2, advantages and log-probabilities tend to be positively correlated, so the probability-weighted mean is generally not exactly zero. This configuration barely changes the original GRPO, so we expect its results to be close to the original GRPO.

Table 2 gives an indirect reference for this expectation: with only positive rollouts deleted, replacing probability-weighted with equal-weight recentering changes the 4B average by only 0.03 points, 65.82 vs. 65.85, with almost unchanged entropy, 0.2174 vs. 0.2140. That is, at least at 4B, the advantage weighting barely changes results, and the main method's gain over GRPO, from 63.03 to 65.85, can hardly be attributed to the weighting itself. This reference was measured with deletion, and at 1.7B the weighting contributes +2.82 points, from 52.19 to 55.01. Due to time constraints, we will add this control in a later version.

We will add count-matched random positive deletion to Table 2; the same control on Qwen3-1.7B is training and will be reported during the discussion period once complete; weighted recentering without deletion will be added in a later version.

AuthorsOpenReview ↗

Response to reviewer 9Sk7 (2/3)

W2: The diversity metric does not measure successful paths separately. Appendix B.2 uses all 32 responses, mixing correct and incorrect solutions. Section 4.3 nevertheless claims to preserve correct-answer path diversity. Neither this metric nor the overall rank establishes that claim; diversity among correct solutions needs to be measured directly.

Q2: Could you measure diversity among correct solutions at matched sample counts, including how you handle too few correct samples? Please also report overall path diversity across all 130 questions, including the 28 omitted from the split analysis.

Thank you for pointing this out. The Summary distance and Vendi in Appendix B.2 use all 32 responses per question, which indeed mix correct and incorrect solutions. Our original argument was indirect: on the ≥8 subset, all six methods have at least 8 correct responses per question, GRPODropout has the highest path diversity there, and its number of distinct wrong answers WW is lower than all baselines except KL-Cov, so its diversity gain comes mainly from correct solutions, not more scattered wrong answers. To answer directly, we keep the main-text threshold of 8 and re-measure diversity using only correct responses on the ≥8 subset and only incorrect responses on the <8 subset, where correct responses are too few, with paired tests at matched sample counts: GRPODropout's correct-solution diversity is significantly higher than GRPO's and ranks first in average rank; overall path diversity on all 130 questions, including the 28 omitted ones, also ranks first. Specifically:

(1) Measuring the diversity of correct and incorrect solutions separately at the main-text threshold of 8. We keep the question split and metric definitions of Table 3, with one refinement: on easy questions, we encourage diversity of correct paths, so the model stays diverse without losing accuracy; on hard questions, we encourage diversity of incorrect paths, so across rollouts the model explores different kinds of paths and is more likely to reach the correct answer. The ≥8 subset is the 65 questions on which all six methods have at least 8 correct responses; here Summary distance and Vendi use only correct responses, directly measuring correct-solution diversity. The <8 subset is the 37 questions on which all six methods have fewer than 8 correct responses; here they use only incorrect responses, measuring whether the paths explored on hard questions are diverse, and are reported with the number of distinct wrong answers WW. As in Table 3, the five columns are ranked separately and the ranks averaged equally; the highest value per column is in bold:

Method≥8 correct Summary≥8 correct Vendi<8 incorrect Summary<8 incorrect Vendi<8 WW ↓Avg. rank ↓
GRPO0.09981.75060.16392.41627.625.6
DAPO0.10711.80930.17972.59678.084.4
Clip-Cov0.10631.81120.21392.97465.222.8
KL-Cov0.10651.81350.20162.84723.002.8
Beyond80200.10731.82090.17262.51977.843.6
GRPODropout0.10751.81820.20912.89614.241.8

On the ≥8 subset, GRPODropout ranks first in correct-solution Summary distance and second in Vendi. On the <8 subset, it ranks second on both incorrect-solution diversity measures, behind only Clip-Cov and significantly above GRPO, DAPO, and Beyond8020, while its WW is only 4.24, the second fewest of the six methods. That is, on hard questions GRPODropout tries more different solution paths while its final wrong answers are more concentrated: it explores different solutions rather than guessing answers more diffusely.

To rule out the effect of the number of correct / incorrect responses, we add a further analysis. For ≥8 correct solutions, we draw 8 correct responses per method and question without replacement, 50 times, and average; GRPODropout rises to first on both measures, 0.1079 and 1.5961. For <8 incorrect solutions, matching sample counts with 25 incorrect responses per question leaves the conclusion unchanged. Across the five columns, GRPODropout is in the top two in every column, with an average rank of 1.8, first of the six methods.

AuthorsOpenReview ↗

Response to reviewer 9Sk7 (3/3)

(2) Overall path diversity on all 130 questions, including the 28 omitted ones. To address concerns about the 28 omitted questions, we add averages over all questions. The protocol is exactly that of Table 3: all 32 responses per question, including incorrect solutions, equal weight per question; the only difference is that the question set expands from the two subsets to all 130.

Methodpass@32Summary distanceVendiWW ↓EntropyAvg. rank ↓
GRPO63.540.13842.16034.43850.04485.8
DAPO69.380.14922.27664.96150.17964.2
Clip-Cov76.880.15872.39032.56150.09002.2
KL-Cov72.710.15522.34511.49230.15093.0
Beyond802073.540.14422.22274.05380.16714.0
GRPODropout75.210.15852.36851.94620.22911.8

pass@32 and entropy are from Table 3, and the average rank ranks the five columns separately and averages equally, as in Table 3. GRPODropout ranks first in entropy and second in the other four, with an average rank of 1.8, first of the six methods, so omitting the 28 questions introduced no bias. In fact, GRPODropout also performs best on these 28 questions. The table below uses all 32 responses per question with the Table 3 protocol, and the average rank ranks the four columns separately and averages equally:

MethodSummary distanceVendiWW ↓pass@32Avg. rank ↓
GRPO0.17912.58207.7964.295.25
DAPO0.18822.70318.6185.714.75
Clip-Cov0.18982.73303.2196.432.13
KL-Cov0.19122.72611.8992.862.25
Beyond80200.17662.56335.8689.295.00
GRPODropout0.19272.72842.2996.431.63

GRPODropout ranks first in Summary distance and second in Vendi and WW, and its pass@32 of 96.43 ties Clip-Cov for first, with an average rank of 1.63, first of the six methods; GRPO is near the bottom on all four. That is, omitting these questions actually removed a part where our method has the advantage.

We sincerely thank the reviewer for this suggestion, which helps make the paper more complete. In the revision, the main text will adopt the protocol of point (1), measuring correct-solution diversity on the ≥8 subset and incorrect-solution diversity on the <8 subset; the conclusions of point (2) on all 130 questions and the 28 omitted ones, with the related discussion, will go into the appendix.

Q3: The average accuracy gaps to strong baselines are small. Could you report independent training-seed results for one representative comparison?

Thank you for the question. It is answered in "Significance of the Gains" in the above Official Comment. Our perspective: GRPODropout is positioned to be significantly better than the original GRPO with less data while on par with strong baselines in accuracy, and to surpass all baselines in entropy; it also selects data at the rollout level, a different granularity from baselines that improve at the token or algorithm level, making it a complementary option.

Reviewer_8BgMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives each rollout's first-order contribution to entropy change. GRPODropout greedily removes only positive-advantage rollouts with the most negative entropy contributions until Cov≥0. Experiments use Qwen3-1.7B/4B/8B trained on DAPO-17K for one epoch, evaluated on ten benchmarks. GRPODropout beats GRPO by 1.55 points on average and KL-Cov by only 0.11.

Critical strengths

The rollout-level entropy decompositionis simple and interpretable. It connects GRPO's entropy collapse to specific rollouts that can be acted on.

Prop. 4 correctly identifies the first-order bias introduced by subsampling a group. Its practical importance is confirmed empirically: without recentering, training collapses (4B average 16.00).

Critical weaknesses

The gains are marginal and not statistically supported. Training is one epoch, apparently with a single seed. Evaluation uses a fixed seed and Mean@1 on several benchmarks. No variance, seed count, or CI is reported. Differences of 0.1–1.5 points cannot be interpreted.

Dropping is a small intervention. Only about one rollout per group is dropped (7.03/8). Dropping only negative-advantage rollouts (4B 65.30) performs about the same as the proposed positive-only variant (65.85).

The theory is not what is implemented. The analysis assumes a KL-proximal, first-order setting with independent logits and uses whole-sequence probabilities.

Questions for authors

How many training seeds were run per configuration? Please report mean ± std (or CIs) for the main table and the ablations.

How sensitive are the results to using length-normalized surprisal instead of sequence log-probability, which is what the theory prescribes?

4 comments
AuthorsOpenReview ↗

Response to reviewer 8BgM (1/3)

W1: The gains are marginal and not statistically supported. Training is one epoch, apparently with a single seed. Evaluation uses a fixed seed and Mean@1 [...]

Q1: How many training seeds were run per configuration? Please report mean ± std (or CIs) for the main table and the ablations.

Thank you for the comments. We use Mean@1 only on the six large benchmarks: GSM8K 1,319 questions, MATH-500 500, OlympiadBench 675, ARC-Challenge 1,172, MMLU-Pro 12,032, and SuperGPQA 26,529, where single-sample randomness is mainly averaged out by the number of questions; the four competition sets have only 30–40 questions, AMC23 40 and AIME24, AIME25, and HMMT25 30 each, so we use Mean@32.

The statistical support is given in "Significance of the Gains" in the above Official Comment. We address this from three angles: first, GRPODropout uses less data than the original GRPO yet is significantly better, i.e., better results with less data; second, it surpasses existing baselines in entropy and is on par with them in accuracy, making it a viable option; third, it mitigates entropy collapse at the rollout level rather than the token or algorithm level, a data-selection perspective at a different granularity whose value is not reflected only in accuracy gaps.

W2: Dropping is a small intervention. Only about one rollout per group is dropped. Dropping only negative-advantage rollouts performs about the same as the proposed positive-only variant.

Thank you for the comment. Following Occam's razor, a small deletion is exactly the design goal we pursue, and the core idea of this paper: mitigate entropy collapse with as small a change as possible, minimally perturbing the original training process. Section 3.3 states this principle: "we aim to minimize perturbations to the original training process and preserve the original algorithm and data distribution as much as possible". Our variants that delete only positive or only negative rollouts serve the same purpose: deleting as little data as possible, to avoid disrupting the distribution of randomly sampled data and under-using the information in existing data. We want to delete only harmful data and keep useful or neutral data as much as possible, a minimal modification faithful to Occam's razor. Therefore, we adopt deleting only positive rollouts as our default. Specifically:

(1) How small the change is. All statistics are on Qwen3-4B. Over all 110 steps, only 19.4% of positive-advantage rollouts and 3.9% of all are deleted per step on average. Among groups with non-zero advantages, 44.7% delete none, and the rest delete only 1.6 on average. Hence, each prompt keeps 7.03 rollouts on average, only 12.1% fewer than GRPO's 8.

GRPODropout keeps the loss function and objective form unchanged, with no new regularization term or hyperparameter to tune. Despite this small change, final-step entropy rises by 446.1% over GRPO on the training set and by 256.0% on the held-out set. For comparison, Beyond8020 masks 80% of tokens, and its final-step training entropy of 0.303 is still below our 0.426. A large effect from a very small change is exactly what "less is more" means.

(2) Deleting more hurts accuracy, which is why minimal perturbation should be pursued. Deleting data loses information, and the more is deleted, the more training may deviate from the original. Table 2 directly supports this. At 4B, deleting both positive and negative rollouts gives the highest entropy, 0.2425, but the lowest accuracy of the three deletion targets, 65.13 < 65.30 < 65.85; likewise at 1.7B, 54.02 < 55.01. Section 4.2 states accordingly: "Deleting both positive and negative candidates yields the largest entropy gain but incurs a slight accuracy loss, showing that data removal can sacrifice performance and supporting our motivation to delete as little data as possible." That is, "the less deleted, the better".

(3) When too much is deleted, the accuracy gain from training vanishes completely. On Qwen3-1.7B, we also trained a more aggressive variant with the same entropy-contribution criterion, also deleting mainly positive-advantage rollouts; it differs in deletion amount: the main method deletes as few rollouts as possible so that Cov(A,s)≥0\mathrm{Cov}(A,s)\ge 0 just holds, while this variant deletes all positive-advantage rollouts that decrease entropy. It deletes 205.1 rollouts per step on average, 83.6% of positive-advantage rollouts and 20.0% of all, about 4.7 times the main method at the same scale. Its entropy indeed does not collapse: step-110 training entropy is 0.334, far above GRPO's 0.097; but its step-110 mean@16 on the AMC23 validation set during training is only 59.84, identical to the untrained base model, versus 69.84 for GRPO and 73.91 for GRPODropout. That is, once deletion exceeds what is needed to remove the negative covariance, what is deleted is no longer only harmful data but the positive signal training relies on.

AuthorsOpenReview ↗

Response to reviewer 8BgM (2/3)

W3: The theory is not what is implemented. The analysis assumes a KL-proximal, first-order setting with independent logits and uses whole-sequence probabilities.

Q2: How sensitive are the results to using length-normalized surprisal instead of sequence log-probability, which is what the theory prescribes?

Thank you for pointing out the gap between theory and implementation and for asking about sensitivity to length normalization. In fact, our theory already goes beyond existing theory and makes further progress. We do not claim the current theory is complete; rather, it motivates the experimental design, and the experimental findings in turn verify it. The Limitations and the appendix paragraph "Length normalization in implementation" already explain this. For sensitivity to length normalization, we ran a dedicated additional ablation. Overall conclusions:

(1) Assumptions: in two respects, advantages and the update rule, this paper is closer to real training than related work. On advantages, Cui et al. [1] assume the true advantage of every action is known and take the expectation over all actions the policy may generate, obtaining the entropy change in an average sense; but real training has no true advantages: GRPO can only estimate group-relative advantages from the 8 responses sampled per prompt, and updates only these 8. This paper conditions directly on this group of actually sampled rollouts and their group-relative advantages, characterizing the entropy change of this very update; the only condition needed, ∑gA(g)=0\sum_g A^{(g)}=0, holds naturally once the group mean is subtracted. Precisely because the result applies to each specific sampled rollout, it can decide which ones to delete. On the update rule, Cui et al. [1] build on natural policy gradient, which the authors themselves note is rarely used in LLM post-training; besides the KL-proximal update, our Proposition 3 also analyzes plain gradient updates with one learning rate shared by all parameters, closer to actual training. On independent logits and not modeling parameter sharing and AdamW, we are the same as related work.

(2) Theoretical content: this paper goes beyond related work in rollout-level entropy analysis and post-deletion recentering. The first is the rollout level. Related work [1–3] analyzes a single token or state, and the corresponding methods all act at the token level; our Proposition 1 decomposes the first-order entropy change into each complete rollout in the group, giving the rollout-level contribution hg=A(g)G(sg−H)h_g=\frac{A^{(g)}}{G}(s_g-H), and Proposition 2 gives the condition Cov(A,s)≥0\mathrm{Cov}(A,s)\ge 0 under which the within-group estimated entropy does not decrease. The second is recentering. Related work all analyzes complete sampled groups without subsampling; our Proposition 4 assumes only that the original group is i.i.d., allows the retained set after deletion to no longer be a random sample of the policy, and proves that sign-biased deletion shifts all unsampled zero-advantage paths as a whole, from which probability-weighted recentering follows. These two points are conclusions we add that prior theory does not cover, and experiments confirm them.

(3) The theory inspires the design and is tested by experiments, which support it. Although our theory goes beyond prior work to some extent, it still makes some strong assumptions. In fact, existing theoretical frameworks can hardly fully explain the complex phenomena of training; the Limitations states: "given the complexity of RL training, our theory itself cannot fully explain our method. In fact, the primary role of the theory is to serve as motivation for method design, acting as the initial starting point. The final method is validated through its advantages over the original GRPO and the ablation study results." This also follows common practice in this direction. Each of the three theoretical predictions matches an experimental result. Proposition 2 predicts that entropy no longer collapses once Cov(A,s)≥0\mathrm{Cov}(A,s)\ge 0 is enforced; in practice, final-step entropy is 446.1% above GRPO on the training set and 256.0% above on the held-out set, and higher at all ten checkpoints of Figure 2. Proposition 3 predicts that deleting positive-advantage rollouts affects entropy more; in Table 2, deleting only positive-advantage rollouts gives entropy 0.2140, above 0.2043 for deleting only negative-advantage ones. Proposition 4 predicts that deleting without recentering shifts all zero-advantage paths as a whole; in Table 2, training without recentering collapses, with averages of only 16.00 and 28.81 at 4B and 1.7B, and probability weighting beats equal weighting by 2.82 points at 1.7B.

AuthorsOpenReview ↗

Response to reviewer 8BgM (3/3)

(4) Why the implementation uses length normalization, and a sensitivity ablation on this choice. The implementation uses length normalization for theoretical and practical reasons.

Theoretically, length normalization is the approximate counterpart of our theory under GRPO's actual loss. In the propositions, sgs_g is the surprisal of the whole response, i.e., the sum of its token surprisals; but GRPO actually uses a token-averaged loss, dividing each rollout's gradient by its length TgT_g, which amounts to replacing the advantage A(g)A^{(g)} with A(g)/TgA^{(g)}/T_g. Substituting this into Proposition 1's entropy contribution hg=A(g)G(sg−H)h_g=\frac{A^{(g)}}{G}(s_g-H) yields exactly sg/Tgs_g/T_g, the average per-token surprisal. Length normalization is thus not an engineering choice detached from the theory, but the form that naturally arises when the theory meets the actual loss; an extra term H/TgH/T_g remains, so the two are not strictly equal, which is why the paper calls them "implementation proxies".

Practically, without normalization, length overrides everything else. Whole-sequence surprisal grows linearly with length, and responses reach 8,192 tokens, so long responses are almost always judged "low-probability" and short ones "high-probability", and short correct responses are deleted first; the recentering weights e−sge^{-s_g} would also concentrate almost entirely on the shortest response in the group. The criterion would then effectively select by length, not by entropy contribution.

Results. To test this choice, we ran an additional ablation on Qwen3-4B that only replaces the selection signal with whole-sequence log-probability, sum, keeping recentering and all other settings as in the main method, including the 128×8 batch, clip 0.2, probability-weighted recentering, 110 training steps, and decoding settings. The table gives ten-benchmark results in the format of Table 1; evaluation entropy is the token-averaged entropy on the four competition sets as in Table 3, and the highest value per column is in bold:

MethodGSM8kMath500AMC23OlympiadAIME24AIME25HMMT25
GRPO94.7791.8089.4558.3742.6033.2318.96
Whole-sequence log-prob sum94.4792.0087.5057.7851.4640.0025.63
GRPODropout94.7791.8088.2858.9651.9843.0225.94
MethodARC-CMMLUPSGPQAAvg.Eval. entropy
GRPO94.1168.3538.6463.030.0448
Whole-sequence log-prob sum94.4569.2240.2765.280.2008
GRPODropout94.1169.3540.2565.850.2291

Length normalization is consistently better in both entropy and accuracy. Excluding all-zero-advantage fallback groups, the sum variant deletes 1.320 rollouts per group on average, 46% more than GRPODropout's 0.901, yet is worse on both: evaluation entropy 0.2008 vs. 0.2291, ten-benchmark average 65.28 vs. 65.85, and four-competition-set average 51.15 vs. 52.31; its step-110 training entropy is also below GRPODropout's 0.3980. Per deleted rollout, GRPODropout gains 0.2045 entropy, the sum variant only 0.1182. This is the degradation predicted above: sequence log-probability is mainly determined by length, reducing filtering to "delete short correct responses first", so more must be deleted to push the within-group covariance to non-negative, and what is deleted is not the rollouts with the most negative entropy contributions. This maps directly to the "minimal perturbation" design goal: length normalization gains more entropy with fewer deletions.

Our criterion is not fragile to this design choice. With the selection signal replaced by the whole-sequence log-probability used in the theory, entropy preservation remains significant: evaluation entropy 0.2008 is 4.48 times GRPO's, step-110 training entropy 0.3711 and the last-10-step mean 0.3554 are far above GRPO's 0.0702 and 0.0660, and the ten-benchmark average of 65.28 is clearly above GRPO's 63.03.

References

  • [1] Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. CoRR, abs/2505.22617, 2025.
  • [2] Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective. CoRR, abs/2510.10150, 2025.
  • [3] Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest K. Ryu. Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models. CoRR, abs/2509.26114, 2025.

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Significance of the Gains (2/2)

Comparisonθ\theta, points95% CISign test pp
GRPO+5.42[+2.66, +8.33]0.038
DAPO+4.64[+2.13, +7.22]0.029

Relative to the original GRPO, GRPODropout is 5.42 points higher on the four competition benchmarks; the 95% CI [+2.66, +8.33] excludes 0, and the per-question sign test, 43–25, gives pp = 0.038, also significant. The same holds relative to DAPO. This matches the main table: GRPODropout outperforms the original GRPO at all three scales, by 1.55 points on the ten-benchmark average, and by +0.88, +2.82, and +0.96 at the three scales. Hence GRPODropout is significantly better than the original GRPO. We do not claim to surpass all baselines; rather, under fair comparison, our method surpasses the original GRPO with less data, surpasses other work in entropy, and is on par with other work in accuracy.

Part 3: Applying the same selection rule to PPO + GAE

To test our method in broader settings, and since there are no good small reasoning models other than the Qwen series, we instead test its generalizability across algorithms, on PPO. We compare two training configurations: PPO samples 8 trajectories per prompt and updates on all of them; PPODropout is identical except that our selector is on. Both use GAE and a value network; since deleting rollouts does not change the GAE advantages of the retained ones, no recentering is needed here. Each configuration is run at three Qwen3 scales, 1.7B, 4B, and 8B, for 110 steps, with all other hyperparameters as in the main experiments. For evaluation, we fix the step-110 checkpoint and run one full ten-benchmark evaluation with each of seeds 3407–3412, with the protocol of Table 1: Mean@32 on the four competition sets and Mean@1 on the other six, decoding temperature 0.6, top-p 0.95, top-k 20, and maximum output 8192. Final-step training entropy is the token-averaged actor entropy of the step-110 rollouts; held-out entropy is computed with teacher forcing on 50 fixed responses. Accuracy below is the mean over six decoding seeds, and the higher value within each scale is in bold:

ScaleMethodGSM8kMath500AMC23OlympiadAIME24AIME25HMMT25
1.7BPPO90.3682.7362.8144.5424.4623.7513.26
1.7BPPODropout89.9384.3363.1046.2526.2523.4714.13
4BPPO94.4090.7783.4555.2350.3038.2123.99
4BPPODropout94.6390.1783.4555.6048.9638.7224.44
8BPPO95.8290.9083.5856.4751.8938.2821.67
8BPPODropout95.7990.5784.5156.8653.3539.8421.79
ScaleMethodARC-CMMLUPSGPQAAvg.Training entropyHeld-out entropy
1.7BPPO88.0456.1828.8451.500.08940.0874
1.7BPPODropout88.0156.0528.8852.040.15480.1425
4BPPO94.1368.7139.4663.870.05960.0749
4BPPODropout94.0869.0439.8963.900.07510.0909
8BPPO95.7373.5844.3965.230.06320.0740
8BPPODropout95.5873.4244.3565.610.07900.0891

(1) Entropy: the same rule also raises entropy under PPO, with the effect decreasing with scale. With the selector on, final-step training entropy exceeds the control at all three scales, by +0.065, +0.016, and +0.016, and held-out entropy moves the same way. The intervention is smaller than under GRPO: only 4–5% of trajectories are deleted per step on average, consistent with our design goal of "minimal perturbation".

(2) Accuracy: under six decoding seeds, turning on the selector brings a significant gain. With the selector on, the ten-benchmark average is higher at all three scales: 52.04 vs. 51.50 at 1.7B, 63.90 vs. 63.87 at 4B, and 65.61 vs. 65.23 at 8B. To show significance, within each seed we average the ten benchmarks equally, then average the difference between the two configurations equally over the three scales, so each seed contributes one paired point, 6 in total. The pooled differences of the six test seeds are +0.3843, +0.3827, +0.1466, +0.4881, +0.2601, and +0.2388 points, all positive; the mean gain is +0.3168 points, the 95% CI [+0.1870, +0.4466] excludes 0, the paired tt-test gives pp = 0.0015, and the exact paired label-swap test gives pp = 0.0312, so both parametric and non-parametric tests pass.

These results show that our method consistently outperforms the original algorithm across algorithms and model scales; with less data, the final entropy and results are significantly better, i.e., better results with less data. They also show that rollout-level methods can be effective, and we hope this offers further options and insights for entropy mitigation.

References

  • [1] Ganqu Cui et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. CoRR, abs/2505.22617, 2025.
AuthorsOpenReview ↗

Significance of the Gains (1/2)

We thank the reviewers for their attention to statistical support, consistency of the gains, and generalizability. Below, we show that our method uses less data than the original GRPO yet achieves significantly better results, i.e., better results with less data: less is more. We also surpass the other baselines in entropy while not falling behind them in accuracy, so our method is a viable option. Most importantly, our method works at a different granularity: these entropy-mitigation methods mostly improve at the token or algorithm level, so our rollout-level theory, findings, and insights fill a research gap, offer valuable insights into data selection for reinforcement learning, and make a valuable contribution to the community. We answer in three parts.

  • Part 1 shows that the entropy effect in the main GRPO experiments is significant. This is the core effect of the paper.
  • Part 2 runs a random resampling test on the existing evaluation results of the main experiments, answering whether the accuracy gap is significant, and narrows the accuracy claim accordingly.
  • Part 3 is a set of supplementary extension experiments: it shows that our selection rule generalizes to more RL algorithms, namely PPO + GAE + value network, over three model scales and nine 110-step training runs; on the fixed final-step checkpoint, we run one full ten-benchmark evaluation with each of six independent decoding seeds.

Part 1: The entropy effect in the main GRPO experiments is significant

The core goal of this paper is to better preserve entropy without losing accuracy. The results show that we clearly surpass existing work on this point.

Figure 2 shows that GRPODropout preserves entropy throughout training, while GRPO suffers entropy collapse. Figure 2 records the token-averaged entropy of Qwen3-4B at ten training checkpoints, steps 11, 22, …, 110, with entropy defined as in Cui et al. [1]. On the training set DAPO-17K, GRPO's entropy falls overall from 0.214 to 0.078 at the final step, with a minimum of 0.066 at step 99, a clear entropy collapse; GRPODropout's entropy rises monotonically from 0.254 to 0.426, is higher than GRPO's at all ten checkpoints, and is 446.1% higher at the final step. The held-out set MMLU-Pro gives the same conclusion: higher at all ten checkpoints and 256.0% higher at the final step. Hence the entropy gain persists throughout training and generalizes beyond the training distribution. Meanwhile, Figure 2(c) shows that GRPODropout updates on only 7.03 rollouts per prompt on average, 12.1% fewer than GRPO, i.e., it preserves more entropy with less data.

Table 3 shows that at the final checkpoint, GRPODropout has the highest entropy of all methods. On the final Qwen3-4B checkpoint, Table 3 samples 32 responses per question on the 130 questions of four competition benchmarks, AMC23, AIME24, AIME25, and HMMT25, and computes token-averaged entropy. GRPODropout reaches 0.2291, over 5 times GRPO's 0.0448, and higher than all entropy-aware baselines: DAPO 0.1796, Beyond8020 0.1671, KL-Cov 0.1509, Clip-Cov 0.0900; even the closest, DAPO, is about 22% lower than ours. These baselines either mask training signals at the token level or modify the objective and clipping rules; GRPODropout only changes which rollouts enter the update, yet obtains the highest entropy.

Table 2 shows that this effect holds at two scales and under multiple deletion configurations. The Table 2 ablation compares deletion targets and advantage treatments on Qwen3-1.7B and Qwen3-4B. Every deletion configuration with recentering has far higher entropy than the original GRPO: 0.2043–0.2425 at 4B and 0.1739–0.2239 at 1.7B, versus only 0.0528 and 0.0503 for GRPO. Our method is about 4 and 3.5 times GRPO at the two scales. This shows the robustness of our method in entropy.

Part 2: Random resampling test of accuracy in the main GRPO experiments on the current results

We next show that our accuracy is significantly better than the original GRPO's. To test whether the accuracy gap is significant, we resample questions directly on the existing evaluation outputs of the main experiments. The test covers the four competition benchmarks of Qwen3-4B, AMC23, AIME24, AIME25, and HMMT25, 130 questions in total, i.e., the Mean@32 subset of the main experiments. Taking each question's accuracy over 32 responses as the unit, we compute the per-question difference between GRPODropout and the comparison method, average it within each benchmark, then average the four benchmarks equally to obtain the difference θ\theta. We then resample questions with replacement within each benchmark 10,000 times, with both methods always drawing the same questions to keep pairing, and take the 2.5% and 97.5% quantiles as the 95% confidence interval. We also report a per-question sign test: looking only at which method scores higher on each question, we drop ties and run a two-sided exact binomial test. Results:

The authors have posted their rebuttal, and the discussion period will end soon. Could you please read it and reply at your earliest convenience, stating whether your concerns have been addressed and whether your rating changes? Thank you so much for your active participation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.