- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method to mitigate entropy collapse by changing which generated rollouts contribute positively to an update. To do this, this paper proposes GRPODROPOUT to remove a small number of high-probability, positive-advantage rollouts. The main idea is to optimize task return while mitigating policy entropy (Shannon entropy) collapse. Main results show that GRPODroupout improves accuracy and mitigates entropy collapse. GRPODropout achieves much higher entropy at the end of training as well, with 12% less rollouts.
Critical strengths
The main strength of this paper is the simplicity of the method and the results that are quite transparent. As mentioned in the paper, GRPODropout doesn't perform well in all cases. The ablation studies are also rather comprehensive and the experiments also show both mathematical and general QA settings as well.
Critical weaknesses
The main concern is that GRPODropout doesn't perform well in majority of the cases compared to other methods. Given such results, I am not sure if it is valid to say removing rollouts is actually effective in general. It actually may be an random artifact of the experiment setting since rollouts are sampled at 1.0 temperature and top-p of 1.0, as it will basically select from any tokens. Also there is another work that is quite similar to this "Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning", which was not cited in this work.
Misc. Some part of the text was unorganized and hard to read. For example, problem formulation is not well organized. The work only was performed on Qwen family of models, so the generalizability of this work is questionable.
Questions for authors
How do we ensure that we get a consistent improvement? For the experiment settings, how are the hyperparameters chosen? Is there a reason for these values in B.1?
4 comments
Response to reviewer Py8h (2/2)
W1 (sampling setting): It actually may be an random artifact of the experiment setting since rollouts are sampled at 1.0 temperature and top-p of 1.0, as it will basically select from any tokens.
Q1: How do we ensure that we get a consistent improvement? For the experiment settings, how are the hyperparameters chosen? Is there a reason for these values in B.1?
Thank you for the questions. The choice of sampling temperature and of the other hyperparameters is one question, so we answer them together, then summarize the results on consistency from "Significance of the Gains".
(1) Training and evaluation hyperparameters follow established settings from prior work, are fixed in advance, and are identical for all six methods. For training, we follow SCOPE-RL [2] and EasyR1 [3]: training data DAPO-17K, 8 rollouts per prompt, AdamW with learning rate 1e-6, weight decay 0.01, clip ratio 0.2, maximum response length 8,192, and rollout sampling temperature 1.0 and top-p 1.0. We made only two changes: batch from 512 to 128 prompts and mini-batch from 128 to 64, and training length of 1 epoch, i.e., 110 steps, to fit three model scales, six methods, and all ablations in the compute budget. Training sampling with temperature 1.0 and top-p 1.0 is the established standard on-policy setting in this direction; DAPO's [4] official training script also uses temperature 1.0, top-p 1.0, and learning rate 1e-6, and its evaluation directly uses temperature 1.0; this setting is identical for all six methods, causes no difference between them, and is also what our method requires, since the surprisal and the importance ratio are both defined relative to the untruncated old policy.
For evaluation, we use the thinking-mode settings officially recommended by Qwen3 [5]: temperature 0.6, top-p 0.95, top-k 20, maximum generation length 8,192, with the same protocol and the same step-110 checkpoint for all methods; that is, evaluation is not at temperature 1.0. Moreover, GRPODropout has no hyperparameters of its own to tune, as determines the deletion amount; all baselines follow their authors' recommended recipes.
(2) Consistency of the gains is answered in "Significance of the Gains" above; summary below. Our perspective: with less data, GRPODropout consistently surpasses existing baselines in entropy, consistently beats the original GRPO in accuracy, and is on par with the other baselines. The entropy gain is consistent: GRPODropout's entropy is above GRPO's at all ten checkpoints, its final-step entropy is the highest of the six methods, and under PPO + GAE it is also higher at all three scales. In accuracy, GRPODropout beats the original GRPO at all three scales; the question-level bootstrap on the 4B competition sets gives +5.42 points, with a 95% CI of [+2.66, +8.33], and the pooled gain across scales over six decoding seeds under PPO is +0.3168 points; it is on par with existing entropy-aware baselines.
References
- [2] Chen Wang, Zhaochun Li, Jionghao Bai, Hexuan Deng, Guanting Dong, Ge Lan, and Yue Wang. SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training. In Advances in Neural Information Processing Systems (NeurIPS), 2026.
- [3] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework. https://github.com/hiyouga/EasyR1, 2025.
- [4] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. CoRR, abs/2503.14476, 2025.
- [5] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report. CoRR, abs/2505.09388, 2025.
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.
Thank you for the comments. Initial concerns are still valid as far as value of contribution, the lack of experiments on other families of models (which do exist despite the authors claims), and the overall organization of the paper, I will keep my score.
Response to reviewer Py8h (1/2)
Thank you for the comments. Both points test the robustness of the method and are answered in "Significance of the Gains" in the above Official Comment, where you can find the detailed results. What we aim to show is: GRPODropout is significantly better than the original GRPO with less data, surpasses existing baselines in entropy and is on par with them in accuracy; and it mitigates entropy collapse at the rollout level rather than the token or algorithm level, a new perspective on data selection for reinforcement learning. Since there have recently been no good small reasoning models other than Qwen, we test generalizability at the algorithm level, extending the method to PPO.
Thank you for pointing out this work; we will cite and discuss PODS [1] in the related work. Both first generate a group of rollouts and then select a subset for the update; they differ mainly in two points.
First, the purpose of deletion differs. PODS targets efficiency: it exploits the asymmetry that generation parallelizes easily while updates are expensive, generating more rollouts and updating on only a subset, to reach the same accuracy faster. GRPODropout targets entropy collapse: it deletes rollouts that reduce entropy, and deletion itself is how the method works, not a way to save compute.
Second, which and how many to delete differ. PODS uses only rewards and keeps the subset with the largest reward variance, i.e., it aims to keep the most diverse data for training and tends to keep the highest-reward data. GRPODropout does the opposite: using both advantages and old-policy probabilities, it shows that high-probability positive-advantage rollouts are counterproductive and accelerate entropy collapse, so it deletes only such rollouts, stopping once the within-group covariance is non-negative.
Therefore, although both work at the rollout level, the target problems, methods, and conclusions are entirely different. The two may even be complementary, and combining them may yield better results. Thank you for the suggestion; we will discuss this fully in a later version.
References
Thank you for the suggestion. We will split the problem formulation into three explicit parts: notation for the prompt, the full sequence space, and policy entropy; the sampled group and each rollout's sg and A(g); and the objective. We will move the subscript conventions to a notation table in the appendix, and add an overview at the start of Section 3 explaining that Propositions 1–2 correspond to deleting positive-advantage rollouts by sg until Cov(A,s)≥0, Proposition 3 to deleting only positive-advantage rollouts, and Proposition 4 to probability-weighted recentering, so that theory and algorithm steps correspond one to one.