- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1(−1)
Read review
Paper summary
The paper introduces HAVER-GRPO which is an adaptive rollout allocation strategy. The authors observe that when a group of rollouts has no correct solution, then there is no learning signal since the advantages have no differences. To counter this, they use a simple allocation strategy which is dependent on the cost of each rollout. Then, if a successful rollout is sampled, they replace it in the group that is used for advantage computation and the policy gradient. They demonstrate improved performance over strategies like DAPO which uses oversampling and regular GRPO.
Critical strengths
- Clarity and simplicity: The paper is easy to read and understand, and the method proposed is simple to implement and poses little overhead. Figure 1 is enough to understand the core contributions of the paper.
- Strength and broadness of results: The experiments are performed over multiple models like the Qwen2.5-1.5B and Qwen2.5-7B Instruct models with evaluations across multiple datasets such as AIME, AMC, MATH500 and OlympiadBench which makes the comparisons sound and significant.
Critical weaknesses
- What is the training dataset: The authors do not name the training dataset in the paper or in the Appendix.
- Insufficient baseline comparison: The authors cite work such as VIP (Nguyen et. al), however do not compare with it directly. Since it is prior art at this time (published in ICLR 2026), it would benefit to have a direct comparison since both works aim to maximize performance within a training rollout budget. The argument of returning K responses to the optimizer in Section 2 doesn't give a clear reason why the baselines would not be relevant.
Questions for authors
I have the 3 following questions:
- Could the authors remake figure 2 with x axis being the number of tokens generated in the rollout phase? It is hard to interpret the plot and Table 2 and Figure 3 all at the same time. Making the figure with x axis being number of tokens would make the comparisons more faithful to the intended use case of the algorithm.
- Why do you only choose the 'all-wrong' group for adaptive allocation of compute and not the 'all-correct' group? This aspect of the paper is under-discussed
- Why is the advantage still decided based on the original group size if you do adaptive allocation? As an example, say it consumed 32 rollouts to obtain 1 successful rollout. Then, the advantages will still be decided as if there was 1 success out of 8. Does this change the population level objective? Does this have some relations with up-weighting harder prompts like MaxRL (Tajwar et. al) or REINFORCE-ADA (Xiong et. al). A discussion around the population objective or relations to re-weighting groups would be beneficial.