Beyond Prompt Selection: Learning to Allocate Rollouts for Efficient GRPO Training

GRPO, a widely used RL method for reasoning models, allocates the same number of rollouts to every prompt, despite differences in the value of additional rollouts. This is particularly wasteful for degenerated prompts, for which sampled responses receive identical rewards and therefore provide no policy gradient. We introduce AdaRoll, an adaptive rollout allocation framework that dynamically concentrates computation on prompts where additional rollouts are most likely to be useful. In the first training epoch, AdaRoll performs pilot rollout initialization, using one rollout per prompt and its entropy and reward to predict prompt degeneracy and the distribution of future correct and incorrect rollouts. It then progressively allocates additional rollouts over multiple rounds according to a cost-benefit tradeoff between their predicted learning value and generation cost. In subsequent epochs, the predictor continuously adapts to the evolving policy using historical and newly generated rollout information, without requiring additional pilot rollouts. Across model scales and mathematical reasoning tasks, AdaRoll's predictions are 31% more accurate than the baselines, saving up to 47% of computation and improving the LLM's final performance by up to 2.5%.