acceptodds
Under review as a conference paper at ICLR 2027

Beyond Prompt Selection: Learning to Allocate Rollouts for Efficient GRPO Training

Abstract

GRPO, a widely used RL method for reasoning models, allocates the same number of rollouts to every prompt, despite differences in the value of additional rollouts. This is particularly wasteful for degenerated prompts, for which sampled responses receive identical rewards and therefore provide no policy gradient. We introduce AdaRoll, an adaptive rollout allocation framework that dynamically concentrates computation on prompts where additional rollouts are most likely to be useful. In the first training epoch, AdaRoll performs pilot rollout initialization, using one rollout per prompt and its entropy and reward to predict prompt degeneracy and the distribution of future correct and incorrect rollouts. It then progressively allocates additional rollouts over multiple rounds according to a cost-benefit tradeoff between their predicted learning value and generation cost. In subsequent epochs, the predictor continuously adapts to the evolving policy using historical and newly generated rollout information, without requiring additional pilot rollouts. Across model scales and mathematical reasoning tasks, AdaRoll's predictions are 31% more accurate than the baselines, saving up to 47% of computation and improving the LLM's final performance by up to 2.5%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, GRPO, RL efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jmsCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  1. This paper proposes ADAROLL, which uses entropy and rewards from current and historical responses to predict the probability of forming non-degenerate groups, then dynamically selects prompts and progressively allocates rollouts.
  2. Experiments across two model sizes and six mathematical benchmarks show that ADAROLL reduces the rollouts needed to obtain useful training groups while maintaining or improving model performance.
Critical strengths
  1. The paper identifies a concrete source of inefficiency in GRPO—rollout groups with identical rewards—and proposes an adaptive sampling strategy that predicts non-degeneracy to reduce wasted computation.
  2. Across two model sizes, ADAROLL reduces the number of rollouts needed to obtain non-degenerate groups while maintaining or improving average accuracy across six mathematical benchmarks.
Critical weaknesses
  1. The evaluation covers only two sizes of Qwen2.5-Math. Since ADAROLL relies on the relationship between rollout entropy, rewards, and non-degeneracy, it remains unclear whether its gains generalize to other model families with different rollout characteristics.
  2. The efficiency evaluation measures rollout counts rather than token usage or end-to-end training time. Since response lengths vary and progressive sampling may introduce additional scheduling overhead, the reported rollout reductions do not establish equivalent computational savings.
Questions for authors
  1. Could you provide results on a model family other than Qwen to assess whether ADAROLL’s efficiency gains generalize beyond Qwen2.5-Math?

  2. Could you report AIME24 average@32 results for ADAROLL and the baselines to assess the robustness of the performance differences?

2 comments

We thank the reviewer for their feedback and recognizing the efficiency and performance of our proposed method. We answer the questions below.

First we provide a brief overview of our method: AdaRoll is an adaptive rollout allocation framework for LLM RL post-training, which works by efficiently and accurately predicting how useful a prompt is for RL training (based on its non-degenerate probability), and dynamically allocates rollouts toward the useful ones. In section 3.1, we present the empirical finding that the prompt non-degeneracy can be predicted (using an MLP predictor) using the entropy and correctness of an individual rollout, motivating progressive rollout allocation to prompts with higher non-degenerate probability. In section 3.2, we show that prompts with the highest predicted benefit-cost tradeoff from an additional rollout should be prioritized in order to ensure efficiency. In section 3.3, we generate more rollouts for prompts with clear benefit-cost tradeoff, while generating less rollout and collect more evidence for those with uncertain benefits.

For the experiments, we made some changes to the manuscript (marked in blue). Specifically, we show with controlled experiments that AdaRoll reaches the same accuracy significantly faster than baselines, and that under the same training budget, AdaRoll achieves higher test accuracy, demonstrating the efficiency. We further show that AdaRoll finds non-degenerate groups accurately and efficiently, explaining the efficient training.

Regarding the reviewers' questions:

  1. Generalization to other model families, and end-to-end training time: We update Figure 3 of the manuscript to report end-to-end training time and present the experiments on different model families, including Qwen2.5-Math, Qwen3, Deepseek-Distilled-Qwen and Phi4. Adaroll remains efficient across all models with a speedup up to 2.78x and an average of 1.87 speedup compared to GRESO and 1.44 speed up compared to vigor.

  2. Speed up in terms of token usage: For the same model, different training methods produce a similar token distribution for individual questions, all within 8% difference of AdaRoll, and the total number of tokens scale with the number of rollouts generated in each iteration. However, we do notice a non-linear relationship between our speed-up and the average response token length. Specifically, Deepseek-Distilled-Qwen1.5b generates longer response length compared to the other models and we noticed the largest speed up. We further quantify this observation in Figure 8, showing that the speedup per iteration scales quadratically as the response length increases. For instance, when training on Deepseek-Distilled-Qwen-1.5b, when the average response length is 500, AdaRoll have a average speed up of 1.28x than VIGOR and 1.89x than GRESO in terms of the wall time per training iteration, and this speedup grows to 1.66x and 2.73x when the average response length is 5000.

  3. Robustness of the performance difference: We reevaluate the AIME24 score we reported in the original Table 1 where we train on Qwen2.5-Math-1.5B with batch size 256 and group size 8 for 250 iterations here.

    MethodAIME24 average@32 (%)Difference vs. Measured average@4
    VIGOR15.73-1.77
    GRESO15.83+0.03
    Dynamic Sampling13.96-1.87
    AdaRoll13.44-1.56

    Indeed, since AIME24 contains only 30 questions, the results can fluctuate a bit. However, when averaged over 6 benchmarks, such fluctuation leads to at most 0.31 percentage points in the final accuracy. We’re also rerunning our experiments with different random seeds and will update the figures and tables with error bands and report the std. So far, across 3 random seeds on Qwen2.5-Math-1.5B, the run-to-run standard deviation averages 0.3 percentage points across checkpoints for each evaluation, further demonstrating the robustness of the evaluation.

We hope these additional results would clarify your questions and demonstrate the efficiency of AdaRoll. If this resolves your concerns, we’d appreciate it if you could consider increasing the assessment of our work; and if you have any questions, or have particular parts of the paper that you find hard to follow, we’re happy to further clarify them.

Dear Reviewer,

Thank you again for time and effort you have invested in reviewing our work. We have posted a detailed response addressing your points, including the additional experiments, the wall-clock evidence, speed up in terms of token usage, and statistical robustness.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.

Best regards,

The Authors

Reviewer_UnfiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that GRPO wastes rollouts on prompts that generate all wrong/right answers, resulting in no GRPO advantage. The proposed method generates one rollout per prompt and gives its reward and entropy to an MLP to predict how probable it is that a prompt will have mixed rewards. Then it ranks prompts by the expected number of rollouts it needs to get mixed rewards, and rolls out from the ones that require the least expected rollouts.

Critical strengths
  • The motivation is strong
  • It flows smoothly from the pivotal observation in figure 1's left column to the proposed method.
Critical weaknesses
  • The total number of rollouts falls by only 15% and 5% for two model sizes in figure 3. With such small gains, the overhead for adaroll should be included: the initial rollouts and the entropy calculation.
  • Dynamic sampling 7B uses more rollouts than GRESO in fig 4, but uses fewer rollouts than GRESO in fig 3. This implies an inconsistent setup in comparisons.
  • It's not clear why adaroll is better than vigor.
Questions for authors
  • Can you report the number of non-degenerate groups per update for each method?
  • Since adaroll has overhead and the rollout reduction percentage is small, could you report a fair training time?
  • Why is adaroll better than vigor? Is it total rollouts? Training time?
3 comments
AuthorsOpenReview ↗

Clarifications to Reviewer Unfi (1/2)

We thank the reviewer for the positive assessment of our work and acknowledging our motivation and the flow of the paper. Regarding the experiments, we made some changes to the experiment section of our manuscript (marked in blue). Specifically, we show with controlled experiments that AdaRoll reaches the same accuracy significantly faster (in wall-clock time) than baselines, and that under the same training time budget, AdaRoll achieves higher test accuracy, demonstrating the efficiency. We’d like to further clarify the concerns raised by the reviewer.

  1. Overhead for AdaRoll, and wall-clock training time.

    • The cost of initial rollouts was already included in our original experimentation results in Fig 3. Originally, we had our efficiency comparison based on the rollout number which had diluted our efficiency gains. However, the main strength of AdaRoll is its speedup (end to end wall-clock time) over baselines. We made changes to the experiment section of our manuscript (marked in blue) to reflect this advantage across different model families, batch sizes, and group sizes. Specifically, when we consider the wall time required to reach the same accuracy (see our updated Fig 3), AdaRoll outperforms the baselines significantly and consistently across different models, with an average speed up of 1.87x over GRESO and 1.44x over VIGOR, across model families. On the other hand, if given the same amount of time (see our updated Table 1), our method ends up outperforming GRESO and VIGOR by an average of 3.3 and 1.1 percentage points, respectively, across different model families.
    • We’d like to clarify first that the rollout reduction percentage is not small. Specifically, as shown in Figure 6, in order to get the same number of non-degenerated groups, we save 26% of rollout generation compared to GRESO. Also, it can be clearly seen in our new Figure 6 (previously Figure 4) where AdaRoll starts with a slightly higher cost at the beginning of training to initialize the predictor for all prompts and has lower cost as training goes on, and a significantly lower cost in total. This figure also shows that even at the beginning of training where we have overhead, the time efficiency and accuracy of identifying non-degenerated groups still significantly and consistently outperform baselines.
    • Calculating the entropy does lead to additional computation, yet this overhead is small. On average, entropy calculation costs an extra overhead of only 2%-3% of the vllm generation time, which corresponds to about 1% of the total training time. This is enabled by reusing the logits vllm already computes and compiling the entropy kernel via torch.compile.
  2. Non-degenerate group per update

    While the total number of non-degenerate groups somewhat reflects the total training time, it doesn’t accurately reflect the time efficiency of the methods. For instance, when training with a batch size of 128, GRESO always generates more rollouts until it collects 128 rollouts per iteration. Therefore, to better compare efficiency, the desired metric should be the average number of non-degenerate groups collected per a fixed, e.g. 1024 rollouts:

    ModelAdaRollVIGORGRESO
    Qwen2.5-Math-1.5B110.028.680.5
    DeepSeek-R1-Distill-Qwen-1.5B101.827.980.6
    Qwen3-4B-Base103.925.982.2
    Phi-4-mini-instruct107.425.881.1

    Note that VIGOR’s count is significantly smaller than AdaRoll and GRESO. This is because while both AdaRoll and GRESO aim to find the most non-degenerate groups, VIGOR aims for the different target of generating more rollout on groups with higher variance. Nevertheless, across all models, AdaRoll is the best at identifying non-degenerated groups, finding 30% more non-degenerated groups than GRESO on average as indicated in the above figure and in Fig. 6.

  3. The set up for comparison

    The key training configurations are identical across both experiments. They differ only in what is being compared, which requires controlling for different factors, which may have caused confusion. In the original Figure 3, we controlled for the cost and compare the accuracy, so every run ends with the similar computation cost; in our revised Figure 3, which has been replotted to further clarify the wall-clock time efficiency of AdaRoll, we compare the exact speed up so we train VIGOR and AdaRoll until they reach the baseline accuracy of GRESO; while in Figure 4 (now Figure 6), we compare the accuracy and rollout cost, and show how it changes across iterations, so all the curves end with the same number of iterations. We have reorganized the figures in different subsections and highlighted the different purposes they serve, and hopefully this clarifies the confusion.

AuthorsOpenReview ↗

Clarifications to Reviewer Unfi (2/2)

  1. The advantage of AdaRoll over VIGOR. We want to highlight that the benefits of AdaRoll lies in wall-clock time efficiency. Specifically, VIGOR starts with two rollouts per prompt and adaptively doubles the number of rollouts only for prompts with the highest reward variance, leading to larger overhead due to generating rollouts in batches. In contrast, AdaRoll directly determines the number of rollouts to generate for each prompt, without restricting the allocation to powers of two. This finer-grained allocation enables more efficient rollout generation and more accurate identification of informative prompts.

    Here we present the mean number of generation rounds (or vLLM inference calls) per RL iteration.

    ModelAdaRollVIGOR
    Qwen2.5-Math-1.5B2.624
    DeepSeek-R1-Distill-Qwen-1.5B3.094
    Qwen3-4B-Base2.894
    Phi-4-mini-instruct2.804

    The larger overhead results in longer time for rollout generation. For example, for Qwen2.5-Math-1.5B, on average, VIGOR spends 10.2 more seconds generating the rollout per iteration.

We hope these additional results would clarify your questions and demonstrate the efficiency of AdaRoll. If this resolves your concerns, we’d appreciate it if you could consider increasing the assessment of our work; and if you have any questions, we’re happy to further clarify them.

Dear Reviewer,

Thank you again for your positive feedback on our work. We have posted a detailed response addressing your points, including the analysis of the overhead, the wall-clock evidence, requested numbers, clarification of the setup, and advantage over VIGOR.

Here, we provide further details for the wall-clock time comparison of the selection overhead for the 3 methods. With DeepSeek-R1-Distill-Qwen-1.5B trained on 2$\times$ H200 GPUs, AdaRoll’s rollout selection and generation takes 35.2 s (44.6%) less than GRESO and 16.3 s (27.1%) less than VIGOR per iteration, demonstrating AdaRoll’s efficiency. We have added these results as Fig. 5 to the paper.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time.

Best regards,

The Authors

Reviewer_sTnvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRPO spends a uniform budget of G rollouts per prompt, even though many prompts are degenerate (all rollouts get identical rewards, hence zero advantage and no gradient signal). The paper recasts rollout allocation as a sequential cost-benefit problem: a lightweight MLP predicts a Beta posterior over each prompt's non-degeneracy from reward and token-entropy features, prompts are selected by marginal cost, and a lookahead rule decides how many extra rollouts to commit per round. On Qwen2.5-Math-1.5B/7B the method reportedly saves up to 47% of rollouts relative to dynamic sampling at comparable final accuracy.

Critical strengths

The reformulation is the main contribution and it is a good one: jointly deciding which prompts to keep and how many rollouts to allocate is strictly more general than prior prompt-selection work, and the Beta-Binomial treatment keeps the method tractable and cheap (the predictor retrains on CPU in under 100ms). The entropy finding — that a single rollout's token-level entropy predicts non-degeneracy, and that this relation drifts over training — is interesting in its own right and backed by a real ablation (AUC drops from 0.70 to 0.60 without it).

Critical weaknesses

The headline numbers are framed against the weakest baseline. VIGOR reports no efficiency figures, and on accuracy it matches or slightly beats the proposed method (44.58 vs 44.53 at 1.5B); against GRESO the accuracy gain is only 0.23–0.85%, which the "up to 2.5%" phrasing in the abstract obscures. So the real selling point has to be efficiency — yet the efficiency comparison against the strongest accuracy baseline is missing.

There is also no wall-clock evidence at all. Rollout count is only a proxy: the progressive small-batch generation schedule this method relies on tends to underutilize GPUs in vLLM, so it is unclear how much of the 28–47% rollout saving survives as actual training time. Statistical rigor is weak too — a single training run, 4 samples per prompt at evaluation, tiny test sets like AIME (30 problems), no error bars or significance testing — while several key margins (e.g. 0.23% at 1.5B) are well within noise. The evaluation is narrow: one model family, math only, G=8, 250 steps.

Questions for authors
  1. Can you report end-to-end wall-clock or GPU-hour savings, including predictor refits and the scheduling overhead of progressive small-batch generation?
  2. Why is VIGOR absent from the efficiency comparison? On an accuracy-vs-rollouts curve, does the method remain Pareto-optimal once VIGOR is included?
  3. Are the final accuracy differences over GRESO/VIGOR significant under multiple training seeds (≥3)?
3 comments
AuthorsOpenReview ↗

Clarifications to Reviewer sTnv (1/2)

We thank the reviewer for the review and acknowledging our general formulation of the problem and our interesting finding about entropy.

Regarding the experiments, we agree that the main strength of AdaRoll is its efficiency. Originally, we had our efficiency comparison based on the rollout number which had diluted our efficiency gains. However, the main strength of AdaRoll is its speedup (end to end wall-clock time) over baselines. We made changes to the experiment section of our manuscript (marked in blue) to reflect this advantage across different model families, batch sizes, and group sizes. Specifically, we show with controlled experiments that AdaRoll reaches the same accuracy significantly faster than baselines, and that under the same training budget, AdaRoll achieves higher test accuracy, demonstrating the efficiency. We wish to address the reviewer’s concern as follows:

  1. Narrow evaluation:
    We added substantial variation to our evaluation. This includes:

    • Model family. We included in Figure 3 and Table 1 more experiments from different model families with varying steps and different batch sizes, including Qwen2.5-Math, Qwen3, Deepseek-distilled-Qwen, and Phi4. Figure 3 shows that AdaRoll consistently and significantly speeds up model training in terms of wall time, with an average speed up of 1.87x over GRESO and 1.44x over VIGOR.
    • Batch size: We included new experiments with batch size 128 instead of 256.
    • Training step: In our new Figure 3 and 4, we train with VIGOR and AdaRoll until the performance exceeds the GRESO baseline to show the speed up. Fig 3 shows the performance of AdaRoll and baselines up to this point and demonstrates that AdaRoll consistently achieves substantial speedups across all models, up to 2.78x over GRESO and 1.66x over VIGOR (on average 1.87× over GRESO and 1.44× over VIGOR across different model families). We also provide longer training results in Fig. 8, and AdaRoll converges faster than the baselines.
    • Group size: We alter the group size to 16 in Figure 4 (for training Qwen 2.5 Math-1.5B), and AdaRoll obtains a speed up of 2.69x over GRESO and 1.76x over VIGOR, consistent with the above conclusion that AdaRoll speeds up training.
    • Response length cutoff. We’re still running the full training experiments on varying response length and will add it to the final version. While we don’t have the exact speed up, we do expect a more significant speed up as we train on longer responses. This is because the rollout cost scales superlinearly with response length, and we show empirically in Figure 8 that when allowing the model to generate longer responses, the efficiency of AdaRoll becomes more pronounced compared to other baselines.

    We hope these results strengthen AdaRoll’s efficiency.

AuthorsOpenReview ↗

Clarifications to Reviewer sTnv (2/2)

  1. Wall-clock Evidence
    We provide the wall-clock time results for all these experiments. In general, the overhead of AdaRoll is small.

    • Refitting the predictor: As indicated in Appendix B.2 (line 863), the cost for refitting the predictor for each RL iteration is only less than 0.1s, which is small compared to the full time of 40~140 seconds per RL iteration that we see in our experiments.

    • Small-batch generation: Although the rollouts are generated in different rounds, each batch is not necessarily small, thanks to the outcome lookahead mechanism described in section 3.3. At a high level, if the predictor is confident about its own prediction of a specific problem, it can send the entire group for generations at once, without the need for generating one rollout after another, thereby reducing the overhead. In practice, among the selected non-degenerated prompts, over 74% can be confidently predicted from history rollout alone. And within each generation batch, each prompt generates 3.2 to 4.6 questions on average, making the total number of rollout generations large in each batch large, and the number of generation batches per RL iteration small.

      Here we present the mean number of generation batches per RL iteration.

      ModelNumber of generation batches per RL iteration
      Qwen2.5-Math-1.5B2.62
      DeepSeek-R1-Distill-Qwen-1.5B3.09
      Qwen3-4B-Base2.89
      Phi-4-mini-instruct2.80

      Compared to VIGOR, which generates rollouts in 4 batches for each RL training iteration to pick the ones with the highest empirical variance, we actually have less overhead in batched generation. To summarize, the small overhead of batch generation is well compensated by the more accurate identification of nondegenerate groups. This allows AdaRoll to beat both GRESO and Vigor in terms of efficiency and as a result accuracy.

    In conclusion, we designed AdaRoll to aim for efficiency and empirical evaluation confirms it.

  2. Regarding the significance of the final accuracy difference

    While VIGOR and AdaRoll perform similarly in performance measured by the number of rollouts, we want to highlight that the benefits of AdaRoll lies in wall-clock time efficiency. Specifically, VIGOR starts with two rollouts per prompt and adaptively doubles the number of rollouts only for prompts with the highest reward variance, resulting in smaller rollout batches overall. In contrast, AdaRoll directly determines the number of rollouts to generate for each prompt, without restricting the allocation to powers of two. This finer-grained allocation enables more efficient rollout generation and more accurate identification of informative prompts. For example, for Qwen2.5-Math-1.5B, on average, VIGOR spends 10.2 more seconds generating the rollout. Therefore, when we consider the wall time required to reach the same accuracy (see our updated Fig 1), or the accuracy each method obtains when training for the same amount of time (see our updated Table 1), AdaRoll outperforms the baselines significantly and consistently across different models, with an average speed up of 1.87x over GRESO and 1.44x over VIGOR, across model families. On the other hand, if given the same amount of time, our method ends up outperforming GRESO and VIGOR by an average of 3.3 and 1.1 percentage points, respectively, across different model families.

  3. Statistical rigour across seeds
    We’re currently rerunning our experiments with different random seeds and will update the figures and tables with error bands and report the std.So far, the statistical error is small: Across 3 random seeds on Qwen2.5-Math-1.5B, the run-to-run standard deviation averages 0.3 percentage points across checkpoints for each evaluation.

    In terms of the final performance at convergence, indeed, in our original evaluations, the average final converging accuracy of AdaRoll is very similar to VIGOR, within a difference of 0.05 percentage points for the original 1.5b and 7b experiments, and to GRESO on the original 1.5b model with a difference of 0.23 percentage points. That said, our method outperforms these baselines by being more efficient in terms of wall clock time.

We hope these additional results would clarify your questions and demonstrate the efficiency of AdaRoll. If this resolves your concerns, we’d appreciate it if you could consider increasing the assessment of our work; and if you have any questions, we’re happy to further clarify them.

Dear Reviewer,

Thank you again for time and effort you have invested in reviewing our work. We have posted a detailed response addressing your points, including the evaluation settings, the wall-clock evidence, advantage over VIGOR, and statistical rigor.

Here, we’re also reporting the wall-clock time comparison of the selection overhead for the 3 methods. With DeepSeek-R1-Distill-Qwen-1.5B trained on 2$\times$H200 GPUs, AdaRoll’s rollout selection and generation takes 35.2 s (44.6%) less than GRESO and 16.3 s (27.1%) less than VIGOR per iteration, demonstrating AdaRoll’s efficiency. We have added these results as Fig. 5 to the paper.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.