- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- This paper proposes ADAROLL, which uses entropy and rewards from current and historical responses to predict the probability of forming non-degenerate groups, then dynamically selects prompts and progressively allocates rollouts.
- Experiments across two model sizes and six mathematical benchmarks show that ADAROLL reduces the rollouts needed to obtain useful training groups while maintaining or improving model performance.
Critical strengths
- The paper identifies a concrete source of inefficiency in GRPO—rollout groups with identical rewards—and proposes an adaptive sampling strategy that predicts non-degeneracy to reduce wasted computation.
- Across two model sizes, ADAROLL reduces the number of rollouts needed to obtain non-degenerate groups while maintaining or improving average accuracy across six mathematical benchmarks.
Critical weaknesses
- The evaluation covers only two sizes of Qwen2.5-Math. Since ADAROLL relies on the relationship between rollout entropy, rewards, and non-degeneracy, it remains unclear whether its gains generalize to other model families with different rollout characteristics.
- The efficiency evaluation measures rollout counts rather than token usage or end-to-end training time. Since response lengths vary and progressive sampling may introduce additional scheduling overhead, the reported rollout reductions do not establish equivalent computational savings.
Questions for authors
-
Could you provide results on a model family other than Qwen to assess whether ADAROLL’s efficiency gains generalize beyond Qwen2.5-Math?
-
Could you report AIME24 average@32 results for ADAROLL and the baselines to assess the robustness of the performance differences?
2 comments
Dear Reviewer,
Thank you again for time and effort you have invested in reviewing our work. We have posted a detailed response addressing your points, including the additional experiments, the wall-clock evidence, speed up in terms of token usage, and statistical robustness.
Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.
Best regards,
The Authors
We thank the reviewer for their feedback and recognizing the efficiency and performance of our proposed method. We answer the questions below.
First we provide a brief overview of our method: AdaRoll is an adaptive rollout allocation framework for LLM RL post-training, which works by efficiently and accurately predicting how useful a prompt is for RL training (based on its non-degenerate probability), and dynamically allocates rollouts toward the useful ones. In section 3.1, we present the empirical finding that the prompt non-degeneracy can be predicted (using an MLP predictor) using the entropy and correctness of an individual rollout, motivating progressive rollout allocation to prompts with higher non-degenerate probability. In section 3.2, we show that prompts with the highest predicted benefit-cost tradeoff from an additional rollout should be prioritized in order to ensure efficiency. In section 3.3, we generate more rollouts for prompts with clear benefit-cost tradeoff, while generating less rollout and collect more evidence for those with uncertain benefits.
For the experiments, we made some changes to the manuscript (marked in blue). Specifically, we show with controlled experiments that AdaRoll reaches the same accuracy significantly faster than baselines, and that under the same training budget, AdaRoll achieves higher test accuracy, demonstrating the efficiency. We further show that AdaRoll finds non-degenerate groups accurately and efficiently, explaining the efficient training.
Regarding the reviewers' questions:
Generalization to other model families, and end-to-end training time: We update Figure 3 of the manuscript to report end-to-end training time and present the experiments on different model families, including Qwen2.5-Math, Qwen3, Deepseek-Distilled-Qwen and Phi4. Adaroll remains efficient across all models with a speedup up to 2.78x and an average of 1.87 speedup compared to GRESO and 1.44 speed up compared to vigor.
Speed up in terms of token usage: For the same model, different training methods produce a similar token distribution for individual questions, all within 8% difference of AdaRoll, and the total number of tokens scale with the number of rollouts generated in each iteration. However, we do notice a non-linear relationship between our speed-up and the average response token length. Specifically, Deepseek-Distilled-Qwen1.5b generates longer response length compared to the other models and we noticed the largest speed up. We further quantify this observation in Figure 8, showing that the speedup per iteration scales quadratically as the response length increases. For instance, when training on Deepseek-Distilled-Qwen-1.5b, when the average response length is 500, AdaRoll have a average speed up of 1.28x than VIGOR and 1.89x than GRESO in terms of the wall time per training iteration, and this speedup grows to 1.66x and 2.73x when the average response length is 5000.
Robustness of the performance difference: We reevaluate the AIME24 score we reported in the original Table 1 where we train on Qwen2.5-Math-1.5B with batch size 256 and group size 8 for 250 iterations here.
Indeed, since AIME24 contains only 30 questions, the results can fluctuate a bit. However, when averaged over 6 benchmarks, such fluctuation leads to at most 0.31 percentage points in the final accuracy. We’re also rerunning our experiments with different random seeds and will update the figures and tables with error bands and report the std. So far, across 3 random seeds on Qwen2.5-Math-1.5B, the run-to-run standard deviation averages 0.3 percentage points across checkpoints for each evaluation, further demonstrating the robustness of the evaluation.
We hope these additional results would clarify your questions and demonstrate the efficiency of AdaRoll. If this resolves your concerns, we’d appreciate it if you could consider increasing the assessment of our work; and if you have any questions, or have particular parts of the paper that you find hard to follow, we’re happy to further clarify them.