acceptodds
Under review as a conference paper at ICLR 2027

Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning

Abstract

Group Relative Policy Optimization (GRPO) assigns a fixed number of rollouts to each query. With binary rewards, all-correct and all-incorrect groups have zero group-relative advantages, while mixed groups may contain more responses than are needed for an effective update. We introduce Adaptive Efficient Rollout Optimization (AERO), which probes queries, reallocates a shared rollout budget to difficult groups, and reduces the responses retained for training. A Bayesian posterior supplies nonzero policy scores for groups that remain homogeneous. On Qwen2.5-family models for mathematical reasoning and single-turn coding, AERO reduces estimated generation-plus-update compute by about 48% and per-step wall-clock time by about 45%, under the same nominal rollout cap, while maintaining comparable Pass@8 and Avg@8. We also test adaptive rollout allocation alone in multi-turn coding, without posterior scores or rejection sampling. With Qwen3.6-27B, the schedule at reaches a 60.8% observed peak pass@1 on a 51-problem SWE-bench Verified split, versus 51.0% for the fixed- GRPO reference, with 2.80 recorded rollout steps per hour. Different update-group targets make this component-level evidence, not a compute-matched evaluation of full AERO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
gentic reinforcement learning, adaptive rollout allocation, event-driven scheduling, multi-turn interaction, training efficiency, GRPO.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KEzfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AERO, an approach for reducing the computational cost of RL by adaptively allocating rollouts and reducing the number of responses used for policy updates. Instead of assigning a fixed number of rollouts to every query, AERO first probes each query and selectively allocates additional rollouts to difficult all-fail groups. It further downsamples responses in mixed groups to reduce update cost, and introduces Bayesian posterior-centered scores for homogeneous groups that otherwise have zero group-relative advantage. The method is evaluated on mathematical reasoning and coding tasks, where it achieves comparable or moderately improved performance relative to GRPO/DAPO while substantially reducing reported training time and estimated FLOPs. An additional SWE-bench experiment evaluates the adaptive allocation component on a larger multi-turn coding setting.

Critical strengths

The paper targets a real source of inefficiency in RLVR: allocating the same rollout budget to every query and retaining all generated responses for the update can waste substantial generation and training compute. Separating rollout generation from response retention is a useful practical perspective, and the reported efficiency gains are substantial while maintaining broadly comparable task performance.

I also appreciate the reduced-budget GRPO comparisons. They provide useful evidence that uniformly reducing rollout count does not recover the same performance-efficiency trade-off, suggesting that selective allocation has practical value beyond simply using less compute.

Critical weaknesses

My main concern is that the efficiency comparison is not fully compute-matched. AERO changes not only the number and allocation of generated responses, but also the number of responses and tokens retained for policy updates and the composition of queries contributing to training. The nominal rollout cap therefore does not control generated tokens, retained tokens, or realized compute. The reduced-budget GRPO controls are helpful, but still do not simultaneously match these quantities. This makes the practical speedup convincing, but makes it harder to attribute the performance-efficiency improvement specifically to adaptive allocation rather than to the overall redistribution of training compute.

A second concern is that the three algorithmic components are not cleanly isolated, especially the Bayesian treatment of homogeneous groups. Assigning a nonzero posterior-centered score avoids a zero per-sample advantage, but a nonzero score alone does not establish that the resulting expected policy-gradient direction is useful. The paper reports larger gradient norms and modest empirical gains, but this does not establish improved gradient alignment, variance, or stability. More generally, posterior scoring and reward-conditioned response retention change the estimator, while the current ablations do not fully isolate their effects.

A further concern is that the algorithmic novelty of AERO is not fully clear. The paper combines several intuitive efficiency heuristics: allocating more rollouts to uncertain queries, filtering/reducing redundant responses before updates, and using a smoothed estimate for homogeneous groups. While the combination is practically useful, the paper does not clearly establish which part constitutes the central methodological contribution beyond existing adaptive sampling, data selection, or compute allocation strategies.

Questions for authors
  1. Could the authors provide a comparison against GRPO under a more tightly matched compute budget, ideally matching generated tokens and update/retained tokens or total realized FLOPs rather than only the nominal response-count allowance? This would materially affect my assessment of whether AERO improves the allocation of a fixed compute budget or mainly benefits from changing the effective training workload.
  2. Could the authors better isolate adaptive allocation, response downsampling, and posterior-centered scoring, and provide additional evidence that the Bayesian scores for homogeneous groups produce a useful learning direction rather than merely increasing gradient magnitude? A gradient alignment/variance analysis or a cleaner standalone ablation would be particularly helpful.
Reviewer_u4AYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AERO, a training-free modification of the GRPO rollout loop for RLVR with binary rewards. It has three parts: an 8-response probe per query, with all-incorrect groups receiving 2 extra rollouts per round from a shared budget until a success; a Beta(1,1)\mathrm{Beta}(1,1) posterior that supplies nonzero scores for homogeneous groups; and retention of all correct responses plus a 1:k1:k subset of incorrect ones for low-success mixed groups.

Critical strengths
  • Wasted rollouts on homogeneous groups are a real cost in RLVR, and over 35% of all-incorrect groups is a reasonable motivation.
  • Treating how many responses to generate and how many to keep as two separate decisions is a sensible way to frame the problem.
  • The method is simple and sits on top of GRPO without extra training or a learned selector, so it is easy to adopt.
Critical weaknesses
  • Lack of an adaptive-allocation baseline. The related work discusses AdaRFT, Reinforce-Ada, AR3PO, NGRPO and Prefix Sampling, all of which target the same zero-advantage problem, yet every table compares only against GRPO and DAPO. Without any of these, the paper shows that AERO beats a fixed-N baseline, which was not in question. This is my main concern
  • The retention component duplicates published work. Training on a selected subset of generated rollouts to cut update cost is PODS [1], whose max-variance down-sampling reduces, under binary rewards, to retaining equal numbers of correct and incorrect responses, i.e. AERO's k=1k=1. PODS is neither cited nor compared, and Theorem C.1 re-derives its motivation.
  • The accuracy claims rest on single runs, and the reported pp-values are over problems rather than seeds. On the 1.5B main result, AERO against GRPO is p=0.28p=0.28.
  • The efficiency claim is not attributable to the method. The N=8N=8 control uses 1863 PFLOPs against AERO's 1811, so the real comparison is a 0.024 Avg@8 gap at matched compute from one run per arm. The paper states that no random-retention or allocation-shuffled control was run, so curriculum effects are not excluded.
  • The multi-turn result does not support the comparison it is placed in: unequal group targets, unequal run lengths, and peak pass@1 over a noisy curve on 51 problems, where 31/51 against 26/51 is five problems. A maximum over repeated evaluations is biased upward by the number of evaluations, which differs across runs. This number appears in the abstract.

Minor issues:

  • Figure 8 and Appendix D.3.2 label the method PGS. Why do you use different names here?
  • Appendix B.1 is an empty section heading.
  • Appendix C.4 derives the posterior score bound under g=1g=1, but the configuration matching the reported 1.5B row uses g=0g=0.
  • Appendix E.1 reports that the ordered-selection flag sorts on padded tensor width rather than response length, so the described selection rule is not the one that ran
Questions for authors
  • How does AERO's retention rule differ from max-variance down-sampling in PODS [1], and how do the two compare empirically?
  • What are the results with a random-retention control and an allocation-shuffled control at matched compute?
  • Can you report at least three seeds for the 1.5B math setting?
  • For Table 1E, how many evaluations does each run have, and what do the final-checkpoint scores look like rather than the peaks?
  • Which configuration actually produced the 1.5B AERO row, given the ordering-flag and GPU-count discrepancies in Appendix E.1?

References [1] Xu, Yixuan Even, et al. "Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning." Transactions on Machine Learning Research, 2026

Reviewer_x3bVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Adaptive Efficient Rollout Optimization (AERO), a modification of GRPO algorithm that targets training computation reduction in reasoning and coding tasks. The proposed approach proposes adaptive rollout allocation for queries based on their relative difficulty. They also use Bayesian posterior to provide nonzero policy scores for homogeneous reward groups, and use selective response retention to reduce policy-update costs. The paper provides experiments on mathematical reasoning and code generation and reports approximately 48% reduction in generation-plus-update compute and 45% lower per-step wall-clock time, while achieving accuracy comparable to GRPO and DAPO.

Critical strengths
  1. The paper addresses an important problem in GRPO training, where fixed rollout counts can lead to unnecessary computation on homogeneous reward groups.

  2. The experimental results show promising computational savings while maintaining comparable accuracy across multiple model sizes and reasoning tasks.

Critical weaknesses
  1. Limited methodological novelty. Firstly, adaptive rollout allocation and recovery of informative samples have already been explored in prior approaches like AR3PO [1], which raised questions on the novelty. AERO combines similar mechanisms with selective response retention, but its technical novelty beyond these methods is limited. Moreover, the paper lacks direct comparisons with these methods which makes it difficult to assess the benefits of the proposed approach.

  2. Insufficient experimental validation. The ablations in do not properly evaluate all three proposed mechanisms under consistent settings. The lack of multiple independent seeds and comparisons with better baselines under matched computational budgets further limits the empirical results. Additionally, the paper is very hard to read, especially the introduction and the experimental section, which is missing explanations beyond the results. The baselines comparisons are also weak.

[1] Zhang et al. (2025). Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse.

Questions for authors
  1. Can the authors provide comparison with baselines like AR3PO under matched training and evaluation settings? This would help clarify the additional benefits of AERO over existing adaptive rollout methods.

  2. Can the authors clarify the experimental configurations used for the main efficiency results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.