- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AERO, an approach for reducing the computational cost of RL by adaptively allocating rollouts and reducing the number of responses used for policy updates. Instead of assigning a fixed number of rollouts to every query, AERO first probes each query and selectively allocates additional rollouts to difficult all-fail groups. It further downsamples responses in mixed groups to reduce update cost, and introduces Bayesian posterior-centered scores for homogeneous groups that otherwise have zero group-relative advantage. The method is evaluated on mathematical reasoning and coding tasks, where it achieves comparable or moderately improved performance relative to GRPO/DAPO while substantially reducing reported training time and estimated FLOPs. An additional SWE-bench experiment evaluates the adaptive allocation component on a larger multi-turn coding setting.
Critical strengths
The paper targets a real source of inefficiency in RLVR: allocating the same rollout budget to every query and retaining all generated responses for the update can waste substantial generation and training compute. Separating rollout generation from response retention is a useful practical perspective, and the reported efficiency gains are substantial while maintaining broadly comparable task performance.
I also appreciate the reduced-budget GRPO comparisons. They provide useful evidence that uniformly reducing rollout count does not recover the same performance-efficiency trade-off, suggesting that selective allocation has practical value beyond simply using less compute.
Critical weaknesses
My main concern is that the efficiency comparison is not fully compute-matched. AERO changes not only the number and allocation of generated responses, but also the number of responses and tokens retained for policy updates and the composition of queries contributing to training. The nominal rollout cap therefore does not control generated tokens, retained tokens, or realized compute. The reduced-budget GRPO controls are helpful, but still do not simultaneously match these quantities. This makes the practical speedup convincing, but makes it harder to attribute the performance-efficiency improvement specifically to adaptive allocation rather than to the overall redistribution of training compute.
A second concern is that the three algorithmic components are not cleanly isolated, especially the Bayesian treatment of homogeneous groups. Assigning a nonzero posterior-centered score avoids a zero per-sample advantage, but a nonzero score alone does not establish that the resulting expected policy-gradient direction is useful. The paper reports larger gradient norms and modest empirical gains, but this does not establish improved gradient alignment, variance, or stability. More generally, posterior scoring and reward-conditioned response retention change the estimator, while the current ablations do not fully isolate their effects.
A further concern is that the algorithmic novelty of AERO is not fully clear. The paper combines several intuitive efficiency heuristics: allocating more rollouts to uncertain queries, filtering/reducing redundant responses before updates, and using a smoothed estimate for homogeneous groups. While the combination is practically useful, the paper does not clearly establish which part constitutes the central methodological contribution beyond existing adaptive sampling, data selection, or compute allocation strategies.
Questions for authors
- Could the authors provide a comparison against GRPO under a more tightly matched compute budget, ideally matching generated tokens and update/retained tokens or total realized FLOPs rather than only the nominal response-count allowance? This would materially affect my assessment of whether AERO improves the allocation of a fixed compute budget or mainly benefits from changing the effective training workload.
- Could the authors better isolate adaptive allocation, response downsampling, and posterior-centered scoring, and provide additional evidence that the Bayesian scores for homogeneous groups produce a useful learning direction rather than merely increasing gradient magnitude? A gradient alignment/variance analysis or a cleaner standalone ablation would be particularly helpful.