acceptodds
Under review as a conference paper at ICLR 2027

CERO: Where and When to Allocate Rollouts for RL Post-Training

Abstract

Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts. We instead study how to coordinate a finite rollout budget over the entire training horizon. We formulate this problem using a concave surrogate utility of cumulative prompt exposure and introduce CERO, an online primal–dual scheduler for prompt admission and budget pacing. In our experiments, each admitted prompt receives a fixed-size response group. CERO instead adapts which prompts are selected, how often they are revisited across rounds, and how many groups are generated in each round. A compact Fenchel representation linearizes the dependence on cumulative exposure, while projected online gradient descent updates prompt-specific supporting slopes and a shared budget price using reward-variation feedback and budget deviations. We establish pathwise guarantees for the surrogate allocation objective against fixed-rate and same-path time-varying benchmarks, with explicit terms for proxy discrepancy and rate variation. Under matched training-response budgets, CERO attains the highest avg@16 macro-average on each of three backbones across five mathematical reasoning benchmarks. Mechanistic analyses link CERO's prompt choices to within-group reward contrast, while multi-seed ablations show gains from adaptive pacing over both uniform and preset spending schedules.

Primary area:
reinforcement learning
Keywords:
RL post-training, optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9XrAOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CERO treats the rollout budget for RL post-training to allocate over the whole training run, deciding both which prompts get rollouts and how much to spend in each round. It models each prompt's value as a concave utility of its cumulative exposure, with the utility's rate estimated by the predicted within-group reward variance from a per-prompt Beta–Bernoulli state. A Fenchel representation reduces each round to ranking prompts by a margin between a prompt-specific slope and a shared budget price, both updated by projected online gradient descent. The authors prove pathwise regret bounds on this surrogate objective against fixed-rate and time-varying oracles. On three Qwen/DeepSeek backbones and five math benchmarks under a matched 256k-response budget, CERO achieves the best avg@16 macro-average, supported by a 3-seed ablation of its pacing and selection components.

Critical strengths
  1. Treating the rollout budget as a horizon-level resource, rather than a per-round one, is a natural and underexplored angle. The online primal–dual machinery from ad allocation and revenue management is a sensible import. Each round reduces to a sort, with closed-form conjugates and supporting exposures, and measured compute overhead is negligible.
  2. The paper reports detailed hyperparameters and the reproducibility looks good.
Critical weaknesses
  1. The theory appears to be vacuous in the experimental regime. The per-prompt regret bound scales with Nmax/NˉN_\text{max}/\bar{N}, where Nˉ=B/(KM)\bar{N}= B/(KM). If B=64,000B = 64,000, K=500K = 500, and MM of roughly 8-9k prompts (half of DAPO-Math-17K), Nˉ≈0.014\bar{N} \approx 0.014. So Nmax/Nˉ≈140N_\text{max}/\bar{N} \approx 140. The optimization term is then about λ×140/500≈6λ\lambda\times140/\sqrt{500}\approx 6\lambda, which exceeds the trivial bound of 1 at λ=1\lambda = 1. The proxy term, 8λ(Nmax/Nˉ)eˉ8\lambda(N_\text{max}/\bar{N})\bar{e}, is about 1100eˉ1100\bar{e}. The guarantees therefore say nothing for the settings actually run. This should be acknowledged, or the regime where the bounds bite should be characterized.
  2. The pacing ablation is confounded. CERO-fixed sets ν = 0, which also changes the admission threshold and the margins, not just the spending schedule. Its gap to CERO therefore mixes selection and pacing effects. A cleaner control would replay CERO's prompt choices under a uniform schedule.
Questions for authors
  1. The selection-lift analysis suggests CERO mostly favors prompts with predicted success near 0.5. How does CERO compare with a simple rule that selects prompts whose Beta mean is closest to 0.5, or with DAPO-style dynamic filtering of all-correct and all-incorrect groups?
  2. Why not compare against the difficulty-aware methods you cite, such as SPEED-RL, Goldilocks RL, and VADE? These target the same "informative prompt" signal.
Reviewer_X6RxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CERO jointly schedules prompt selection, repeated exposure, and per-round rollout expenditure under a finite training budget. It estimates future within-group reward variation from historical outcomes, and defines a concave surrogate utility with diminishing returns, and uses a primal–dual scheduler to coordinate prompt priorities and budget pacing. The theoretical guarantees concern the surrogate allocation objective. Experiments on three backbones and mathematical reasoning benchmarks compare CERO with GRPO and KnapsackRL, with controls targeting prompt assignment and temporal pacing.

Critical strengths
  1. The formulation captures a shared budget across rounds. Accounting for cumulative exposure and remaining resources makes the scheduling problem more complete than independent within-round allocation.
  2. The paper includes meaningful component controls. Random-matched CERO preserves the allocation pattern while changing prompt assignment, and fixed, preset, and greedy alternatives help separate selection from pacing. Computational costs are also measured rather than inferred solely from response counts.
Critical weaknesses
  1. There remains a substantial gap between the theoretical objective and final policy quality. Reward variation measures available reward contrast, not necessarily gradient usefulness or downstream generalization. Diminishing returns are also imposed through the surrogate. The dynamic guarantee holds alternative allocations against a common realized rate path; it does not establish near-optimality when a different schedule would change policy evolution.
  2. Evidence for the central pacing contribution is still localized. Improvements over KnapsackRL are approximately 3.28, 0.67, and 0.06 points across the three backbones. Multi-seed pacing ablations are encouraging but focus on the smallest backbone. The observed departures from uniform expenditure are also relatively limited, leaving robustness across model scales and budget regimes uncertain.
Questions for authors
  1. Does adaptive pacing consistently outperform CERO-fixed on the larger backbones? Can you report uncertainty across training seeds for the main baseline differences?
  2. Under the same proxy and budget constraints, how much does the complete primal–dual method improve over a simple proxy-ranking method with a preset spending schedule?
  3. Is there evidence that the proxy predicts actual learning progress beyond reward contrast? If not, could the mechanism claims more explicitly reflect this limitation?
Reviewer_PAgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CERO is a primal–dual scheduler that jointly adapts prompt selection and per-round rollout spending under a fixed training budget. It provides surrogate allocation guarantees and evaluates the approach on mathematical reasoning benchmarks.

Critical strengths
  1. The paper addresses a meaningful extension of rollout allocation: coordinating spending across the training horizon.
  2. Matched-budget experiments and prompt-randomization controls provide useful evidence for the allocation strategy.
Critical weaknesses
  1. The theoretical guarantees concern a surrogate objective with a fixed realized rate path, rather than allocation-dependent policy evolution. Moreover, the regret bound becomes vacuous under the experimental configuration.

  2. Evidence for adaptive pacing remains limited: ablations cover only the smallest backbone, while the main comparisons lack multi-seed uncertainty estimates, particularly for the small gains over KnapsackRL on larger models.

Questions for authors
  1. Can you provide a non-vacuous guarantee under the experimental configuration and clarify its connection to actual policy improvement?

  2. Can you compare against schedules matching CERO’s batch-size distribution and update count, and report multi-seed results on larger backbones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.