- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CERO treats the rollout budget for RL post-training to allocate over the whole training run, deciding both which prompts get rollouts and how much to spend in each round. It models each prompt's value as a concave utility of its cumulative exposure, with the utility's rate estimated by the predicted within-group reward variance from a per-prompt Beta–Bernoulli state. A Fenchel representation reduces each round to ranking prompts by a margin between a prompt-specific slope and a shared budget price, both updated by projected online gradient descent. The authors prove pathwise regret bounds on this surrogate objective against fixed-rate and time-varying oracles. On three Qwen/DeepSeek backbones and five math benchmarks under a matched 256k-response budget, CERO achieves the best avg@16 macro-average, supported by a 3-seed ablation of its pacing and selection components.
Critical strengths
- Treating the rollout budget as a horizon-level resource, rather than a per-round one, is a natural and underexplored angle. The online primal–dual machinery from ad allocation and revenue management is a sensible import. Each round reduces to a sort, with closed-form conjugates and supporting exposures, and measured compute overhead is negligible.
- The paper reports detailed hyperparameters and the reproducibility looks good.
Critical weaknesses
- The theory appears to be vacuous in the experimental regime. The per-prompt regret bound scales with , where . If , , and of roughly 8-9k prompts (half of DAPO-Math-17K), . So . The optimization term is then about , which exceeds the trivial bound of 1 at . The proxy term, , is about . The guarantees therefore say nothing for the settings actually run. This should be acknowledged, or the regime where the bounds bite should be characterized.
- The pacing ablation is confounded. CERO-fixed sets ν = 0, which also changes the admission threshold and the margins, not just the spending schedule. Its gap to CERO therefore mixes selection and pacing effects. A cleaner control would replay CERO's prompt choices under a uniform schedule.
Questions for authors
- The selection-lift analysis suggests CERO mostly favors prompts with predicted success near 0.5. How does CERO compare with a simple rule that selects prompts whose Beta mean is closest to 0.5, or with DAPO-style dynamic filtering of all-correct and all-incorrect groups?
- Why not compare against the difficulty-aware methods you cite, such as SPEED-RL, Goldilocks RL, and VADE? These target the same "informative prompt" signal.