acceptodds
Under review as a conference paper at ICLR 2027

Critical Batch Size for LLM Policy Optimization

Abstract

Policy optimization has become the dominant post-training technique for large language models, but it is computationally expensive. In pretraining and supervised learning more broadly, the critical batch size (CBS), which is the batch size beyond which further increases yield diminishing returns in training steps, is well studied and guides how compute is parallelized at scale. No analogous understanding exists for policy optimization, where the question is harder: the objective changes with the policy, sequences generated from the same prompt are not independent, and the batch size itself decomposes into the number of prompts and the number of rollouts per prompt. We develop a theoretical and empirical framework to study the CBS of policy optimization, focusing on GRPO in the on-policy regime. Theoretically, we present a noise model that decomposes the gradient noise of GRPO into a prompt-level and a rollout-level term, and predicts that adding prompts reduces both while adding rollouts reduces only one. Empirically, for GRPO on mathematical reasoning, we observe near-linear reductions in optimizer steps up to 16K sequences per step when scaling prompts. Together, we give concrete guidance on scaling prompts, rollouts, and learning rate, while shedding further light on GRPO batch scaling dynamics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
optimization, critical batch size, GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Sc2fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper research the critical batch size (CBS) for LLM policy optimization by decomposing the batch into the number of distinct prompts (B) and the number of rollouts per prompt (K). Through a gradient covariance decomposition, the authors explain why increasing prompt reduces both between prompt and within prompt gradient noise, wheras increasing rollouts only reduces the latter. Experiments using Qwen2.5-Math-1.5B-Instruct on DAPO-Math, with AIME evaluation, show an extended regime of approximately linear reductions in optimizer steps when scaling prompts. The paper also examines learning-rate scaling, KL regularization, and advantage estimation.

I find the research question practically important and the empirical observations promising. However, the current submission does not provide sufficient evidence for the robustness of its central quantitative findings or for the connection between its theoretical model and the actual GRPO experiments.

Critical strengths

Strengths 1: An Clear Formulated Research Question. Allocating computation across prompts, rollouts, and sequential optimization steps in a practical challenge in LLM reinforcement learning (RL). Separating batch size into (B) and (K) provides more insight than considering only the total number of sequences.

Strengths 2: Below equation is well motivated. I did not identify an obvious algebraic error in the main theorem under its stated assumptions. The framework clearly explains why repeatedly sampling the same prompt cannot eliminate between-prompt noise. ΣpromptB+ΣrolloutBK\frac{\Sigma_{\mathrm{prompt}}}{B} + \frac{\Sigma_{\mathrm{rollout}}}{BK}

Strengths 3: Substantial experimentation with attention to learning-rate tuning. The authors report 469 runs, separately sweep prompts and rollouts, and tune learning rates across batch configurations. The difference between ((B, K)=(1024,16))((\mathrm{B}, \mathrm{~K})=(1024,16)) and  ((128,128)) \text { ((128,128)) } at the same sequence budget is practically informative. Reporting unsuccessful trajectories, configurations, and implementation corrections in the appendix also helps reader assess the results.

Critical weaknesses

Weaknesses 1: The central CBS estimates lack sufficient uncertainty and robustness analysis. According to Appendices B.2, B.5, and C, most configurations use a single random seed. For each batch size, the reported result is the fastest first crossing of the target across the learning-rate and KL configurations explored. Runs that subsequently collapse still count toward this measurement.

This protocol can characterize the best observed first-passage time, but it is sensitive to evaluation noise, the extent of hyperparameter search, and unstable runs. Evaluation frequency also differs across batch sizes. The paper does not adequately establish how these factors affect the fitted knee, the prompt-versus-rollout gap, or the learning-rate scaling exponent.

These issues do not necessarily invalidate the main trend. However, they make it difficult to assess how robust the conclusions are to randomness and measurement choices. A large number of hyperparameter trials does not substitute for independent repetitions of the key comparisons.

Appendix B.2 also reports an unintended additional (1/K) gradient scaling in some runs. The authors state that Adam absorbs this scaling and that they verified its effect, but the paper does not provide sufficient paired results to evaluate that claim. Exact equivalence is not automatic in the presence of gradient clipping and numerical stabilization terms. This further motivates repeating the central comparisons using a consistent, corrected implementation.

Weaknesses 2: The connection between the theory and practical GRPO remains largely qualitative. The main theorem analyzes a single SGD update under a local quadratic approximation. The experiments use AdamW, gradient clipping, and KL regularization, and estimate CBS from evaluation accuracy over an entire training trajectory. The authors acknowledge this gap, but do not directly measure the two noise components or establish whether the theoretical quantities predict the observed scaling behavior.

In particular, Corollary 2 establishes an (O(1/K^2)) correction for an RLOO estimator with fixed response lengths and no standard-deviation normalization. The experiments instead use GRPO advantages that share an estimated group mean and sample standard deviation. Standard-deviation estimation and variable response lengths can affect the gradient’s expectation, covariance, and relative weighting across prompts. Consequently, this corollary does not directly establish that advantage-estimation noise is unimportant for CBS in the experimental setting.

I am not asking for a general convergence theorem for non-convex AdamW. However, the paper should distinguish more clearly between what the theory establishes and what the experiments suggest. A targeted mechanism-validation experiment, or a corresponding narrowing of the mechanistic claims, would strengthen the contribution.

Questions for authors

Question 1: Could the authors repeat the key comparisons at equal sequence budgets and a small number of configurations near the fitted knee, using multiple seeds and the corrected implementation? Do the prompt rollout gap and estimated CBS persist when requiring the target to be reached across consecutive evaluations or excluding runs that subsequently collapse?

Question 2: What do paired comparisons before and after the gradient-normalization correction show? Did the scaling affect both the policy and KL losses, and did it change how frequently gradient clipping was activated.

Question 3: Is there direct gradient-noise measurement or a ablation involving the actual standardized GRPO estimator that supports the proposed mechanism? Which conclusions about advantage estimation noise follow from the theorem, and which are supported only by the current experiment?

Question 4: For the central comparisons at equal sequence budgets, what are the actual generated token counts, per-step runtimes, and CPU time to target? If the intended contribution concerns statistical efficiency under ideal parallelism, could the wording be made consistent throughout?

Reviewer_mikoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the critical batch size (CBS) for on-policy GRPO in LLM post-training, where the batch decomposes into prompts (B) and rollouts per prompt (K). Extending McCandlish et al.’s noise-scale model, the authors split gradient noise into a prompt-level and a rollout-level term and argue that, at a fixed sequence budget, adding prompts permits a larger step size than adding rollouts. Empirically, they train Qwen2.5-Math-1.5B-Instruct on DAPO and measure steps to 50% on AIME 1983-2024 across 469 runs. They report near-linear step reductions up to ~16K sequences per step when scaling prompts, earlier saturation when scaling rollouts, a √B optimal learning rate along the prompt axis (roughly flat along the rollout axis), and a CBS that is insensitive to KL strength and advantage quality but grows over training. They conclude that GRPO is far more parallelizable along the prompt axis than current practice assumes.

Critical strengths
  1. Timely question (CBS for GRPO based RL post-training) with an actionable finding: steps fall near-linearly with prompts up to ~16K sequences, while rollouts saturate about 10× earlier.
  2. Large, carefully documented empirical effort (469 runs, per-batch LR sweeps, √B learning-rate rule that transfers from K=16 to K=64).
  3. Theoretical soundness. The paper gives a clean, explicit decomposition of GRPO gradient noise into a prompt-level term (σ²_prompt) and a rollout-level term (σ²_rollout/K), and derives closed-form expressions for the optimal step size and the local critical batch size along each axis (Theorem 1). This extends the McCandlish et al. noise-scale model to the two-axis structure of GRPO batches, and it yields a simple, transparent explanation of why, at a fixed sequence budget, allocating samples to prompts supports a larger step size than allocating them to rollouts (Corollary 1).
Critical weaknesses
  1. Overclaimed scope. Title says “LLM policy optimization,” but this is one on-policy, single-step GRPO variant on one 1.5B model and one task. Although I lean towards an accept, I still recommend a fix of the scoping.
  2. Theory and data seem to be in tension. Theorem 1 gives B_crit(K) = σ²_prompt + σ²_rollout/K. Converting to total sequences N = B·K, this implies N_crit(K) = K·σ²_prompt + σ²_rollout, which grows linearly in K whenever σ²_prompt > 0. However, §4.2 reports that the sequence-level CBS is approximately the same for K = 16 and K = 64. Can the authors reconcile this, e.g., by showing σ²_prompt is small or by measuring the noise scales directly?
Questions for authors

(see weaknesses)

Reviewer_GpFtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work study how prompts per update and number of roll out per prompt affect the RL outcome and attempts to theoretically model the influence as well as a strategy for optimal learning rate in GRPO. They use Qwen 2.5 model and evaluated on AIME and showed that with same number of sequences per update, more prompts with less rollout per prompt shows performs better than more rollout per prompt. Near-linear reduction in optimizer step is observed when scaling prompts on AIME up to 16k sequence per update, while scaling rollout stops at 2k. They also find the best learning rate for the setup is roughly square root of the number of prompts but is nearly constant when scaling rollout, which align with previous study results on batch size of AdamW.

Critical strengths

They study an interesting question on GRPO RL strategy and made a meaningful attempt on specific model/task combination.

Critical weaknesses
  1. Too few models and tasks, this heavily limited the paper’s batch-allocation guidance beyond tested model. Task diversity is very limited; rules that work on one old math setting (which may already largely present in model pretrain data) may not generalize to code, agentic, and other tasks.
  2. Does not establish scaling up property of the observation, and it need more seeds (error bar), and this is critical for small training runs.
Questions for authors

How do you know your results is not limited to the specific model weight? Does this more prompts win over more rollouts generalize to other important tasks such as coding? What happen when we scaling up (1) the model weight and (2) the dataset? Without these answered, the observation and rule established here are hard to justify for its contribution to the research field or industry applications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.