- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper research the critical batch size (CBS) for LLM policy optimization by decomposing the batch into the number of distinct prompts (B) and the number of rollouts per prompt (K). Through a gradient covariance decomposition, the authors explain why increasing prompt reduces both between prompt and within prompt gradient noise, wheras increasing rollouts only reduces the latter. Experiments using Qwen2.5-Math-1.5B-Instruct on DAPO-Math, with AIME evaluation, show an extended regime of approximately linear reductions in optimizer steps when scaling prompts. The paper also examines learning-rate scaling, KL regularization, and advantage estimation.
I find the research question practically important and the empirical observations promising. However, the current submission does not provide sufficient evidence for the robustness of its central quantitative findings or for the connection between its theoretical model and the actual GRPO experiments.
Critical strengths
Strengths 1: An Clear Formulated Research Question. Allocating computation across prompts, rollouts, and sequential optimization steps in a practical challenge in LLM reinforcement learning (RL). Separating batch size into (B) and (K) provides more insight than considering only the total number of sequences.
Strengths 2: Below equation is well motivated. I did not identify an obvious algebraic error in the main theorem under its stated assumptions. The framework clearly explains why repeatedly sampling the same prompt cannot eliminate between-prompt noise.
Strengths 3: Substantial experimentation with attention to learning-rate tuning. The authors report 469 runs, separately sweep prompts and rollouts, and tune learning rates across batch configurations. The difference between and at the same sequence budget is practically informative. Reporting unsuccessful trajectories, configurations, and implementation corrections in the appendix also helps reader assess the results.
Critical weaknesses
Weaknesses 1: The central CBS estimates lack sufficient uncertainty and robustness analysis. According to Appendices B.2, B.5, and C, most configurations use a single random seed. For each batch size, the reported result is the fastest first crossing of the target across the learning-rate and KL configurations explored. Runs that subsequently collapse still count toward this measurement.
This protocol can characterize the best observed first-passage time, but it is sensitive to evaluation noise, the extent of hyperparameter search, and unstable runs. Evaluation frequency also differs across batch sizes. The paper does not adequately establish how these factors affect the fitted knee, the prompt-versus-rollout gap, or the learning-rate scaling exponent.
These issues do not necessarily invalidate the main trend. However, they make it difficult to assess how robust the conclusions are to randomness and measurement choices. A large number of hyperparameter trials does not substitute for independent repetitions of the key comparisons.
Appendix B.2 also reports an unintended additional (1/K) gradient scaling in some runs. The authors state that Adam absorbs this scaling and that they verified its effect, but the paper does not provide sufficient paired results to evaluate that claim. Exact equivalence is not automatic in the presence of gradient clipping and numerical stabilization terms. This further motivates repeating the central comparisons using a consistent, corrected implementation.
Weaknesses 2: The connection between the theory and practical GRPO remains largely qualitative. The main theorem analyzes a single SGD update under a local quadratic approximation. The experiments use AdamW, gradient clipping, and KL regularization, and estimate CBS from evaluation accuracy over an entire training trajectory. The authors acknowledge this gap, but do not directly measure the two noise components or establish whether the theoretical quantities predict the observed scaling behavior.
In particular, Corollary 2 establishes an (O(1/K^2)) correction for an RLOO estimator with fixed response lengths and no standard-deviation normalization. The experiments instead use GRPO advantages that share an estimated group mean and sample standard deviation. Standard-deviation estimation and variable response lengths can affect the gradient’s expectation, covariance, and relative weighting across prompts. Consequently, this corollary does not directly establish that advantage-estimation noise is unimportant for CBS in the experimental setting.
I am not asking for a general convergence theorem for non-convex AdamW. However, the paper should distinguish more clearly between what the theory establishes and what the experiments suggest. A targeted mechanism-validation experiment, or a corresponding narrowing of the mechanistic claims, would strengthen the contribution.
Questions for authors
Question 1: Could the authors repeat the key comparisons at equal sequence budgets and a small number of configurations near the fitted knee, using multiple seeds and the corrected implementation? Do the prompt rollout gap and estimated CBS persist when requiring the target to be reached across consecutive evaluations or excluding runs that subsequently collapse?
Question 2: What do paired comparisons before and after the gradient-normalization correction show? Did the scaling affect both the policy and KL losses, and did it change how frequently gradient clipping was activated.
Question 3: Is there direct gradient-noise measurement or a ablation involving the actual standardized GRPO estimator that supports the proposed mechanism? Which conclusions about advantage estimation noise follow from the theorem, and which are supported only by the current experiment?
Question 4: For the central comparisons at equal sequence budgets, what are the actual generated token counts, per-step runtimes, and CPU time to target? If the intended contribution concerns statistical efficiency under ideal parallelism, could the wording be made consistent throughout?