acceptodds
Under review as a conference paper at ICLR 2027

Let the Batch Decide, On- or Off-Policy: Effective Sample Size for RL Post-Training

Abstract

Reinforcement learning (RL) is structurally harder than supervised learning because the policy changes the data distribution it learns from. The resulting fragility is especially visible in large-model training using nominally on-policy RL, where the training and rollout systems can differ in numerical precision, sampling and other implementation details. The common approach handles this difference by constraining the update with a fixed clip range, so the algorithm becomes more sensitive to its configuration and must be retuned whenever the task, model scale or distribution mismatch changes. We argue that the fragility traces to two linked concerns that this one fixed range has to cover. The first is a trust-region concern: an update should not move the policy too far from its current value. The second is an off-policy concern: data from older or different behavior policies should influence the update only to the extent that the update remains reliable. Neither concern is a constant to set in advance, and the policy ratios of the current batch show how severe each one is. We adopt P3O, a simple yet effective batch-adaptive objective from prior work that replaces fixed clipping with the normalized effective sample size of the per-token policy ratios, which we take against the rollout engine's log-probabilities. The same statistic caps the score-function weight and sets the strength of an off-policy regularizer, so the update stays close to the on-policy update when the ratios are close to one. When stale or mismatched data concentrate the ratios, the update tightens but keeps a nonzero score-function weight on every token. Experiments on math reasoning show that P3O compares favorably with tuned baselines and, under FP8 rollouts on Qwen3-8B-Base, avoids the late collapse of GRPO. P3O adds no new objective hyper-parameters and removes the clip range.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Large language models, off-policy, RL Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7bSwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of fixed clipping in large language model reinforcement learning when the training and rollout policies differ. The authors adapt the existing P3O algorithm to LLM post-training, using the normalized effective sample size (ESS) of token-level importance ratios to determine both the cap on policy-gradient weights and the strength of regularization toward the behavior policy. Experiments examine clipping sensitivity, sampling temperature, mixed-precision rollouts, and asynchronous training. P3O performs comparably to tuned GRPO in some settings and avoids GRPO’s late-stage collapse in the Qwen3-8B FP8-rollout experiment in some configurations.

Critical strengths

The key strength of this paper is replacing fixed clipping thresholds, which require tuning across different training settings, with a batch-adaptive mechanism based on effective sample size (ESS). By using ESS to jointly control importance-weight capping and regularization strength, P3O provides a unified objective for handling varying degrees of policy mismatch without manually selecting a clipping range.

Critical weaknesses
  1. The method has limited novelty and insufficient insight because the core component (ESS) is inherited from prior works. This paper mainly adapts ESS to LLM training, but lacks a systematic understanding of why it works and when it is beneficial.
  2. The paper lacks explanation on why the method works and why it might fail. The results support the effectiveness of the objective in some configurations, but do not establish that its batch-adaptive mechanism is responsible for the gains. The inconsistent outcomes across temperatures and models make this explanatory gap particularly important. More effort should be devoted to explaining the observations from the experiments rather than simply displaying the results.
Questions for authors
  1. Figure 1 shows that the commonly used clipping threshold ϵ=0.2\epsilon=0.2 performs substantially worse than other choices in GRPO and P3O. Could you provide explanation on why this happens, and how different clipping thresholds in standard algorithms affect the stability of training?
  2. Figure 2 shows that P3O remains stable at higher temperatures but collapses at lower temperatures. What is the reason for this phenomenon and is it a limitation of this method?
  3. The experiments show that P3O can prevent training collapse, but do not explain why GRPO collapses while P3O remains stable. Could the authors analyze the evolution of policy mismatch, importance-ratio distributions, ESS, and gradient statistics throughout training?
  4. The FP8 experiments attribute P3O's stability advantage to robustness against precision mismatch, but lack matched BF16-rollout controls and comparisons with GRPO under different clipping thresholds. Moreover, GRPO collapses on Qwen3-8B but not on Qwen2.5-1.5B, without an explanation for this discrepancy. Could you include BF16-versus-FP8 rollout comparisons on the same models and analyze the actual probability mismatch?
Reviewer_kvcuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the paper adapts P3O (Fakoor et al., 2019) to rl post-training of llms. instead of a fixed clip range, the per-token score-function weight is capped at the normalized effective sample size (ess) of the batch’s policy ratios, and a kl regularizer toward the behavior policy is added with coefficient (1 - ess). ratios are taken against the rollout engine’s recorded log-probabilities, so the cap applies to every token, and group-relative advantages replace the critic. the objective has no clip hyper-parameter. experiments on math reasoning with qwen models compare p3o against grpo under a clip sweep, asynchronous training, rollout temperature mismatch, and bf16 train + fp8 rollout. the main claimed result is that grpo collapses late in training on qwen3-8b-base with fp8 rollouts while p3o does not.

Critical strengths
  • the paper correctly identifies the two concerns that fixed clipping has to cover at once, the trust-region concern and the off-policy concern, and most of the recent work on llm policy optimization objectives is indeed about handling these two issues. the point that stale or mismatched data can destabilize training while discarding them wastes rollouts is also well made.
  • the objective is simple and clearly presented. the relation to cispo (replace e_b by a fixed cap and drop the kl term) and the reduction to grpo when all ratios equal one are stated precisely, and table 1 is a useful summary of how prior objectives re-parameterize the clip.
  • the fp8 rollout experiment on qwen3-8b-base is a practically relevant setting, and the authors are honest about where p3o is sub-optimal.
Critical weaknesses
  • the learning rate is not swept. all runs use lr = 1e-5 with warmup cosine (table 5), and the clip sweep in fig. 1 varies only ε at this fixed lr. instability of grpo at a given ε is plausibly an lr issue, and grpo may well be stable at ε = 0.2 with a tuned lr. without an lr sweep for at least the grpo baselines, i do not trust any of the experimental findings.
  • the sensitivity claim itself is overstated. the text says grpo’s reward “varies substantially with ε” on qwen3-4b-thinking-2507, but of the six grpo clip runs across the two models in fig. 1, only one (ε = 0.2 on qwen3-4b) is unstable. the other five are v close to p3o.
  • the grpo collapse under fp8 rollouts (fig. 3a) is compared against vanilla grpo with std-normalized advantages and no correction. prior work already shows that this kind of late collapse can be addressed by removing the std normalization (Liu et al., 2025) or by dapo-style changes, and the ScaleRl study (Khatri et al., 2025) establishes that these loss-level choices matter for stability. the relevant baseline is a stabilized grpo variant at matched settings, not vanilla grpo.
  • probably my biggest complaint: the authors show comparison against an averaged grpo baseline ε ∈ {0.2, 0.4, 0.6} in the main paper. this baseline doesn't make sense. the true comparison is against the best clip, and the paper’s own appendix shows that grpo with ε = 0.6 is above p3o on every benchmark at the 16k-token budget (table 4, sec. b.3). this is acknowledged in one clause of sec. 4.4 but not in the figure or the main table. the averaged baseline should be replaced by per-clip numbers, with the best clip reported.
  • because of this, the central motivation, that ε must be retuned per setting, is not supported by any experiment. the same clip (ε = 0.6) is best on all five benchmarks in table 4, ε = 0.4 is used throughout the off-policy experiments, and on qwen3-4b-base (table 3) p3o lies between the best and worst grpo clip on every benchmark and is best on none, while dapo with a fixed asymmetric clip (0.2, 0.28) is best on three of five columns at an earlier iteration (30 vs. 50). if the optimal ε barely moves across these settings and sensitivity to it is low, a batch-adaptive cap is not needed, and the results suggest that setting the cap at the ess may simply be a sub-optimal choice of effective clip rather than a benefit of adaptivity. i am not sure what benefits p3o offers and why one would use this algorithm.
Questions for authors
  • what is the range of e_b observed during training in each experiment family (clip, temperature, fp8, async)? in the one-step clip runs, where the only source of ratio deviation is the engine log-probability gap, how far below one does e_b get? if it stays near one, p3o is effectively cispo?
  • what happens if the learning rates are tuned for the grpo baselines in these experimental settings?
  • what do the new result figures look like if you plot the correct baselines, i.e., the best \epsilon for grpo?
Reviewer_KCrNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper applies the previously proposed P3O objective function to post-training of LLMs using reinforcement learning. It replaces a fixed pruning range by using the normalized effective sample size of the proportion of lexical-level policies in each batch to set the upper bound of gradient weights and the regularization term for the unfolding policy. The authors evaluate this objective function on mathematical reasoning tasks under synchronous training, asynchronous unfolding, and unfolding-training accuracy mismatch conditions. The most significant results show that P3O avoids the late-stage performance degradation problem of the GRPO algorithm when using FP8 unfolding on the Qwen3-8B-Base dataset

Critical strengths

This paper explores a practical problem: even fresh rollouts can be off-policy when rollout and training engines assign different token probabilities. The proposed adaptive method is easy to integrate and eliminates the clipping range selection problem in the objective function. An FP8 comparison provides a concrete example illustrating the importance of this. Besides, this paper reports mixed and negative results, including settings where both methods collapse, and explicitly discusses limits of the ESS statistic.

Critical weaknesses
  1. The central ESS-based cap and regularizer come from P3O (2019). The contribution is thus primarily its adaptation and evaluation for LLM post-training, so the methodological novelty is modest.
  2. The comparisons with GSPO, CISPO, and quantized-rollout adaptive clipping methods are missing. The advantage is also inconsistent: a tuned GRPO clip outperforms P3O in the longer-budget Qwen3-4B-Thinking evaluation.
  3. ESS measures concentration of the observed ratios, not policy distance, and depends on how tokens are grouped into micro-batches. The sampled KL approximation can also produce an unbounded gradient as a ratio approaches zero, so the method does not guarantee a bounded update.
Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.