acceptodds
Under review as a conference paper at ICLR 2027

Ratio-Free Policy Optimization: A Bias-Variance View of On-Policy Fine-Tuning

Abstract

Practical LLM reinforcement learning is never strictly on-policy: each batch of rollouts is reused for many optimizer steps, and rollouts come from a separate inference engine. We show that GRPO degrades under both drifts — and on coding collapses — and explain why. We formalize on-policy fine-tuning (OPFT), updating a model on its own rollouts, as the OPFT loop, of which GRPO and iterative rejection-sampling SFT are realizations, and propose RFPO (Ratio-Free Policy Optimization), a novel realization that optimizes the surrogate — a reward-weighted maximum-likelihood objective requiring no importance ratio — as a local proxy for the on-policy expected reward . We identify three mechanisms — z-score advantages, negative samples, and trust-region clipping — that make this surrogate stable, each with a distinct role verified by theory and ablations; the analysis also explains the well-known collapse of iterative rejection-sampling SFT. GRPO realises analogous mechanisms through different machinery: the core updates of RFPO and GRPO coincide at strict on-policy and diverge under the practical sampling gaps that any modern LLM RL stack faces (multiple optimizer steps per batch; decoupled rollout/training engines). A bias-variance analysis predicts that GRPO's importance-sampling variance scales with the gap while RFPO's bias is controlled by a per-token trust region, so RFPO dominates in MSE under sufficient drift. On Qwen3-4B training on POLARIS/DeepCoder we verify the prediction along both drift axes: as we increase optimizer steps per batch, GRPO degrades from 0.90 (MATH-500)/0.67 (LCBv6) at 1 step through 0.87/0.38 at 8 steps to 0.85/0.24 at 32 steps, while RFPO stays at 0.90/0.67; under bypass-mode rollout drift (8 steps), GRPO collapses () while RFPO is nearly unaffected (0.90/0.67) — and, needing no reference model, KL term or importance ratio, RFPO trains 1.3–1.9 faster per step.

Primary area:
reinforcement learning
Keywords:
LLM, reinforcement learning, on-policy fine-tuning, ratio-free policy optimization, training stability, policy drifting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5jvsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ratio-Free Policy Optimization (RFPO) to address training instability caused by policy drift and rollout drift in LLM reinforcement learning. RFPO replaces importance-ratio-based optimization with an advantage-weighted maximum likelihood objective and introduces three stabilizing mechanisms: z-score advantages, negative samples, and absolute probability clipping. Experiments on mathematical reasoning and coding tasks show that RFPO is more stable than GRPO under increasing policy drift and achieves better training efficiency.

Critical strengths
  1. The paper addresses an important practical problem in LLM reinforcement learning, particularly the instability caused by policy drift and differences between rollout and training engines.
  2. RFPO offers a simple optimization approach without importance ratios or a reference model, potentially reducing implementation complexity and computational cost.
  3. The experimental results show promising improvements over GRPO, particularly on coding tasks. The ablation studies also provide useful evidence for the proposed stabilization mechanisms.
Critical weaknesses
  1. The variance analysis in Appendix B assumes that GRPO clipping keeps importance ratios within [1−ϵ,1+ϵ][1-\epsilon,1+\epsilon], which standard clipping does not guarantee. The authors should correct this assumption and directly measure gradient bias, variance, and MSE to support their main theoretical claim.
  2. Section 4.3 claims that negative samples make the RFPO objective well-defined. However, the advantage-weighted log-likelihood can still be unbounded when negative advantages are included. The authors should clarify the conditions under which the objective remains bounded.
  3. Section 4.3 uses token-level probability clipping to control policy drift, but Appendix B acknowledges that this does not provide a meaningful sequence-level bound for long responses. The authors should provide stronger theoretical justification or experiments examining sequence-level divergence.
  4. RFPO removes importance ratios and introduces absolute clipping simultaneously. The current experiments cannot clearly identify which change drives the improvement. A controlled comparison of ratio-based and ratio-free objectives under both clipping strategies is needed.
  5. The experiments lack comparisons with stronger alternatives such as GSPO and DAPO. Moreover, the LiveCodeBench results mainly use a single random seed, and MATH-500 validation uses only 100 questions. More competitive baselines and repeated evaluations are needed.
  6. The efficiency comparison in Section 5.4 reports a 1.3× speedup on mathematical reasoning, but compares RFPO with mini-batch size 8 against GRPO with mini-batch size 256. A matched mini-batch comparison is needed to establish the actual speedup.
Questions for authors

see weeknesses

1 comment

Thank you for the specific and actionable review. Our general response summarizes the new experiments.

W1. GRPO's ratio is not bounded by clipping; measure bias, variance and MSE directly. You are right, and Appendix B is corrected. A token with negative advantage whose probability rose keeps a ratio-weighted gradient, capped only by verl's dual clip at rt=3r_t = 3, which fires on 9.6×10−59.6\times10^{-5} of tokens at mini-batch size 8 and never at 256. As you suggested, we measured both estimators directly. From training checkpoints, we run k=0,…,31k = 0, \dots, 31 inner updates on one rollout batch and compare each estimator with a fresh on-policy estimate of ∇J\nabla J. The squared bias of the importance-weighted estimator grows to 9-27 times the noise floor at large kk, while that of the ratio-free estimator stays at 0.9-1.7 (rising to 3.9 at k=31k = 31 at the GRPO checkpoint). The ratio-free estimator's MSE is 1.3-2.5 times lower once k≥16k \ge 16, but at k=4k = 4-88 the importance-weighted estimator can be better. Section 4.2 and Figure 1 now state this (protocol: Appendix H).

W2. The objective with negative advantages is unbounded. Agreed; "well-defined" was the wrong word. M2 removes the degenerate fixed point of positive-only refitting, but it does not bound the loss. Each update is bounded by the probability floor (a token with πold≤δ\pi_{\text{old}} \le \delta cannot fall by more than δ\delta), by M3 on already-likely tokens, and by the bounded gradient of log⁡π\log\pi with respect to the logits. Section 4.3 (M2 paragraph) now states these conditions.

W3. Per-token clipping gives no sequence-level bound. Agreed, and we measured it. After 31 inner updates, the mean absolute summed log-ratio per sequence, ∣∑tlog⁡(πθ/πold)∣|\sum_t \log(\pi_\theta/\pi_{\text{old}})|, is 3.3 at the base model, 8.6 at an RFPO checkpoint and 11.7 at a GRPO checkpoint (99th percentile up to 81), while per-token ratios stay within about 1% of 1. So the per-token cap does not bound drift at the sequence level, as you say, and we now describe M3 as a local regularizer. A sequence- or prefix-level trust region on the ratio-free surrogate, in the style of GSPO or MinPRO, is the natural extension (Section 4.3, M3 paragraph; Appendix D).

W4. Removing the ratio vs. changing the clip. We ran the full 2x2 at mini-batch size 8 without KL. On full MATH-500: ratio + ratio clip 0.818, ratio + absolute clip 0.860, ratio-free + ratio clip 0.858, and ratio-free + absolute clip (RFPO) 0.869±0.0110.869 \pm 0.011. On coding (LCBv6, median over steps 240-270), both cells with the ratio collapse: ratio + absolute clip to 0.00, and ratio + ratio clip (GRPO with KL; KL-free GRPO also collapses) to 0.12-0.23 by step 110. Both ratio-free cells stay stable, at 0.65 and 0.66-0.67. Under heavy drift, the ratio is what breaks, whichever clip is used.

W5. Stronger baselines and repeated evaluations. GSPO reaches 0.702 on MATH-500 and 0.27 on LCBv6. DAPO (without dynamic sampling, which our trainer does not support) reaches 0.858 on MATH-500 and collapses to 0.09 on coding. Every arm is now evaluated on all 500 MATH-500 problems and on AIME 2024/2025, and RFPO and GRPO at mini-batch size 8 have three independent runs each on math (0.869±0.0110.869 \pm 0.011 vs. 0.823±0.0120.823 \pm 0.012) and on coding (GRPO collapses in all three).

W6. Efficiency at matched mini-batch size. Agreed, and checking this revealed an error in our own comparison: the coding numbers in Section 5.4 of the submission (now Section 5.5) set a GRPO run on 2 nodes against an RFPO run on 1 node. On one 8xH100 node at mini-batch size 8 (median seconds per step), a math step takes 367 s for RFPO and 514 s for KL-free GRPO, and a coding step 756-773 s for RFPO and 1585-1691 s for GRPO with KL. (GRPO with KL runs FSDP2; wall-clock includes generation.) At matched mini-batch size 8, RFPO is thus 1.4 times faster per step than KL-free GRPO (367 vs. 514 s) and 1.6 times faster than GRPO with KL (586 s); the revision and the abstract now state the speed-up this way.

We have tried to address each of your points with the experiments you suggested. If they resolve your concerns, we would be grateful if you would consider updating your score.

Reviewer_Pk2COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines RL fine-tuning of LLMs when updates aren’t strictly on-policy, distinguishing policy drift (the actor moving away from its batch-start snapshot over multiple mini-batch updates) from rollout drift (mismatch between the inference and training engines). It proposes RFPO, which maximizes an advantage-weighted log-likelihood without importance ratios, stabilized by group-normalized advantages, mean-zero advantages that provide negative samples, and a per-token trust region that stops a token’s gradient once its probability has shifted too far in the advantage direction. The authors argue that GRPO’s importance-sampling variance grows with drift while RFPO’s surrogate bias stays controlled, so RFPO should win when drift is large.

Critical strengths

Relevant, well-framed problem. Separating within-batch policy drift from engine mismatch, and turning both into controllable knobs (mini-batch size, bypass mode), matches how current LLM RL stacks are run and gives a clean experimental frame.

Simple drop-in update with a clear empirical signal. RFPO changes only the update step and needs no reference model.

Informative ablations and candid scoping. The build-up and leave-one-out ablations and the failed-stabilizer audit show clearly that positive-only training is unstable and that negatives need a trust region. The KL-free GRPO control and the loss-scaling check remove two obvious confounds, and the authors state openly that the objective is a known reward-weighted MLE and that no sharp crossover is derived.

Critical weaknesses

Thin statistical evidence. "MATH-500" is a fixed 100-problem subset scored greedily, and coding, bypass and ablation runs are single-seed; more seeds on key coding cells and the full benchmark would help.

Questions for authors

Does tolerating drift pay off? Late accuracy at large K only matches strict on-policy GRPO, and the speedup appears to come from dropping the reference model and KL, which KL-free GRPO also avoids. Please report steps to a target accuracy, and timings against KL-free GRPO at matched mini-batch size and response length.

Suggestion (related work; does not affect my rating): "Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?" (NeurIPS 2026) studies the same question and proposes LGPO, which drops the old-policy likelihoods used to form importance ratios and enforces a soft trust region through likelihood-based gating. How does RFPO differ from it, both in the update rule and in the bias–variance analysis? Discussing this work would help position the paper.

1 comment

Thank you for the positive assessment of the framing and the ablations. Our general response above summarizes all new experiments and changes; here we answer your points directly.

Statistics. Agreed, and done. Every arm is now evaluated on all 500 MATH-500 problems and on AIME 2024/2025, through the same validation code that produced the in-training numbers, and RFPO and GRPO at mini-batch size 8 have three independent runs each (Section 5.1, Table 2; per-run values in Appendix I, Table 9):

full MATH-500AIME25 avg@16
base model0.6780.094
GRPO, mini-batch 256 (strict on-policy)0.8740.340
GRPO, mini-batch 8 (3 runs)0.823±0.0120.823 \pm 0.0120.232
RFPO, mini-batch 8 (3 runs)0.869±0.0110.869 \pm 0.0110.344

Table 1's drift sweep is now also on full MATH-500, from re-runs with distinct data seeds: GRPO 0.879 / 0.861 / 0.824 at 1 / 8 / 32 optimizer steps per batch, RFPO 0.873-0.875. On coding we added two runs per method; RFPO under bypass at 8 steps per batch also has three runs, while GRPO's bypass cells and the mechanism ablations remain single runs, which the revision states. GRPO at mini-batch size 8 collapses in both (0.12 and 0.23 on LCBv6 by step 110), while RFPO stays at 0.66-0.67 (median over steps 240-270; base model 0.41).

Does tolerating drift pay off? RFPO does not exceed strict on-policy accuracy; it matches it (0.869 vs. 0.874). What drift tolerance buys is reaching that accuracy with many optimizer steps per rollout batch, and therefore sooner:

  • RFPO at mini-batch size 8 reaches 0.85 on the 100-problem MATH-500 validation subset within 10-20 steps (about 1.9 h); strict on-policy GRPO needs 30 steps (about 5.7 h).
  • RFPO reaches 0.60 on LCBv6 at step 25 (step 65 at mini-batch size 32); strict on-policy GRPO needs 135 steps, and GRPO at mini-batch size 32 or 8 never reaches it, with or without KL.

You are right that KL-free GRPO also avoids the reference model. On the same single node at mini-batch size 8, a math step takes 367 s for RFPO and 514 s for KL-free GRPO (median; wall-clock includes generation, so it also reflects response length, and we do not separate the two). The decisive difference is stability: KL-free GRPO at mini-batch size 8 reaches 0.818 on full MATH-500, against 0.869 for RFPO, and collapses on coding (0.214 in the paper). The efficiency section (Section 5.4 in the submission, now Section 5.5) is also corrected. Its coding comparison set a GRPO run on 2 nodes against an RFPO run on 1 node, and its RFPO figure (~800 s/step) does not match our logs (667-724 s/step). On one node at mini-batch size 8, a coding step takes 756-773 s for RFPO and 1585-1691 s for GRPO with KL (median).

Relation to LGPO. Thank you for the pointer (Zhang et al.; ICLR 2026 workshop, arXiv 2611.15404). Both optimize the advantage-weighted log-likelihood without old-policy likelihoods; they differ in three ways. (1) Trust region: LGPO zeroes whole sequences whose current average log-likelihood exceeds a fixed threshold, regardless of advantage sign or snapshot; RFPO masks individual tokens whose probability moved more than δ\delta from the batch-start snapshot in the direction of the advantage, which caps the drift that drives the surrogate's bias. (2) Scope: LGPO's main experiments use 2-4 updates per batch with 1.5B-3B models; we study up to 32 inner updates, engine drift, and coding, where GRPO collapses. (3) Bias-variance: both methods share the ratio-free surrogate's bias, which our analysis and direct measurement quantify against the importance-weighted estimator's error as drift grows (Section 4.2); RFPO's per-token cap bounds that drift directly, while LGPO's gate bounds it only through the sequence likelihood. The revision discusses LGPO in related work (Section 2).

We hope the full-benchmark results and repeated runs resolve your concern about the statistics. If so, we would appreciate it if you would consider raising your score.

Reviewer_fwPeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how policy drift and rollout drift affect GRPO’s performance. The authors propose Ratio-Free Policy Optimization (RFPO), a simpler method that uses an advantage-weighted log-likelihood objective without importance ratios. They provide a bias–variance analysis of the on-policy fine-tuning (OPFT) loop and introduce three mechanisms for training stability, including z-score advantages, negative samples, and absolute per-token probability clipping. Experiments using Qwen3-4B-Instruct on math reasoning and coding compare RFPO and GRPO across mini-batch sizes and different rollout settings. RFPO shows better stability and lower training costs.

Critical strengths
  • This paper provides empirical studies of two sources of policy mismatch. The results show clear improvements in training stability, especially on the coding task.
  • The ablation studies add three mechanisms incrementally and remove them individually, helping clarify each mechanism’s contribution. The results show that training stability depends on combining negative samples with clipping, providing a practical recipe.
Critical weaknesses
  • The paper’s bias–variance claims are not fully supported. The authors conclude that RFPO achieves lower overall MSE than GRPO under sufficient drift, while treating RFPO’s variance as independent of policy drift and GRPO’s importance ratios as bounded by clipping. However, RFPO’s variance may increase as the model gradient changes with policy drift. In GRPO, clipping only suppresses gradient contributions, but it does not enforce a range for every importance ratio.
  • The experimental evaluation is not sufficient. Coding shows larger stability gains, but an ablation study that isolates ratio removal from clipping is missing for the coding task. Some results are based on single runs without standard deviations. Evaluation on more LLM backbones and comparisons with other RL methods are needed.
Questions for authors
  • How is the absolute clipping threshold δ=0.1\delta=0.1 selected? How sensitive are the results to it?
  • What does it mean that including negative samples makes the objective “well-defined”?
  • What are the sources of rollout drift? Which specific numerical settings differ between the rollout and training engines in the reported experiments?
1 comment

Thank you for the careful review. You found a real error in our analysis; we corrected it and replaced the argument with a direct measurement. Our general response summarizes the new experiments.

W1a. Clipping does not bound GRPO's importance ratios. You are right. Clipping removes a token's gradient only once its ratio leaves [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon] in the direction that improves the objective. A token with negative advantage whose probability rose therefore keeps the term rtA∇log⁡πθr_t A \nabla\log\pi_\theta with rt>1+ϵr_t > 1+\epsilon. The only cap is verl's dual clip at c=3c = 3, which is active in our runs: it fires on 9.6×10−59.6\times10^{-5} of tokens at mini-batch size 8 and never at 256. The per-step variance bound is therefore about c2=9c^2 = 9 times the on-policy level, not (1+ϵ)2(1+\epsilon)^2. Appendix B is rewritten accordingly.

W1b. RFPO's variance also changes with drift. Agreed. What holds is only that the ratio-free estimator carries no ratio factor, so neither bound settles the comparison. We therefore measured it. From a training checkpoint, we take one rollout batch, run k=0,…,31k = 0, \dots, 31 inner updates, and at each kk compare both estimators with a fresh on-policy estimate of ∇J\nabla J (Section 4.2, Figure 1; protocol and per-kk values in Appendix H, Table 7).

  • At an early RFPO checkpoint (step 25), the squared bias of the importance-weighted estimator grows to 15-27 times the noise floor for k≥8k \ge 8, while that of the ratio-free estimator stays at 0.9-1.7; the ratio-free estimator's MSE is 2.5 times lower at k=16k = 16 and 3131.
  • At a GRPO checkpoint, with its own optimizer state, the pattern is the same at large kk (9-10 vs. 1.4 and 3.9 at k=16k = 16 and 3131; MSE 1.3 times lower for the ratio-free estimator), but at k=4k = 4-88 the importance-weighted estimator has the lower MSE because its variance is lower.

Section 4.2 now states exactly this: the ratio-free estimator wins once drift is large, not at every drift level.

W2. Experiments.

  • Removing the ratio vs. changing the clip. We ran the full 2x2 at mini-batch size 8 without KL. On full MATH-500: ratio + ratio clip 0.818, ratio + absolute clip 0.860, ratio-free + ratio clip 0.858, and ratio-free + absolute clip (RFPO) 0.869±0.0110.869 \pm 0.011. On coding (LCBv6, median over steps 240-270, base model 0.41), both cells with the ratio collapse: ratio + absolute clip to 0.00, and ratio + ratio clip (GRPO with KL; KL-free GRPO also collapses) to 0.12-0.23 by step 110. Both ratio-free cells stay stable, at 0.65 (ratio clip) and 0.66-0.67 (RFPO). Under heavy drift, the ratio is what breaks, whichever clip is used (Section 5.3, Table 3).
  • Repeated runs. RFPO and GRPO at mini-batch size 8 now have three independent runs each on math (0.869±0.0110.869 \pm 0.011 vs. 0.823±0.0120.823 \pm 0.012 on full MATH-500) and three on coding.
  • Baselines. GSPO reaches 0.702 on MATH-500 and 0.27 on LCBv6. DAPO (without dynamic sampling, which our trainer does not support) reaches 0.858 on MATH-500 and collapses on coding (0.09). Appendix F repeats the coding drift study on a second backbone, Phi-4-mini-reasoning.

Q1. How was δ=0.1\delta = 0.1 chosen, and how sensitive is it? From a small sweep, and the result is flat: δ\delta = 0.05 / 0.1 / 0.2 gives 0.874 / 0.869 (three runs, ±0.011\pm 0.011) / 0.858 on full MATH-500. For scale, δ\delta caps the change in a token's probability from the batch-start snapshot, so δ=0.1\delta = 0.1 is more permissive than ratio clipping with ϵ=0.2\epsilon = 0.2 for every token with πold<0.5\pi_{\text{old}} < 0.5.

Q2. What does "well-defined" mean? The word was imprecise, and we replaced it. With positive-only advantages, refitting on one's own samples shrinks the support toward a point mass; mean-zero advantages (M2) remove that degenerate fixed point. M2 does not make the loss bounded. What bounds each update is the probability floor (a token with πold≤δ\pi_{\text{old}} \le \delta cannot drop by more than δ\delta), M3 on already-likely tokens, and the bounded gradient of log⁡π\log\pi with respect to the logits.

Q3. Where does rollout drift come from? vLLM samples the rollouts, and the FSDP actor recomputes their log-probabilities. Both run in bf16 but with different kernels and reduction order, so the same weights give slightly different probabilities. On the same tokens we measure a mean ∣Δlog⁡p∣|\Delta\log p| of 0.014 (99th percentile 0.19) and a mean ∣Δp∣|\Delta p| of 0.0057 (99th percentile 0.06). The default anchor is recomputed by the training engine; bypass mode uses vLLM's log-probabilities and so mixes the two engines. Appendix A ("Engines and the rollout-training gap") now lists the engine settings.

Thank you again for finding the flaw in our variance argument; fixing it made the paper stronger. If the corrected analysis and the new measurements address your concerns, we would be grateful if you would reconsider your assessment. We are happy to answer further questions.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. The reviews agree on the value of separating policy drift from engine drift and on the ablations, and they raise the same main concerns: the variance analysis, the confound between removing the ratio and changing the clip, the statistics, the baselines, and the efficiency comparison. We ran the requested experiments and uploaded a revised PDF. The main changes:

1. The theory is corrected and now measured directly. Appendix B assumed that ratio clipping keeps every importance ratio within [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon]. That is wrong: a token with negative advantage whose probability rose keeps a ratio-weighted gradient, capped only by verl's dual clip at rt=3r_t = 3. We also overstated that RFPO's variance does not depend on drift. Section 4.2 now gives the argument without these claims and supports it with a direct measurement (Figure 1; protocol in Appendix H). From training checkpoints, we run up to 31 inner updates on one rollout batch and compare both gradient estimators with a fresh on-policy estimate of ∇J\nabla J. As the policy drifts, the squared bias of the importance-weighted estimator grows to 9-27 times the noise floor, while that of the ratio-free estimator stays near the floor. The ratio-free estimator has the lower MSE once drift is large (16 or more inner steps); at small drift the importance-weighted estimator is as good or better.

2. Full benchmarks and repeated runs. Every arm is now evaluated on all 500 MATH-500 problems and on AIME 2024/2025. Over three independent runs at mini-batch size 8 (Section 5.1, Table 2), RFPO reaches 0.869±0.0110.869 \pm 0.011, GRPO 0.823±0.0120.823 \pm 0.012, and strict on-policy GRPO 0.8740.874. We also re-ran the drift study of Table 1 with distinct data seeds (the original runs did not set one) and evaluated it the same way: as optimizer steps per batch grow from 1 to 8 to 32, GRPO goes from 0.879 to 0.861 to 0.824, while RFPO stays at 0.873-0.875; under rollout drift at 8 steps per batch, GRPO falls to 0.718 and RFPO stays at 0.866. One result changes: with rollout drift and 32 steps per batch, both methods degrade on math to about 0.79, and RFPO's advantage there is on coding only (0.57 vs. 0.29).

3. Removing the ratio vs. changing the clip. RFPO differs from GRPO in two ways: the objective (GRPO's importance-weighted gradient rtA∇log⁡πθr_t A \nabla\log\pi_\theta vs. RFPO's ratio-free A∇log⁡πθA \nabla\log\pi_\theta) and the trust region (PPO's clip of rtr_t to 1±0.21\pm 0.2 vs. RFPO's mask on the absolute change in token probability, δ=0.1\delta = 0.1). We crossed the two, at mini-batch size 8 and without KL (Section 5.3, Table 3):

objective \ trust regionratio clipabsolute clip
importance ratiomath 0.818; coding collapses (0.12-0.23)∗^*math 0.860; coding collapses (0.00)
ratio-freemath 0.858; coding 0.65math 0.869±0.0110.869 \pm 0.011; coding 0.66-0.67 (RFPO)

Math is full MATH-500 at the final step; coding is LCBv6, median over steps 240-270 (base model 0.41). ∗^*GRPO with its KL term; KL-free GRPO also collapses (0.214 in the paper). On math, either change closes most of the gap, and only GRPO's combination falls behind. On coding, where drift is heavier, the objective decides: both cells with the ratio collapse whichever clip they use, and both ratio-free cells stay stable. We also added two baselines at the same settings: DAPO (without dynamic sampling, which our trainer does not support) reaches 0.858 on math and collapses on coding (0.09); GSPO reaches 0.702 and 0.27.

4. Efficiency, with a correction. The coding comparison in Section 5.4 of the submission (now Section 5.5) was wrong: it compared a GRPO run on 2 nodes with an RFPO run on 1 node, and the RFPO figure it reported (~800 s/step) does not match our logs (667-724 s/step). The revision replaces it with all arms on one 8xH100 node at mini-batch size 8 (median seconds per step): on math, RFPO 367 and KL-free GRPO 514; on coding, RFPO 756-773 and GRPO with KL 1585-1691. Steps to a target accuracy: RFPO reaches 0.85 on the MATH-500 validation subset in 10-20 steps (strict on-policy GRPO: 30) and 0.60 on LCBv6 at step 25 (135); GRPO at mini-batch size 32 or below never reaches 0.60.

5. Smaller points, all addressed in the revision.

  • Section 4.3 states what negative samples do (they remove the degenerate fixed point of positive-only training) and what bounds each update.
  • The sensitivity to the clip threshold is flat: δ\delta = 0.05 / 0.1 / 0.2 gives 0.874 / 0.869 / 0.858 on full MATH-500.
  • We measured the gap between the rollout engine and the training engine on the same tokens: mean ∣Δlog⁡p∣=0.014|\Delta\log p| = 0.014, with a 99th percentile of 0.19.
  • We discuss LGPO (Zhang et al.), which reviewer Pk2C pointed out, in related work.

The individual replies below give the details for each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.