- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Ratio-Free Policy Optimization (RFPO) to address training instability caused by policy drift and rollout drift in LLM reinforcement learning. RFPO replaces importance-ratio-based optimization with an advantage-weighted maximum likelihood objective and introduces three stabilizing mechanisms: z-score advantages, negative samples, and absolute probability clipping. Experiments on mathematical reasoning and coding tasks show that RFPO is more stable than GRPO under increasing policy drift and achieves better training efficiency.
Critical strengths
- The paper addresses an important practical problem in LLM reinforcement learning, particularly the instability caused by policy drift and differences between rollout and training engines.
- RFPO offers a simple optimization approach without importance ratios or a reference model, potentially reducing implementation complexity and computational cost.
- The experimental results show promising improvements over GRPO, particularly on coding tasks. The ablation studies also provide useful evidence for the proposed stabilization mechanisms.
Critical weaknesses
- The variance analysis in Appendix B assumes that GRPO clipping keeps importance ratios within , which standard clipping does not guarantee. The authors should correct this assumption and directly measure gradient bias, variance, and MSE to support their main theoretical claim.
- Section 4.3 claims that negative samples make the RFPO objective well-defined. However, the advantage-weighted log-likelihood can still be unbounded when negative advantages are included. The authors should clarify the conditions under which the objective remains bounded.
- Section 4.3 uses token-level probability clipping to control policy drift, but Appendix B acknowledges that this does not provide a meaningful sequence-level bound for long responses. The authors should provide stronger theoretical justification or experiments examining sequence-level divergence.
- RFPO removes importance ratios and introduces absolute clipping simultaneously. The current experiments cannot clearly identify which change drives the improvement. A controlled comparison of ratio-based and ratio-free objectives under both clipping strategies is needed.
- The experiments lack comparisons with stronger alternatives such as GSPO and DAPO. Moreover, the LiveCodeBench results mainly use a single random seed, and MATH-500 validation uses only 100 questions. More competitive baselines and repeated evaluations are needed.
- The efficiency comparison in Section 5.4 reports a 1.3× speedup on mathematical reasoning, but compares RFPO with mini-batch size 8 against GRPO with mini-batch size 256. A matched mini-batch comparison is needed to establish the actual speedup.
Questions for authors
see weeknesses
Thank you for the specific and actionable review. Our general response summarizes the new experiments.
W1. GRPO's ratio is not bounded by clipping; measure bias, variance and MSE directly. You are right, and Appendix B is corrected. A token with negative advantage whose probability rose keeps a ratio-weighted gradient, capped only by verl's dual clip at rt=3, which fires on 9.6×10−5 of tokens at mini-batch size 8 and never at 256. As you suggested, we measured both estimators directly. From training checkpoints, we run k=0,…,31 inner updates on one rollout batch and compare each estimator with a fresh on-policy estimate of ∇J. The squared bias of the importance-weighted estimator grows to 9-27 times the noise floor at large k, while that of the ratio-free estimator stays at 0.9-1.7 (rising to 3.9 at k=31 at the GRPO checkpoint). The ratio-free estimator's MSE is 1.3-2.5 times lower once k≥16, but at k=4-8 the importance-weighted estimator can be better. Section 4.2 and Figure 1 now state this (protocol: Appendix H).
W2. The objective with negative advantages is unbounded. Agreed; "well-defined" was the wrong word. M2 removes the degenerate fixed point of positive-only refitting, but it does not bound the loss. Each update is bounded by the probability floor (a token with πold≤δ cannot fall by more than δ), by M3 on already-likely tokens, and by the bounded gradient of logπ with respect to the logits. Section 4.3 (M2 paragraph) now states these conditions.
W3. Per-token clipping gives no sequence-level bound. Agreed, and we measured it. After 31 inner updates, the mean absolute summed log-ratio per sequence, ∣∑tlog(πθ/πold)∣, is 3.3 at the base model, 8.6 at an RFPO checkpoint and 11.7 at a GRPO checkpoint (99th percentile up to 81), while per-token ratios stay within about 1% of 1. So the per-token cap does not bound drift at the sequence level, as you say, and we now describe M3 as a local regularizer. A sequence- or prefix-level trust region on the ratio-free surrogate, in the style of GSPO or MinPRO, is the natural extension (Section 4.3, M3 paragraph; Appendix D).
W4. Removing the ratio vs. changing the clip. We ran the full 2x2 at mini-batch size 8 without KL. On full MATH-500: ratio + ratio clip 0.818, ratio + absolute clip 0.860, ratio-free + ratio clip 0.858, and ratio-free + absolute clip (RFPO) 0.869±0.011. On coding (LCBv6, median over steps 240-270), both cells with the ratio collapse: ratio + absolute clip to 0.00, and ratio + ratio clip (GRPO with KL; KL-free GRPO also collapses) to 0.12-0.23 by step 110. Both ratio-free cells stay stable, at 0.65 and 0.66-0.67. Under heavy drift, the ratio is what breaks, whichever clip is used.
W5. Stronger baselines and repeated evaluations. GSPO reaches 0.702 on MATH-500 and 0.27 on LCBv6. DAPO (without dynamic sampling, which our trainer does not support) reaches 0.858 on MATH-500 and collapses to 0.09 on coding. Every arm is now evaluated on all 500 MATH-500 problems and on AIME 2024/2025, and RFPO and GRPO at mini-batch size 8 have three independent runs each on math (0.869±0.011 vs. 0.823±0.012) and on coding (GRPO collapses in all three).
W6. Efficiency at matched mini-batch size. Agreed, and checking this revealed an error in our own comparison: the coding numbers in Section 5.4 of the submission (now Section 5.5) set a GRPO run on 2 nodes against an RFPO run on 1 node. On one 8xH100 node at mini-batch size 8 (median seconds per step), a math step takes 367 s for RFPO and 514 s for KL-free GRPO, and a coding step 756-773 s for RFPO and 1585-1691 s for GRPO with KL. (GRPO with KL runs FSDP2; wall-clock includes generation.) At matched mini-batch size 8, RFPO is thus 1.4 times faster per step than KL-free GRPO (367 vs. 514 s) and 1.6 times faster than GRPO with KL (586 s); the revision and the abstract now state the speed-up this way.
We have tried to address each of your points with the experiments you suggested. If they resolve your concerns, we would be grateful if you would consider updating your score.