acceptodds
Withdrawn from ICLR 2027

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instability and memory overhead. Even where a critic is trained, it is discarded once training ends, although it has learned to predict outcomes. We revisit this trend and show that a pretrained critic's ability to predict future outcomes can make it a valuable asset for efficient long-horizon reasoning. First, we find that instability in critic-based RL for long chain-of-thought reasoning is largely an optimization artifact: keeping policy updates small and low in variance restores stable convergence. Second, a well-pretrained critic estimates the posterior probability of eventual success from later trajectory states and unfinished prefixes. Its predictions therefore provide outcome-derived, dense, per-prefix learning signals that, during policy optimization, require neither completed rollouts, step-level annotations, nor external reward labels. Building on this insight, we introduce Reward-Free Policy Optimization (RFPO), which repurposes a single calibrated, frozen critic as a rollout-level reward, a value baseline for generalized advantage estimation, and a success forecaster for unfinished prefixes. We further show that binarizing the debiased score stops the policy from exploiting the critic's length bias. Binarized, RFPO matches supervised PPO without a single label in the training loop, while substantially cutting compute and memory overhead. This makes RFPO particularly well suited to long-horizon reasoning tasks, where outcomes arrive late and generation dominates cost: because rollouts can be rewarded before they finish, training no longer has to pay for waiting on every trajectory to complete. Our findings challenge the prevailing critic-free paradigm and establish critic-based, reward-free optimization as a scalable and computationally efficient path for LLM post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning for LLMs, LLM post-training, policy optimization, PPO, value function, critic, reward-free reinforcement learning, long chain-of-thought reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PivmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose RFPO, a method that uses trained critic as reward model and critic model at the same time. This method first trains a critic model in standard PPO. Then, it freezes the critic model, applies debiase term to the value prediction, and use the debiased value prediction as reward of a rollout. At the same time, the freezed critic model also predict value for each token and use GAE to calculate final advantage. Experiments shows that RFPO, which does not require reward, has comparable performance with PPO.

Critical strengths
  1. This paper finds and proposes an interesting observation that well-trained critic model is strong enough to classify whether a partial rollout will result in correct or wrong answer.
  2. The analysis use various metrics and settings to prove the performance of the critic model.
Critical weaknesses
  1. Unconventional Terminology: This manuscript frequently uses confusing phrases. For example, the titles of Figure 2 ("The critic is manufactured over the run / by a run that was doing something else") is uninformative and grammatically awkward. Similarly, the use of the clinical A B-testing term "arm" to refer to experimental configurations is misleading in the RL literature.
  2. Motivation and Experiment Mismatch: The introduction says that “The question matters because the reward is where these pipelines run out of resources ... a number attached to every rollout—at a price that does not scale with the amount of RL one would like to run”. It seems the main motivation is about reduce the requirement of annotation? However, all the experiments are conducted on math data that already has golden answer. There is no evidence that proves RFPO can generalize to unseen data or data without any label.
Questions for authors
  1. The initial policy and critic is a special checkpoint from a ppo training and all the later experiments are based on this checkpoint. Why chooses this checkpoint and why this checkpoint is representative? Can the result and observation generalize to different experiment settings?
  2. Is there any experiments that can show RFPO generalize to unseen data or data without any label?
Reviewer_iPeSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Reward-Free Policy Optimization (RFPO), which takes the critic from a supervised PPO run and uses it as the reward for further RL post-training. With γ=λ=1, the critic’s value at a prefix approximates the probability that the response will end up correct. After freezing the critic, the authors calibrate its terminal value once on a small set of verifier-labeled rollouts, using a length correction and a threshold. This binarized value then serves as the reward, including for rollouts truncated at the generation cap, while the same frozen critic serves as the GAE baseline. On Qwen3-4B, starting from the same PPO checkpoint, the paper reports that RFPO matches supervised PPO on math benchmarks without verifier labels in the RL loop, with about 28% lower step time and less memory.

Critical strengths
  1. The paper separates overall AUC, which rewards a critic for recognizing hard problems, from within-problem AUC, which tests whether it can rank attempts at the same problem.
  2. The calibration ablations show how raw and continuous critic rewards get exploited through length, in both directions.
Critical weaknesses
  1. "Reward-free" is overclaimed: The critic needs 563k verifier labels from a prior PPO run (more than PPO uses in the compared stage), and the RL stage appears to reuse those same labeled prompts. In effect it is a learned reward model.
  2. Related work: Many PRMs learn from outcomes only (e.g., Math-Shepherd) and forecast outcomes just as the critic does. Value- and advantage-based PRMs (PQM, PAV, OVM), reuse of the PPO critic (PPO-MCTS) and reward-model overoptimization are missing.
  3. Weak evaluation: Neither arm moves much off the initial policy (average 40.0 → 41.8 for PPO, 41.1 for RFPO). Under the 4,096-token cap, RFPO's pass@1 is essentially the initial policy's.
  4. Weak and missing baselines: The 4,096-cap PPO baseline gives zero reward to truncated rollouts, so it learns to shorten its responses. There are no GRPO/DAPO, label-free or outcome-reward-model baselines.
  5. Contribution (i) is weakly supported: The stability evidence comes from a confounded look back over 74 past runs. A single update per batch amounts to on-policy policy gradient with a pretrained critic, which is the fix VC-PPO already proposed.
  6. Presentation: The abstract is scattered, the contributions read like results, and key terms (e.g., explained variance, defined only in App. E) are used before they're defined. The prose is dense and hard to follow.
Questions for authors

Please see weaknesses.

Reviewer_2pBxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reward-Free Policy Optimization (RFPO), which reuses a critic obtained through supervised PPO as both a frozen reward model and an advantage baseline. A length-dependent correction and binary threshold convert critic scores into rewards for completed responses or unfinished prefixes. Experiments on Qwen3-4B investigate critic quality, reward exploitation, training stability, and the cost of continued optimization without verifier labels.

Critical strengths
  • The critic analysis distinguishes problem difficulty from attempt quality. Table 10 reports that within-problem AUC increases from 0.765 at critic step 100 to 0.837 at step 800. Table 11 reports AUC 0.844 for unfinished 4,096-token prefixes. These measurements support the claim that the critic learns useful information for comparing attempts at the same problem.

  • The proposed method saves computation while achieving better performance on some reasoning benchmarks.

Critical weaknesses
  • Almost all the step-time saving comes from removing the 151-second critic update (Table 15), but there is no verifier-supervised baseline with a frozen critic. This would separate the benefit of freezing from replacing verifier rewards. Likewise, the only group-relative variant uses the unstable continuous reward (Appendix F.2); a baseline using the same calibrated binary reward is missing. These controls are necessary to establish the benefit of using the critic as both reward and baseline.

  • This paper does not test learning on new unlabeled prompts or another task. Consequently, it remains unclear how broadly this supervision investment can be reused or whether RFPO improves the overall accuracy-cost tradeoff.

Questions for authors

How do verifier rewards with a frozen critic, and calibrated binary critic rewards with group-relative advantages, compare at matched compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.