- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper propose RFPO, a method that uses trained critic as reward model and critic model at the same time. This method first trains a critic model in standard PPO. Then, it freezes the critic model, applies debiase term to the value prediction, and use the debiased value prediction as reward of a rollout. At the same time, the freezed critic model also predict value for each token and use GAE to calculate final advantage. Experiments shows that RFPO, which does not require reward, has comparable performance with PPO.
Critical strengths
- This paper finds and proposes an interesting observation that well-trained critic model is strong enough to classify whether a partial rollout will result in correct or wrong answer.
- The analysis use various metrics and settings to prove the performance of the critic model.
Critical weaknesses
- Unconventional Terminology: This manuscript frequently uses confusing phrases. For example, the titles of Figure 2 ("The critic is manufactured over the run / by a run that was doing something else") is uninformative and grammatically awkward. Similarly, the use of the clinical A B-testing term "arm" to refer to experimental configurations is misleading in the RL literature.
- Motivation and Experiment Mismatch: The introduction says that “The question matters because the reward is where these pipelines run out of resources ... a number attached to every rollout—at a price that does not scale with the amount of RL one would like to run”. It seems the main motivation is about reduce the requirement of annotation? However, all the experiments are conducted on math data that already has golden answer. There is no evidence that proves RFPO can generalize to unseen data or data without any label.
Questions for authors
- The initial policy and critic is a special checkpoint from a ppo training and all the later experiments are based on this checkpoint. Why chooses this checkpoint and why this checkpoint is representative? Can the result and observation generalize to different experiment settings?
- Is there any experiments that can show RFPO generalize to unseen data or data without any label?