- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Then propose an alternative to GPRO that gets rid of groups (hence less off-policy due to waiting for groups to fill), and add value functions back. They show then can train the value function by freezing attention, updating the critic 2x as often as the policy model, and using pretraining data. They also use skip-observation GAE to avoid predicting the value function of a noisy environment, and DIS clipping.
Critical strengths
Strong results and impressive that they did something that the field has been trying to work out how to do for a long time. Their method combines lots of sensible and interesting components which they have explained clearly and validated with impressive results.
Critical weaknesses
- No major weaknesses but it would be good to explain how the GRPO-based baselines were tuned.
Questions for authors
- Can you expand more on the difference in compute cost between SAO and GRPO.
2 comments
Thank you for your response. I will maintain my score - I am somewhat surprised by the low scores of the other reviewers and can't see a strong reason to reduce mine.
Thank you for acknowledge the contribution of our work.
The paper evaluates two GRPO-based baselines using the same backbone and asynchronous pipeline as SAO (Appendix B):
The training costs are summarized below: