- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces the VPPO algorithm to improve the reliability of PPO.
Critical strengths
A clear, rigorous paper on a well-motivated problem. The experimental result that VPPO outperforms PPO on some IsaacLab tasks is particularly interesting.
Critical weaknesses
The original TRPO solves a constrained optimization problem with approximated Hessian and backtracking line search. Despite being theoretically sound, its computational overhead is high as it estimates the Hessian of neural network parameters which is very large. PPO makes a trade-off: it is much less rigorous than TRPO, but simplifies the implementation by replacing those optimization-inspired methods with clipping. In the past few years, some algorithms have tried to make PPO as rigorous as TRPO by bringing back components like quadratic constraints and/or line search. This paper looks just like another work of this kind. The proposed algorithm is more complicated, slower, but does not offer better performance in most cases.
By the way, I believe that a detailed discussion on why VPPO outperforms PPO on IsaacLab tasks offers more insights and strengthens the paper, but the current empirical performance is not appealing enough to convince me to use VPPO in my problems.
Questions for authors
Please see Weaknesses.