Revisiting Amari's $\alpha$-Divergence to improve clipped surrogate PPO

Proximal Policy Optimization (PPO) is one of the most popular policy gradient algorithms due to its widespread practicality and scalability. However, its performance is known to be highly sensitive to increasing the clipping threshold and policy network depth. To address this sensitivity, we introduce ALDIPPO, which replaces the clip with an $\alpha$-divergence penalty on the importance ratio. This makes the penalty stiffer for samples outside the $ϵ$ range, which PPO rejects by switching their gradient off, wasting the rollout budget spent to collect them. We devise several online update rules for $\alpha$ based on on-policy batch statistics, and show that ALDIPPO beats PPO from toy MiniGrid environments to standard MuJoCo tasks, and generalizes better in the Procgen benchmark.