WONI: Weight-Space Orbit structured Natural Gradient for Scalable Policy Optimization via Inverse-Fisher Preconditioner

In deep RL with multi-objective and constrained settings, the natural gradient provides a principled way to handle the correlations among gradients within the policy geometry, while its computation, which grows with the number of gradients, remains a bottleneck. In this paper, we study curvature approximations for inverting the Fisher information matrix, which can be applied to a set of gradients to obtain multiple natural gradients efficiently. We propose a curvature approximation structured by the weight-space symmetry of the policy network and calibrated with its spectrum estimated from samples. Empirical results show that applying the proposed method in place of the conjugate gradient method used in various second-order algorithms achieves comparable or better performance with substantially less computation.