How Much of the Natural Gradient's Value Do Cheap Approximations Retain?

Cheap approximations of natural gradient descent (NGD), such as K-FAC, are justified by the argument that a direction close to the natural gradient inherits its benefits. We test this argument on small dense and convolutional networks whose exact Fisher is computable and strongly ill-conditioned. Every update is normalised to unit Fisher norm, damping and horizon are shared, and each method tunes its own step size. We score each method by its retention: the fraction of exact NGD's improvement over gradient descent that it achieves. Under this protocol, on one-hidden-layer networks with at most $106$ parameters, K-FAC, EK-FAC and the exact diagonal retain $41\%$–$71\%$ of the training-loss improvement, and Kronecker approximations beat diagonal ones in every configuration tested. The angle to exact NGD ranks the approximations, but a GD-to-NGD interpolation at the same angle predicts only a third to a tenth of their retention. At a $60^{\circ }$ Euclidean angle, error in the lowest-curvature decile keeps all of NGD's benefit, while error in the highest-curvature decile makes the update far worse than gradient descent; at a matched damped-Fisher angle this contrast largely disappears, so it mainly reflects the Fisher-norm step budget. On dense networks, most retention survives at exact NGD's step size. These results point to relative curvature fidelity as a promising design target. In this regime, the exact layer-wise block retains $115\%$, so much higher fidelity is achievable, at higher cost.