Natural Gradient Learning in Predictive Coding through Lateral Circuits

Predictive coding (PC) trains neural networks through local energy-minimization dynamics without depending on a global error signal that backpropagation (BP) requires. Although PC is a promising alternative algorithm for efficient learning, the computational cost of inference dynamics currently outweighs the resulting learning benefits in practice. In this work, we take inspiration from optimization theory to extract more learning benefit from each inference phase. We introduce Natural PC, which augments standard PC with a lateral circuit, enabling exact and efficient natural gradient learning locally in space and time. Our lateral circuit learns the activity correlations that determine the Fisher information matrix and implicitly computes its inverse, avoiding expensive matrix inversion. Across all tested vision benchmarks, Natural PC achieves state of the art in terms of sample efficiency and accuracy compared to existing PC approaches and even surpassing BP trained with AdamW. Most notably, Natural PC outperforms BP on ImageNet reconstruction at a comparable computational cost, while using an order of magnitude fewer training samples. For the first time, our results establish PC as a practical alternative to BP.