- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Predictive coding (PC) typically performs a gradient-based parameter update after a computationally expensive inference phase. The authors argue that conventional PC does not account for differences in curvature across parameters, leading to slow convergence. To address this issue, the paper proposes Natural PC, which rescales gradients using the inverse Fisher information matrix. To avoid expensive matrix inversion, the authors further introduce a biologically motivated lateral circuit to approximate the required matrix computations. The authors further show that natural gradient learning in PC coincides with Newton and Gauss-Newton optimization. Experimental results demonstrate that Natural PC achieves better sample efficiency and classification accuracy than existing PC approaches.
Critical strengths
-
Predictive coding (PC) is a promising alternative learning algorithm to standard backpropagation (BP), and this paper addresses an important challenge in PC.
-
The proposed method is intuitive and easy to understand. In particular, using a biologically plausible lateral circuit to approximate the Gram matrix is an interesting idea.
-
The theoretical analysis connecting natural gradient learning in PC to Newton and Gauss–Newton optimization provides deeper insights into the proposed method.
-
The experimental results are promising, and the proposed method demonstrates scalability to the large-scale ImageNet dataset.
Critical weaknesses
-
The alignment between the lateral weights and the Gram matrix is only validated using a three-layer network, raising concerns about whether this property holds in deeper networks. It would be helpful to provide further validation using deeper architectures.
-
The experimental comparisons are mainly limited to PC and iPC baselines. More comprehensive comparisons with other existing PC methods would strengthen the experimental evaluation.
-
The authors are encouraged to discuss more related work on alternative learning algorithms to BP, such as local learning methods [1, 2]. This would help better position the proposed method in the broader literature and broaden the readership of the paper.
[1] Training Neural Networks with Local Error Signals, ICML 2019.
[2] Scaling Supervised Local Learning with Augmented Auxiliary Networks, ICLR 2024.
Questions for authors
-
Is the issue of different curvature across parameters specific to the predictive coding learning rule, or is it a general challenge in gradient-based optimization? If the latter, why is this issue particularly important for PC?
-
Could the authors include more PC baselines in Table 2 to provide a more comprehensive comparison?
-
Could the authors compare Natural PC with other biologically inspired learning algorithms on the classification datasets?
2 comments
Thank you for your detailed responses. I maintain my original positive score and vote for the acceptance of this paper.
Rebuttal
Thank you for your time and effort. In addition to our general response, we will address your specific critiques separately here:
the Gram matrix is validated only using a three-layer network because, as we explain in the introduction, computing and inverting the Fisher at scale is intractable, which is the reason why we implement natural gradient learning through a lateral circuit.
Although we follow the procedure for "Benchmarking Predictive Coding Networks" (Pinchetti, 2025), we acknowledge this limitation and added 3 more PC baselines in all benchmarks. As you can see in the revised benchmark section, which will surely be to your liking, these additional baselines strengthen our results.
Our contributions specifically target the PC community, and have therefore framed it as such. We think discussing other frameworks would make the paper more convoluted, but thank the reviewer for their suggestion. We do not view the absence of other local learning rules a critical weakness.
To address your questions:
The learning geometry is highly important in other methods, this is the principal reason why different optimizers such as Adam(W), Muon, and SOAP were created, as they attempt to learn within the approximately correct geometry. Because the 'correct' information geometry is usually intractable in large networks (the Fisher is approximated by the Identity matrix in backpropagation for example), our theoretical results show that the correct natural geometry is much more tractable in local learning settings, including PC, and we show that this issue is particularly suited to be solved within PC.
Yes, thank you for the suggestion. We now compare with four BP methods trained with different optimizers, and 5 PC methods which cover the most important and recent methods in the field. As you will see in the revised benchmarking section, these baselines further strengthen our results.
We have, as suggested by the other reviewers, narrowed our scope to focus solely on the image reconstruction problems. Therefore, we hope the reviewer accepts that this is deferred to future work.