acceptodds
Under review as a conference paper at ICLR 2027

Natural Gradient Learning in Predictive Coding through Lateral Circuits

Abstract

Predictive coding (PC) trains neural networks through local energy-minimization dynamics without depending on a global error signal that backpropagation (BP) requires. Although PC is a promising alternative algorithm for efficient learning, the computational cost of inference dynamics currently outweighs the resulting learning benefits in practice. In this work, we take inspiration from optimization theory to extract more learning benefit from each inference phase. We introduce Natural PC, which augments standard PC with a lateral circuit, enabling exact and efficient natural gradient learning locally in space and time. Our lateral circuit learns the activity correlations that determine the Fisher information matrix and implicitly computes its inverse, avoiding expensive matrix inversion. Across all tested vision benchmarks, Natural PC achieves state of the art in terms of sample efficiency and accuracy compared to existing PC approaches and even surpassing BP trained with AdamW. Most notably, Natural PC outperforms BP on ImageNet reconstruction at a comparable computational cost, while using an order of magnitude fewer training samples. For the first time, our results establish PC as a practical alternative to BP.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Predictive Coding, Biologically Plausible Learning, Alternatives to Backpropagation, NeuroAI, Computational Neuroscience, Theoretical Neuroscience

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fH5JOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Predictive coding (PC) typically performs a gradient-based parameter update after a computationally expensive inference phase. The authors argue that conventional PC does not account for differences in curvature across parameters, leading to slow convergence. To address this issue, the paper proposes Natural PC, which rescales gradients using the inverse Fisher information matrix. To avoid expensive matrix inversion, the authors further introduce a biologically motivated lateral circuit to approximate the required matrix computations. The authors further show that natural gradient learning in PC coincides with Newton and Gauss-Newton optimization. Experimental results demonstrate that Natural PC achieves better sample efficiency and classification accuracy than existing PC approaches.

Critical strengths
  1. Predictive coding (PC) is a promising alternative learning algorithm to standard backpropagation (BP), and this paper addresses an important challenge in PC.

  2. The proposed method is intuitive and easy to understand. In particular, using a biologically plausible lateral circuit to approximate the Gram matrix is an interesting idea.

  3. The theoretical analysis connecting natural gradient learning in PC to Newton and Gauss–Newton optimization provides deeper insights into the proposed method.

  4. The experimental results are promising, and the proposed method demonstrates scalability to the large-scale ImageNet dataset.

Critical weaknesses
  1. The alignment between the lateral weights and the Gram matrix is only validated using a three-layer network, raising concerns about whether this property holds in deeper networks. It would be helpful to provide further validation using deeper architectures.

  2. The experimental comparisons are mainly limited to PC and iPC baselines. More comprehensive comparisons with other existing PC methods would strengthen the experimental evaluation.

  3. The authors are encouraged to discuss more related work on alternative learning algorithms to BP, such as local learning methods [1, 2]. This would help better position the proposed method in the broader literature and broaden the readership of the paper.

[1] Training Neural Networks with Local Error Signals, ICML 2019.

[2] Scaling Supervised Local Learning with Augmented Auxiliary Networks, ICLR 2024.

Questions for authors
  1. Is the issue of different curvature across parameters specific to the predictive coding learning rule, or is it a general challenge in gradient-based optimization? If the latter, why is this issue particularly important for PC?

  2. Could the authors include more PC baselines in Table 2 to provide a more comprehensive comparison?

  3. Could the authors compare Natural PC with other biologically inspired learning algorithms on the classification datasets?

2 comments

Thank you for your time and effort. In addition to our general response, we will address your specific critiques separately here:

  1. the Gram matrix is validated only using a three-layer network because, as we explain in the introduction, computing and inverting the Fisher at scale is intractable, which is the reason why we implement natural gradient learning through a lateral circuit.

  2. Although we follow the procedure for "Benchmarking Predictive Coding Networks" (Pinchetti, 2025), we acknowledge this limitation and added 3 more PC baselines in all benchmarks. As you can see in the revised benchmark section, which will surely be to your liking, these additional baselines strengthen our results.

  3. Our contributions specifically target the PC community, and have therefore framed it as such. We think discussing other frameworks would make the paper more convoluted, but thank the reviewer for their suggestion. We do not view the absence of other local learning rules a critical weakness.

To address your questions:

  1. The learning geometry is highly important in other methods, this is the principal reason why different optimizers such as Adam(W), Muon, and SOAP were created, as they attempt to learn within the approximately correct geometry. Because the 'correct' information geometry is usually intractable in large networks (the Fisher is approximated by the Identity matrix in backpropagation for example), our theoretical results show that the correct natural geometry is much more tractable in local learning settings, including PC, and we show that this issue is particularly suited to be solved within PC.

  2. Yes, thank you for the suggestion. We now compare with four BP methods trained with different optimizers, and 5 PC methods which cover the most important and recent methods in the field. As you will see in the revised benchmarking section, these baselines further strengthen our results.

  3. We have, as suggested by the other reviewers, narrowed our scope to focus solely on the image reconstruction problems. Therefore, we hope the reviewer accepts that this is deferred to future work.

Thank you for your detailed responses. I maintain my original positive score and vote for the acceptance of this paper.

Reviewer_EWngOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work presents a new way of computing natural gradients using predictive coding networks. Interestingly, this happens using only local local information: they show that this can be achieved by simply adding recurrent weights at every layer of the model. This is followed by an experimental section that shows the performance of the method on multiple image generation tasks.

Critical strengths

The lateral-circuit mechanism is an elegant, biologically motivated implementation of a known second-order method, and it is well explained in the theory/methods section. Also, some speedups look substantial: this could have an impact in the PC community as people working on generative models could test them easily.

Critical weaknesses

The weaknesses are in the experimental section, that do not allow me to properly understand whether there is an actual advantage or not: the models are too small to be meaningful generators of validation images of CIFAR10-kind of models. I'm also skeptical of the reconstructions themselves: How come they are not shown, to allow a qualitative inspection of the performance?

One idea could be to replicate some associative memory experiments (Salvatori et al., Neurips 2021): They would allow you to test how well your PC generative model is able to fit a dataset of natural images, and retrive them. The performance on associative memory tasks on small datasets are already very good in the original work, but the model trains (and evaluates!) quite slowly: this is where I would find interesting to see a proper advantage, same performance, faster/cheaper training and evaluation. You have already done something similar by following Pinchetti et al. (ICLR 2025). Bonus points: Does your method also improve on the capacity? This would be a nice result, but I'm not sure it would hold.

Other comments on the presentation, that makes some overclaims that should be avoided:

  • “Weight updates” is the headline metric, but per-update cost differs by orders of magnitude. The interneuron relaxation alone is 2000 steps, more than inference.

  • The comparison with The Kerjan et al. in Sec. 4.2 is unfair, as they train a classifier, while you train a reconstruction model. Claiming the gap to BP is “6 times smaller” compares training times on different tasks.

Questions for authors

How come in the FLOP table (Tab. 4), iPC is 10–30× cheaper than Natural PC?

5 comments

Thank you for your time and effort. In response to your concerns and as explained in the general comment, we added four more PC baselines to all of the experiments, including the ImageNet runs, which show that none of the other PC methods train at this scale. We also included FLOPs in all experiments to show the advantages of our method. We invite the reviewer to assess the new benchmark section as we are sure it will be to your liking.

On the other comments on the presentation, indeed weight updates is the headline metric, while we generalize our findings by showing the FLOPs (including the settling phase) required to reach the reconstruction errors, which additionally strengthen the compute efficiency claim. We added that the interneuron relaxation is done by a single cheap solve instead of a settling phase, and thus the lateral weight do not add a substantial computational overhead. We also acknowledge the Kerjan statement was unfair, both from a hardware and learning setting perspective, and removed this statement.

To your only question, iPC employs much lower FLOPs because iPC interleaves the weight updates in the settling step, and because it already converges at a much higher loss.

The new experimental section is of my liking, indeed, but I still believe that something is missing: (1) the task picked makes it very hard to understand the actual performance of the model, and I believe it could be very easily evaluated by properly showing the generated images (in an associative memory way); (2) Training a small, 3 layer model in Imagenet is not indicative of performance or scalability of the algorithm, as the complexity of the dataset here play little to no role (you could replicate the experiments with a dataset of HQ 1000x1000 images, and you would still get similar performance).

I actually have a positive opinion of the paper, and I can see myself using this method in the future. It is just that the experimental section, in its current status, leaves many questions unanswered: I'm curious how it would perform in a MCPC setting, in an associative memory, or image generation at large; I'd like to understand if this method pays off in discriminative tasks, etc.., and I feel it should be the author's job to show at least some of the applications of this method in a research paper, instead of relying on a single set of experiments (3 layer, generative direction) and have the reader guess the performance on the rest.

AuthorsOpenReview ↗

Rebuttal #2

It appears you have misread the experimental section: we do not train a three-layer MLP on ImageNet. Rather, as we state in the paper, “We implement a top-down VGG-style decoder that maps the images through nine 3x3 convolutional layers with an 8,192-dimensional bottleneck using phi=tanh, comprising 212.6M parameters. To the best of our knowledge, this network is roughly four times larger than the current largest PC model ever successfully trained.” Moreover, following the benchmarking protocol of Pinchetti et al. (2025), we evaluate both reconstruction and classification, and test both at scale (width scaling, Table 3; depth scaling, Table 4; classification, Table 7; classification scaling, Table 8). Regarding your concerns about the reconstruction performance, we report an MSE of about 0.002, computed on ImageNet mean/std-normalised inputs; undoing this normalisation (std ≈ 0.225) gives a pixel-space MSE of roughly 1×10⁻⁴, which corresponds to a PSNR of about 40 dB, so the reconstructions are essentially visually indistinguishable from the original images.

Therefore, we believe we have addressed the critical weakness in your review. We would like to ask the reviewer, if he agrees with our rebuttal, and as mentioned is positively dispositioned to our paper and intends to use the method himself, to update the score accordingly.

(bonus: we also replicated the suggested paper on associative memory experiments (Salvatori et al., Neurips 2021) where NPC achieved better (or qual 100%) retrieval performance across all strip and model sizes at 3% to 10% of FLOPS compared to the approach proposed by Salvatori et al. The full grid is still running but we wanted to give you the results in time and will add the full results to the appendix. Thanks for the suggestion!)

(bonus #2: we attempted to apply MCPC (Oliviers et al., 2024) on top of our experiments, but it does not scale to architectures of this size without very substantial compute. MCPC’s mixing time, and hence its compute, grows with the dimensionality of the latent space. The largest MCPC model in Pinchetti et al. (2025) has fewer than 800 latent units (30–256–256–256 on MNIST), yet already requires 1,000 Langevin steps per training iteration and 10,000 steps per generated sample, compared to T = 20 inference steps for their convolutional PC autoencoder. Our network has on the order of 10⁶ latent variables, roughly 1,250× more. Even assuming mixing time scales only linearly with dimension, a single training iteration would therefore require on the order of 10⁶ Langevin steps, about 6×10⁴ times more than the T = 20 steps of a comparable PC autoencoder, and generating a single sample would require on the order of 10⁷ steps.)

Reviewer_cyryOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Natural Predictive Coding (Natural PC), which incorporates lateral circuits that learn presynaptic activity correlations to implement natural-gradient weight updates. By improving the learning benefit obtained from each costly inference phase, the method aims to address a major computational bottleneck of predictive coding and reports substantial improvements in convergence speed, sample efficiency, and reconstruction accuracy over existing PC methods. The proposed mechanism is supported by a principled mathematical formulation and extensive empirical analyses, including large-scale ImageNet reconstruction experiments.

However, my primary concern is the substantial mismatch between the scope of the empirical evidence and the paper's central claims, particularly regarding PC's practical competitiveness with backpropagation. Despite recognizing the technical contributions, my initial recommendation is Weak Reject. How the authors address this discrepancy will be important to my final assessment.

Critical strengths

1. Well-motivated optimization framework

The paper identifies an important computational bottleneck in predictive coding and applies a well-established optimization principle that is particularly suited to the underlying problem. Although natural-gradient optimization itself is not novel, implementing its components through lateral circuitry and integrating them into an effective PC learning algorithm constitutes a meaningful technical contribution. The authors also provide systematic analyses supporting the proposed mechanism.

2. Substantial improvements over standard PC

Natural PC demonstrates substantial improvements in learning efficiency and reconstruction performance compared to standard PC, suggesting that the proposed approach effectively addresses an important limitation of existing PC algorithms. In particular, successfully training a generative PC model with tens of millions of parameters on ImageNet-1k represents a noteworthy empirical achievement.

Critical weaknesses

1. Overclaiming beyond the scope of empirical evidence

While the improvements in optimization efficiency for generative PC are meaningful and potentially publishable in their own right, the broader claims in the Abstract and Introduction, particularly that Natural PC "establishes PC as a practical alternative to BP," are not sufficiently supported by the presented evidence. Substantial limitations remain in deep discriminative learning, as the authors themselves acknowledge in the Discussion. Moreover, the ImageNet comparison involves different inference procedures and considerable computational overhead, further complicating the claim of practical competitiveness with BP.

I strongly encourage the authors to moderate these claims and align the framing with the demonstrated contributions. I consider this claim–evidence mismatch a major concern that directly affects my overall assessment, rather than merely a matter of presentation.

Questions for authors

Q1. Fairness of the optimization baseline

According to Appendix H (p. 19), the BP baseline differentiates the input reconstruction MSE through 1,000 unrolled inference steps, whereas the PC methods update their weights using the equilibrium prediction-error energy. Although the methods share the same inference dynamics, they appear to differ not only in their weight-update rules but also in their optimization objectives.

Could the authors clarify whether this constitutes a fair comparison for evaluating the proposed natural-gradient learning rule? A controlled comparison using the same optimization objective and inferred activities, but different weight optimizers, would help isolate the contribution of Natural PC.

Q2. Comparability of the ImageNet reconstruction results

In the ImageNet experiments, Natural PC obtains latent representations through iterative inference, whereas the BP autoencoder uses a feedforward encoder, despite sharing the same decoder architecture.

Given this difference, to what extent can the lower reconstruction MSE of Natural PC be attributed to the proposed weight-learning algorithm rather than the use of iterative latent inference? Could the authors clarify how this comparison supports the claimed advantage over BP, ideally with a comparison under matched test-time inference conditions or computational budgets?

Q3. Hyperparameter tuning and convergence of the ImageNet BP baseline

Appendix H reports learning-rate sweeps for the smaller benchmarks, but the extent of hyperparameter tuning for the ImageNet BP baseline is less clear.

I understand that extensive hyperparameter sweeps and multiple-seed experiments can be computationally expensive at this scale. Nevertheless, Table 8 in Appendix L shows that the BP autoencoder's reconstruction MSE continues to decrease from 8.85 at epoch 9 to 8.62 at epoch 10, suggesting that its convergence may not have been fully established.

Could the authors clarify the hyperparameter selection procedure for the ImageNet BP baseline and provide further justification that the reported comparison adequately reflects its achievable performance?

4 comments

Thank you for your thorough and critical comments. As we explain in the general comment, we fully agree that our framing was too strong, and we limited all of our claims to image reconstruction only, which is the scope that the empirical evidence supports. We still include our small-scale classification results as they are promising, but no longer claim that we establish PC as a practical alternative to BP. We rephrased such that the core claim regards advancing the PC field. With the only critical weakness on the scope of empirical evidence that we now have fully resolved, we hope you can revisit the review.

To address your specific questions:

  1. We agree with the difficulty of finding a fair comparison between PC and BP. As a result of your question, we have limited the comparison on the smaller datasets to solely compare different PC methods, as indeed adding an inference phase to BP was an attempt to make the comparison fair, perhaps wrongly so. The comparison with BP is thereby limited to the ImageNet-1k reconstruction, where BP is trained with an autoencoder without inference phase. This, to us, would be the fairest comparison.

  2. In the revised ImageNet experiments we now include, the BP autoencoder trained with AdamW obtains a lower reconstruction error than Natural PC. However, while not reaching the the lowest converged error, our method reaches the same target errors using significantly less samples and compute. In response to your comment and to avoid further confusion, we include a figure on the ImageNet runs that clearly shows this distinction. Moreover, in the text, we now clearly name only improvements in sample and compute efficiency, not reconstruction MSE.

  3. In light of your question, we have increased to four BP baselines with different optimizers (SGD, AdamW, Muon, SOAP) and have increased and reran the hyperparameter tuning for all these methods. Additionally, we ran the BP baselines for significantly longer time (10 to 25 epochs), and we invite the reviewer to reassess this benchmark. Your suspicion is true, the convergence was not fully established. We also invite the reviewer to consider the theoretical nicety of our method, which does not require a tuning of optimizers nor learning rate.

Thank you for the detailed response and substantial revisions! I appreciate that the authors have narrowed the claims and addressed most of my concerns.

One remaining point I would encourage the authors to clarify is the interpretation of the efficiency advantage over BP. While the new experiments provide convincing evidence of improved sample and compute efficiency relative to the tested BP baselines, this should not necessarily be interpreted as an intrinsic advantage of PC over BP. Natural-gradient preconditioning can also be applied to BP, as demonstrated by methods such as K-FAC. Although Muon and SOAP are valuable additional baselines, neither directly evaluates Fisher-based natural-gradient optimization in BP. Similarly, curvature-aware learning-rate adaptation is not exclusive to PC, even if the proposed implementation is particularly well suited to its learning structure. A more direct test would be to apply the proposed presynaptic-correlation preconditioning to BP itself. However, I understand that additional large-scale experiments may not be feasible at this stage. At minimum, I would appreciate an explicit acknowledgment that the relative efficiency of Natural PC versus natural-gradient-preconditioned BP remains an open question.

Nevertheless, I recognize the meaningful technical contribution to predictive coding research and the strong improvements over existing PC methods. Given the substantial revisions and the much better alignment between the claims and evidence, I am inclined to raise my rating.

AuthorsOpenReview ↗

Rebuttal #2

Thank you for your thoughtful and constructive feedback, we really appreciate the reviewer helping make the paper more proper. Sometimes us authors get a little too excited from the results, thanks for the useful and necessary hedging perspectives.

In light of the reviewer's feedback we have revised the conclusion of the manuscript, we wrote the following changes:

  • We added "While the efficiency of Natural PC relative to natural-gradient-preconditioned BP, which is tractable only through approximations (Martens & Grosse, 2015; Grosse & Martens, 2016; Martens, 2020), remains an open question, our results suggest that natural gradient learning seems particularly well suited to PC."
  • We revised the final sentence to "Although curvature-aware learning rates are not exclusive to PC (LeCun et al., 1993; Schaul et al., 2013; Martens & Grosse, 2015), existing PC models and the tested BP baselines require specifically chosen optimizers and learning rates, whereas both emerge in Natural PC from the network's dynamics."
  • "BP methods" to "tested BP baselines"

Indeed, the full K-FAC tuning will require some time and compute infeasible before the discussion deadline, but we will make sure to include K-FAC in our camera-ready version with the appropriate interpretation.

Thanks again!

(for reference the full conclusion now reads:

Standard predictive coding wastes expensive inference by taking a single Euclidean step over the energy landscape, causing networks to require many inference phases before weights fit the equilibrium activities. Natural PC replaces the Euclidean learning step with a natural gradient step by adding a lateral circuit of interneurons that encode and implicitly invert the presynaptic correlations that define the natural gradient. We thereby improve the trade-off between inference and learning and show that biologically inspired local computation can compete with the tested BP baselines in both sample efficiency and computational cost in image reconstruction tasks. While the efficiency of Natural PC relative to natural-gradient-preconditioned BP, which is tractable only through approximations (Martens & Grosse, 2015; Grosse & Martens, 2016; Martens, 2020), remains an open question, our results suggest that natural gradient learning seems particularly well suited to PC. Beyond improving performance, the lateral circuit determines both the geometry and learning rate of weight updates. Although curvature-aware learning rates are not exclusive to PC (LeCun et al., 1993; Schaul et al., 2013; Martens & Grosse, 2015), existing PC models and the tested BP baselines require specifically chosen optimizers and learning rates, whereas both emerge in Natural PC from the network's dynamics.)

Thanks for the clarification. This fully addresses my remaining concern. I appreciate the authors' efforts and have happily raised my rating. Best of luck!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to the reviewers

We thank the reviewers for their time and effort in providing critical feedback to the manuscript. All 3 reviewers had useful comments on the scope and empirical results that we fully acknowledge. To address your comments, we limited our claims to advancing the PC field and being competitive with BP in image reconstruction only. We also agree that the paper's main limitations were regarding the experimental setting, and have since worked on improving it. We now compare against 4 BP and 5 PC baselines in the reported benchmarks. Most importantly, we now report two orders of magnitude in sample and one order of magnitude in compute efficiency on imagenet-1k reconstruction. We, therefore, kindly ask the reviewers to read the revised benchmarks section, as we are sure it will be to your liking.

Nonetheless, while we fully acknowledge your concerns, we kindly ask the reviewers to recognize the impact this work will have on the PC community. Before this paper, the discussion of whether PC could demonstrate similar sample and compute efficiency to BP in any setting was unthinkable, let alone show improvements over BP, and even more so given the scale we trained on. Beyond performance, our theoretical contributions solve many of the empirical challenges the PC community faces, such as learning rate tuning, optimizers, robustness, and training generatively at scale. We now specifically name that classification at scale is for future work, but are convinced that the method and theory is foundational to the PC community and ask it to be recognized as such.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.