acceptodds
Under review as a conference paper at ICLR 2027

Closed-form predictive coding via hierarchical Gaussian filters

Abstract

Predictive coding (PC) offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks, yet to date, it remains slower, and performance degrades sharply as network depth increases. We trace both problems to a single simplification: current PC networks fix the precision matrix to the identity, discarding precision-weighted prediction errors that the variational derivation requires to be fast, local, and Bayesian. We close this gap by expressing predictive coding networks as deep hierarchical Gaussian filters (HGFs) and restore precision-weighted message passing, yielding dynamic uncertainty estimates and Hebbian-compatible update rules at every layer. The resulting networks can simultaneously learn activations, weights, and precisions under a single free-energy objective, with no global error signal, and resolve inference without requiring iterations or automatic differentiation. On FashionMNIST, our solution approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs, and competes with it on online, data efficiency, and concept-drift tasks. Beyond simple fully connected networks, it also matches backpropagation’s performance on convolutional architectures at various depths on CIFAR-10/100. We thus establish that closed-form variational inference with online precision learning provides a tractable foundation for deep, architecture-agnostic predictive coding networks without requiring iterative relaxation or global error signals.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
predictive coding, variational inference, bayesian ml, hierarchical gaussian filter, active inference, deep neural networks

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WuCnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work recasts predictive coding networks (PCNs) as hierarchical gaussian filters. This framing introduces the advantage of being able to predict not only the values of children nodes but also their precision. The authors claim this allow to address long standing issues with predictive coding: the speed of convergence of PCNs and their performance degradation with depth. They characterize the training procedure of their framework and In the experiment sections they are able to train depth 2 and 8 MLPs and various sizes of VGG (5-7-9) on FashionMNIST, CIFAR10 and CIFAR100. Their networks are trained with a closed form inference algorithm that allows them to reach backpropagation speed.

Critical strengths

As they author underlined, PC community individuated in precision tuning a valuable direction for scaling predictive coding. This work is a relevant step in this direction and shows how precision learning can be used to overcome limitations that arise when training PCNs.

Closed form inference-phase is also very relevant and allows to perform fast training and testing of PCNs on GPU.

Critical weaknesses

The authors state that this procedure could be the foundation for deep architecture agnostic predictive coding networks but the experiments are performed on a small set of architectures and on relatively shallow networks. PC community already reports results with ResNet18 and deeper VGG architectures (Towards the training of deep predictive coding neural networks Qi et al. 2025) and muPC trains up to 100+ layers MLP (although, as the authors correctly state, it does not propose convolutional architectures). I think the claim of architectural agnosticism would be better supported with results on different architectures and deeper networks.

While speed of convergence is a relevant quality, the PC community tends to target hardwares that are different from GPUs. It is not clear whether their closed form precision would fit (even at high level) these hardwares or if it would have limitations that standard PC networks do not present (e.g. iterative refinement of the inference phase, while being slower on a GPU, could allow asynchronous inference on local hardwares).

The paper does not contain a limitation section, I believe having one is good practice as it offers the readers a way to clearly understand unresolved issues.

Given these weaknesses I am inclined to decide Weak Rejection. Yet I am open to change my score if the authors can address these points and explore the limitations I underlined (ResNet18, depth limit)

Questions for authors

Is the method able to train a ResNet18 on CIFAR10?

Is there some depth at which performance start degradating?

What advantages/disadvantages would this method have on local hardware with respect to standard Predictive Coding Networks?

2 comments

We thank the reviewer for their constructive feedback and for recognising the value of precision learning and closed-form inference for predictive coding. We agree that the generalisation to complex architectures should be clearly established, and we have adapted our framework to ResNet18 and a GPT-style transformer to illustrate this point. For these kinds of networks, our method recovers the backpropagation gradient and performs on par with it. It can be compared to other existing local implementations of backpropagation, like Z-IL, but do not suffer from the same scheduling issues, which makes it a relevant candidate for implementation on local hardware.

We address your questions and comments below.

  1. Relation to Qi et al. We would like to add one nuance to your comparison. While Qi et al. do report results on ResNet18 and deeper VGGs, they reach near-BP performance by adding a hand-designed precision schedule, auxiliary neurons on every residual connection, a PC-specific batch normalisation, and 18-24 relaxation steps per batch. HGF reaches BP performance on ResNet18 without any of these.

  2. ResNet18. Please see the updated manuscript (and the code supplement); the new section “Other architectures” adds a ResNet18 on CIFAR-10 and a GPT-style transformer (6 blocks, 10.8M parameters) on Tiny Shakespeare. HGF matches BP on both (Table 1). Neither required any change to its update rules or architecture, and our ResNet18 omits batch normalisation in both arms. To our knowledge, these are the first convolutional, residual, and transformer networks of that size trained to BP-level performance by a predictive coding model without iterative inference.

  3. Depth. We explore and characterise the depth limit, which is referred to in section 4.1, the Limitations section and appendix A.4. In plain fully connected networks, at larger depths (>8), the HGF's scaling depends on its posterior precision: with the precision capped, it matches BP up to 64 hidden layers, otherwise it degrades. However, deeper architectures such as ResNet18, VGG9 and the GPT do not show this degradation because each of their learnt parts has at most one interior layer. We also further expand on this in the new Limitations section.

  4. Local hardware. We agree this is central to PC’s motivation and now discuss it in the Limitations section. We still consider this model to be a strong candidate for local hardware, and measures of uncertainty have direct applications to improve control and planning in robotic systems. One of the difficulties will be the use of rather complex update equations that would not easily transfer to any neuromorphic architecture.

We hope these additions address the reviewers' concerns, particularly the ResNet-18 and depth questions.

i thank the authors for the rebuttal. My concerns have been mostly addressed and I will raise my score to weak accept.

Reviewer_chtyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reformulates predictive coding using hierarchical Gaussian filters, replacing iterative activity inference with approximate closed-form updates and using precision-dependent local weight learning.

Critical strengths

The paper addresses a meaningful limitation of predictive coding that is to reduce iterative inference costs while retaining local learning. Incorporating precision into explicit update rules is an interesting direction.

Critical weaknesses
  1. The main text does not adequately substantiate the central efficiency claim. Line 085 highlights wall-clock efficiency as a benefit, but the only timing analysis is in Appendix A.5/Figure 4 (lines 792–802) and covers only small FashionMNIST MLPs in a latency-bound regime. Since efficiency is one of the contribution, its essential evidence should appear in the main text instead of appearing in the Appendix. Please include a structured computational-cost analysis supported by representative timing results. In addition, I did not find time-wall analysis for corresponding VGG architectures, and per-epoch cost alone does not establish time to a comparable accuracy. Speaking of the architecture, the experiments cover MLPs and VGG5/7/9 (lines 291–292, 319–323), leaving the architecture-agnostic claim in the abstract insufficiently validated. I am wondering whether architecture of transformer, e.g., ViT still works under your framework.

  2. The comparative evaluation and model-selection protocol leave the practical advantage unclear. Figure 2 presents baseline learning curves in the main text, but their numerical summary is relegated to Table 4 in the appendix (lines 837–846), which reports three-seed means without standard deviations. Please include a concise main-text comparison with mean ± standard deviation. The paper does include μPC (lines 281–297), so a modern PC comparison is present. However, the convolutional comparison omits it, and the online experiment explicitly reports HGF performing worse than μPC with T = H (lines 415–419). Section 5 compares neural learning-rule baselines rather than dedicated online or drift-adaptation methods, and some panels use different batch sizes across methods (lines 403–410). Moreover, learning rates and HGF volatility are selected using oracle test performance (lines 293–297, 408–410), and the VGG protocol reports best test accuracy over training (lines 782–787). Please use validation-based selection, compare methods under matched sample and update budgets, and include representative streaming or drift-adaptation baselines under clearly stated memory constraints. .

Questions for authors
  1. Computational efficiency: You highlight wall-clock efficiency as an improvement, but the only timing analysis is in the appendix and only includes FashionMNIST. Could you include a more structured computational-cost analysis and representative timing results in the main text? Please also report VGG time to a common accuracy under the same loss as the accuracy experiments.
  2. Architectural scope: How would the closed-form local update handle a ViT's attention and normalization operations while retaining the claimed locality and tractability? Please provide an explicit derivation or restrict the architecture-agnostic claim to the structures supported by the current analysis and experiments.
  3. Baselines and selection: How do the conclusions change when hyperparameters and checkpoints are selected without test-set access, and when compared with dedicated online/drift-adaptation methods at matched sample and memory budgets? Please move the key numerical baseline summary into the main text and include the standard deviation across the three seeds for Table 4 (lines 837–846).
3 comments

We thank the reviewer for their comments. We have updated several parts of the manuscript that should now be able to address the point raised.

  1. Efficiency. Our efficiency claim rests on the algorithm, not on a particular implementation: one HGF update uses the same matrix products as one BP update (a forward pass and one backward sweep), plus one more product for the precisions, while PC repeats its sweep for every inference step. Section 4.1 states the resulting per-epoch cost in the main text, with the full timings in Appendix A.7. Therefore, this solution will always be slightly slower, as one more variable per node is being traced, but it is not slower by an order of magnitude, as it is the case with iterative relaxation approaches. But importantly, this variable enables faster convergence, which would make our method the fastest under specific settings, as reported in the manuscript.

  2. ViT. Please see the updated manuscript and the supplementary code: the new section "Other architectures" adds a ResNet-18 and a GPT-style transformer, and the HGF matches BP on both (Table 1). The GPT contains the attention and LayerNorm operations the reviewer asks about; a ViT differs from it mainly in its patch embedding, which is a linear layer. Errors pass through attention, LayerNorm, and residual connections in closed form, and every weight is still updated only from its own child's prediction error and its parent's activity, so locality and tractability are kept with no change to the update rules.

  3. Baselines and numbers. The FashionMNIST numbers behind Figure 2 are in Table 4, which now also reports the standard deviations across seeds. μPC is absent from the convolutional comparison because its implementation does not support convolutional or attention layers, as stated in Section 4.2. We report that μPC with T = H outperforms the HGF in the online setting because that is the result; we do not claim the HGF wins every benchmark. Our claim is rather that a single model can do both: match BP on standard supervised benchmarks across multiple architectures, and adapt in online, low-data, and non-stationary settings.

  4. Batch sizes. Thank you for pointing this out. Every panel of Figure 3 now uses the same batch size for every method, with the alternative batch sizes in the appendix A.9. The conclusions hold in both settings.

  5. Drift baselines. Section 5 studies how a single parameter, the tonic volatility, trades plasticity against stability within a learning rule, on the benchmark suite and against the baselines of Song et al. (2024). It does not claim state-of-the-art continual learning, so dedicated drift-adaptation methods, which add replay buffers or regularisers on top of a learning rule, are outside its scope.

  6. Model selection. We use the same selection protocol as the works we compare against: best test accuracy over training, as in PCX for the VGGs, and the benchmark protocol of Song et al. (2024) for Section 5. It is applied identically to every method, including BP, so it favours none of them, and it is stated in every caption and protocol.

To summarise our contribution: to our knowledge, this is the first predictive coding model to train convolutional, residual and transformer networks to BP-level performance without iterative inference, with precisions inferred online rather than fixed or scheduled. The same model, through a single interpretable parameter, also adapts in online, low-data, and non-stationary settings.

We hope that our changes have the potential to address most of the concerns the reviewers expressed.

Reviewer_chtyOpenReview ↗

Response

Thank you for your detailed response, which has addressed most of my concerns. However, I still have reservations about the efficiency of the proposed approach. I will consider raising my score based on the other reviewers’ feedback and further discussion with the AC.

Thank you for your reply; we are glad we could address most of your concerns. We fully respect your decision to wait for the other reviewers and the discussion with the AC.

We would like to briefly clarify the paper's perspective on efficiency. A major obstacle to scaling predictive coding has been iterative inference: every weight update requires many relaxation steps, which makes PC networks substantially slower than BP. The HGF removes this loop entirely. Our efficiency claim is therefore that the HGF brings predictive coding close to BP’s cost, which is a large improvement over both canonical PC and current state-of-the-art PC methods. We do not aim to outperform BP itself in terms of wall clock time per pass, but the gradient we are using can make it more efficient and require fewer of these passes under some conditions. This can make the overall process faster.

This matters on standard GPUs, where it makes PC networks practical to implement at the scale of modern architectures (where it could then benefit from faster convergence), and, importantly, on local and neuromorphic hardware (which is now discussed in the Limitations section), where avoiding iterative inference also reduces the communication between units. We will make this framing explicit in the introduction so that the efficiency claim is read relative to predictive coding rather than to BP.

Reviewer_vxTjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose to reformulate predictive coding through hierarchical gaussian filters. The core argument is to finally inference over the posterior precisions at runtime. The paper runs experiments on fashionMNIST vs BP and muPC as well as on online learning, and concept drift .

The main contributions are competitive performance vs BP (albeit on fashionMNIST) using fixed points for the variational estimation of the activations without iterative relaxation, showing that online precision estimation potentially works, and connecting tonic precsion to stability-plasticity tradeoff in continual learning

Critical strengths
  • The paper gets very close to BP using a predictive coding learning algorithm without needing iterative relaxation for the iterations, meaning that its computational cost is similar to BP
  • the continual and online learning results are encouraging
  • Getting the online fixed point variational updates to seemingly work
Critical weaknesses
  • Mathematically, their fixed point updates assume that the other variables are fixed. However, deep networks are coupled such that this assuption does not actually hold. Therefore, their updates are not actually correct as a single step but need to be put together into a fixed point iteration algorithm for the posterior means and precisions.
  • I am confused by how the algorithm in appendix A works. Specifically, it is stated that the activations are initialized to the forward pass values, and then the algorithm computes first the predicted means as the forward pass then the prediction errors. But would this not just set all prediction errors to zero so there is no update?
  • it is unclear how the learned posterior update generalizes (Eq 8) to nodes with many parents (i.e. fully connected networks) since it is presented in the scalar case. Is a full rank covariance matrix estimated or just a diagonal approximation?
  • The results presented are on very small datasets and networks at this point -- CIFAR and FashionMNIST. My experience with PC networks is that they can be made to work decently well in these cases but then fail to match BP on larger networks /datasets and transformers.
Questions for authors
  • How the appendix A algorithm + the learned posterior in the vector case works
  • Could you please try to scale this up and see if it still works there. This would make the paper dramatically more impressive to me.
1 comment

We thank the reviewer for their very careful review. We agree that the mathematical exposition of our algorithm did not make this explicit, so we have updated all equations in the manuscript to express updates in vectorised form, which is what the code actually performs.

We address your comments and questions below:

  1. Algorithm 1. This was an error in how we wrote the algorithm, not in our implementation, we apologise for the confusion. The revised Algorithm 1 (Appendix A.3) now follows the code: the posterior and prediction error updates are interleaved, so each layer updates its posterior from the layer below, and only then forms its own prediction error.

  2. Vector case. The posterior is diagonal: each node carries one mean and one precision, and no covariance matrix is estimated. A node with several children sums their contributions. The updates are now written in vector form in Appendix A.1 (Eqs. 7–8).

  3. Single-step updates. You are right, and our wording overstated it. One sweep is a single closed-form step from the prediction, not the joint fixed point of the coupled layers. Section 3.2 now says so.

  4. Scaling up. Please see the updated manuscript: the new section "Other architectures" adds a ResNet18 on CIFAR-10 and a GPT-style transformer (6 blocks, 10.8M parameters) on Tiny Shakespeare. HGF matches BP on both (Table 1), with no change to its update rules. We also characterise its depth limit in plain fully connected networks (Appendix A.4 and the Limitations section).

We hope that these additions address the reviewers’ concerns and that the scale-up attempt and mathematical descriptions will appear more convincing.

Reviewer_eKrVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses two issues with Predictive Coding Networks: 1) "inference learning" incurs a training cost O(T)O(T) times higher than that of backpropagation (BP), and 2) PCN performance degrades sharply as model depth increases. The authors attribute these problems to the absence of precision-weighted predictive errors. To resolve this, they reformulate PCN as a HGF, transforming inference learning into a single top-down initialization pass followed by a single bottom-up posterior update—eliminating the need for iterative relaxation or automatic differentiation. Experimental results on CIFAR-10 and CIFAR-100 demonstrate that the proposed method outperforms standard PCN across various depths. The model also yields positive results when evaluated on other tasks, while maintaining a per-epoch computational cost comparable to that of BP.

Critical strengths
  1. The proposed closed-form method in PC is novel.
  2. The experimental tasks cover a comprehensive range.
Critical weaknesses
  1. The single-step inference approach is not new. Z-IL, proposed in the cited paper "Can the Brain Do Backpropagation?—Exact Implementation of Backpropagation in Predictive Coding Networks", already provides such an approach. Although the original paper describes Z-IL as performing inference learning followed by a weight update layer by layer, my understanding is that this procedure is equivalent to first performing a single bottom-up update sweep through the network, as in the present work, and then updating all weights afterward. The difference is that Z-IL uses standard PCN updates to minimize (F_t) rather than the closed-form updates proposed here. The activities obtained from this single inference sweep serve as local targets, and their differences from the initial feedforward predictions provide the prediction errors used to update the weights. A direct comparison with Z-IL would therefore help clarify the contribution of the proposed method, since Z-IL is also a variant of PCN and should have a computational cost similar to that of the proposed method.
  2. The paper lacks ablation studies that disentangle the sources of the improvement in deep networks. Does the single bottom-up update sweep improve performance by reducing the condition number of the Hessian, or does the improvement come from the proposed closed-form update itself? The current experiments do not allow us to distinguish between these explanations.
  3. The weight update is unclear to me. When updating aℓa_\ell, does the proposed method compute μℓ\mu_\ell using the initial state of layer ℓ+1\ell+1, or its updated state? Algorithm 1 seems to suggest the second, whereas Eq. (10) appears to suggest the first. Could the authors include an ablation comparing these two alternatives and report their respective results?
Questions for authors

See the Weaknesses above.

2 comments

We thank the reviewer for their careful review and for pointing out the similarity to Z-IL. We address your comments below.

  1. Z-IL. We agree that Z-IL has some similarities with our approach: starting from the forward pass, it also computes its updates in a single sweep, and we agree with the reviewer that its layer-specific updates are equivalent to updating all weights after that sweep, and we now discuss it in the manuscript. More specifically, Z-IL can be seen as a special case of our approach where precisions are high, and the precision ratio between layers is close to 1.0. In that case, the HGF reduces to Z-IL, and hence to BP, which support training of deep and complex architectures without dropping the locality. What differs is what the HGF adds beyond that limit. Z-IL’s equivalence to BP requires a step size of one, zero initial errors, and unit precision; in the HGF, each node’s step is set by its inferred precision, which is a belief updated online. Because of this, the same model adapts in online, low-data, and drift settings through its tonic volatility (Section 5), which Z-IL cannot do without giving up its equivalence to BP. Additionally, Z-IL has so far been evaluated only on small MLPs and single weight updates; we train a ResNet-18 and a GPT to BP-level performance (Table 1).

  2. Source of the gain in deep networks. In our formulation, it is not possible to assess whether the performance gains come from either the closed-form update or the reduction in the condition number of the Hessian. Our interpretation is that the closed-form update recovers a different minimisation point than what is recovered by iterative relaxation, where precision acts as a per-layer plasticity control that can accelerate learning in some cases (faster saddle-to-saddle dynamics). This effect can compound with depth and explain performance degradation observed, e.g., at 128 layers (see our depth study in appendix A.4), but this is under control if we prevent precision overflow, as reported. Where precision cannot act, as in every weight of our ResNet-18, the HGF simply reproduces BP's update exactly.

  3. Which state the weight update uses. This point was not clear because of an error in the algorithm. We have updated the algorithm and produced the vectorised version of all update equations for clarity. The weight update uses the updated posterior state of layer ℓ+1, after the sweep (similar to a prospective configuration step). Using the predicted value of the layer above instead would bring us back to standard backpropagation, which is what happens when the parent precision is high: the updated posterior barely moves away from the prediction and the weights use this value as reference. Whether this configuration or the lower precision one performs best depends on the task: online learning and concept drift get an advantage with the low precision, whereas deep and complex structures train better when simply recovering the backpropagation gradient. One of the advantages of our method is to be able to express both through a precision parameter. Please refer to the revised Algorithm 1 (Appendix A.3) and Appendix A.1, as well as our reply to reviewer vxTJ.

To our knowledge, this is the first predictive coding model to train convolutional, residual, and transformer networks to BP-level accuracy without iterative inference, with precisions inferred online rather than fixed; the same model, through a single interpretable parameter, the tonic volatility, trades plasticity against stability in online, low-data, and drift settings. The updated manuscript adds a ResNet-18 and a GPT-style transformer (Section 4.2 and Table 1), and characterises how performance scales with depth (Appendix A.4).

We hope these additions, together with the discussion of Z-IL, address the reviewer's concerns.

Thanks for your response. It addressed my concerns, so I will increase my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.