- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work recasts predictive coding networks (PCNs) as hierarchical gaussian filters. This framing introduces the advantage of being able to predict not only the values of children nodes but also their precision. The authors claim this allow to address long standing issues with predictive coding: the speed of convergence of PCNs and their performance degradation with depth. They characterize the training procedure of their framework and In the experiment sections they are able to train depth 2 and 8 MLPs and various sizes of VGG (5-7-9) on FashionMNIST, CIFAR10 and CIFAR100. Their networks are trained with a closed form inference algorithm that allows them to reach backpropagation speed.
Critical strengths
As they author underlined, PC community individuated in precision tuning a valuable direction for scaling predictive coding. This work is a relevant step in this direction and shows how precision learning can be used to overcome limitations that arise when training PCNs.
Closed form inference-phase is also very relevant and allows to perform fast training and testing of PCNs on GPU.
Critical weaknesses
The authors state that this procedure could be the foundation for deep architecture agnostic predictive coding networks but the experiments are performed on a small set of architectures and on relatively shallow networks. PC community already reports results with ResNet18 and deeper VGG architectures (Towards the training of deep predictive coding neural networks Qi et al. 2025) and muPC trains up to 100+ layers MLP (although, as the authors correctly state, it does not propose convolutional architectures). I think the claim of architectural agnosticism would be better supported with results on different architectures and deeper networks.
While speed of convergence is a relevant quality, the PC community tends to target hardwares that are different from GPUs. It is not clear whether their closed form precision would fit (even at high level) these hardwares or if it would have limitations that standard PC networks do not present (e.g. iterative refinement of the inference phase, while being slower on a GPU, could allow asynchronous inference on local hardwares).
The paper does not contain a limitation section, I believe having one is good practice as it offers the readers a way to clearly understand unresolved issues.
Given these weaknesses I am inclined to decide Weak Rejection. Yet I am open to change my score if the authors can address these points and explore the limitations I underlined (ResNet18, depth limit)
Questions for authors
Is the method able to train a ResNet18 on CIFAR10?
Is there some depth at which performance start degradating?
What advantages/disadvantages would this method have on local hardware with respect to standard Predictive Coding Networks?
2 comments
i thank the authors for the rebuttal. My concerns have been mostly addressed and I will raise my score to weak accept.
We thank the reviewer for their constructive feedback and for recognising the value of precision learning and closed-form inference for predictive coding. We agree that the generalisation to complex architectures should be clearly established, and we have adapted our framework to ResNet18 and a GPT-style transformer to illustrate this point. For these kinds of networks, our method recovers the backpropagation gradient and performs on par with it. It can be compared to other existing local implementations of backpropagation, like Z-IL, but do not suffer from the same scheduling issues, which makes it a relevant candidate for implementation on local hardware.
We address your questions and comments below.
Relation to Qi et al. We would like to add one nuance to your comparison. While Qi et al. do report results on ResNet18 and deeper VGGs, they reach near-BP performance by adding a hand-designed precision schedule, auxiliary neurons on every residual connection, a PC-specific batch normalisation, and 18-24 relaxation steps per batch. HGF reaches BP performance on ResNet18 without any of these.
ResNet18. Please see the updated manuscript (and the code supplement); the new section “Other architectures” adds a ResNet18 on CIFAR-10 and a GPT-style transformer (6 blocks, 10.8M parameters) on Tiny Shakespeare. HGF matches BP on both (Table 1). Neither required any change to its update rules or architecture, and our ResNet18 omits batch normalisation in both arms. To our knowledge, these are the first convolutional, residual, and transformer networks of that size trained to BP-level performance by a predictive coding model without iterative inference.
Depth. We explore and characterise the depth limit, which is referred to in section 4.1, the Limitations section and appendix A.4. In plain fully connected networks, at larger depths (>8), the HGF's scaling depends on its posterior precision: with the precision capped, it matches BP up to 64 hidden layers, otherwise it degrades. However, deeper architectures such as ResNet18, VGG9 and the GPT do not show this degradation because each of their learnt parts has at most one interior layer. We also further expand on this in the new Limitations section.
Local hardware. We agree this is central to PC’s motivation and now discuss it in the Limitations section. We still consider this model to be a strong candidate for local hardware, and measures of uncertainty have direct applications to improve control and planning in robotic systems. One of the difficulties will be the use of rather complex update equations that would not easily transfer to any neuromorphic architecture.
We hope these additions address the reviewers' concerns, particularly the ResNet-18 and depth questions.