- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Combines well-known concepts of PC trained via EP and recurrence as used in DEQs, with a theoretical connection via IFT. The authors show that circular PC equilibria match DEQ fixed points and EP gradients converge to the implicit loss gradient under stated assumptions, with empirical comparisons to BPTT and IFT. Additional runtime improves task performance and enables length generalization. Their model stays behind DEQ on TinyStories, accompanied by training instabilities whose causes are not fully explained.
Critical strengths
- The quality of the scientific approach and writeup is high.
- Theoretical derivation that circular PC matches DEQ equilibria and PC+EP matches the IFT gradient under stated assumptions.
- Quantitative studies ranging from simple to complex examples, including detailed investigations of gradients.
Critical weaknesses
- Limited novelty: the method combines established PC and EP techniques with DEQ-like recurrence. Beyond the theoretical connection, the additional methodological contribution or demonstrated practical advantage remains limited.
- Performance as a function of the number of state nodes in the loop is not covered. Especially looping a single state node would have been interesting, since this aligns with the standard DEQ formulation.
- No discussion of looped transformers as related work on shared-weight recurrence and test-time computation.
Questions for authors
Questions
- Could you clarify the motivation for the network architectures and methods in Table 1, and the conclusions drawn?
- How are the models in Fig. 2 trained, and at which training checkpoints are the gradients compared?
- How is the DEQ model trained in detail? Does it follow the original DEQ implementation?
- Which components of the NLP training procedure are necessary, and how does the Hessian-based linear-response solve relate to the claimed locality?
Nits
- Fix the citep formatting.
- Clarify “global linear solve” and “solve iteration”.
- Define IFT at first use.
- Add panel labels (a–d) to Fig. 2.
- Define “median commitment iteration”.
- Use consistent names for the proposed method and comparison methods.
8 comments
Q1: Motivation for the network architectures and methods in Table 1
We agree with the reviewer that the section/table could have been better explained. We have addressed it now, and specify here: In the literature of PC and Eqprop, most of the works compare their performance on VGG-like models (L convolutional blocks, and a feedforward classifier at the end). To this end, we have compared our circular model against a VGG5 model with the same number of parameters. In short, Why VGG5 as baseline? It is the golden standard in the field.
The motivation is that the table tests whether depth obtained by relaxation can replace depth obtained by stacking, under PC training. Reading the table: The left block compares CPC with feedforward PC models of the same size (VGG5), trained with the standard PC variants of Pinchetti et al. (2025) and the EP formulation of Kerjan et al. (2026). The right block gives the best published deep-PC results (ResNet-18, about 11M parameters, Qi et al. 2025), which reach that accuracy by modifying the learning rule. Conclusion: at equal size, closing the loop beats every feedforward PC variant, and it matches models about 7× larger, (and 4x deeper), without changing the learning rule.
Q2: How are the models in Fig. 2 trained, and at which training checkpoints are the gradients compared?
- Unroll length: 60 sweeps for 1–2 layers, 90 for 3 layers, 120 for 4 layers.
- Optimiser: Adam, learning rate 1e-3, batch 128.
- Duration: 3 epochs on MNIST and FashionMNIST, 200 epochs on 4x4 Sudoku with symmetry augmentation.
- Seeds: 4 per configuration.
- Where the gradients are compared: at two points only, always on the same 128 test samples.
- Initialization: untrained weights. Panels (a) and (b) of the main figure use this point, as in the published Fig 1.
- Final checkpoint: the end of training. Panels (a) and (b) of the figure use this point, and so do panels (c) switch time and (d) relaxation in both figures.
Q3: How is the DEQ model trained in detail? Does it follow the original DEQ implementation?
Same architecture, caps, normalization, optimizer and schedule as CPC. The forward pass uses the same solver and budget as the CPC free phase. The backward pass solves (I−J_Tᵀ)w = ∇_z C. We have implemented everything using the official repo (TorchDEQ).
Q4: Which components of the NLP training procedure are necessary, and how does the Hessian-based linear-response solve relate to the claimed locality?
We empirically find that QK normalization, LayerNorm and linear response nudging are necessary for the full-scale experiment to maintain the network contractive during training and keep the update aligned with backprop gradient. Without QK normalization cosine similarity of the gradient update drifts during training and without LayerNorm the network becomes heavily non-contractive. Using the linear response to set momentum and step size by estimating the extreme values of the Hessian allows for faster convergence of the nudged phase.
The Hessian based solver does not remove locality, as the Hessian is actually never directly computed: the solver needs only Hessian vector products, that in PC energy reduce to quantities that require the same computation each node performs in the free phase, namely its own prediction error and the errors sent by the neighbours. To this end, the method is fully faithful to the local computations of predictive coding. As stated in the paper, momentum and step size are set by two global scalars that bound the eigenvalues of the Hessian that are estimated once per training step.
Thank you again for the time spent on our work. As the deadline for the discussion period is one day away, we kindly ask the reviewer to acknowledge and comment our rebuttal, and to ask any other clarification if needed.
The authors
We kindly disagree on novelty and on the limited practical advantage:
While you are correct that each ingredient (PC, EP, DEQs) already exists, what is novel is the connection among them that never appeared in the literature before: it is non trivial, as we needed two theoretical results to establish it. What is also not novel is the fact that this enables the PC energy to turn any feedforward block, including non-symmetric and normalized ones, into an energy able to perform test-time computations.
Limited demonstrated advantage:
With this change we outperform state of the art predictive coding algorithms on image classification tasks, NLP, and constrained reasoning tasks, despite the use of a much smaller number of parameters -- a 1.5M-parameter, 5-layer model matches PC ResNet-18s, about 11M parameters, without modifying the learning rule. We also provide the very first evidence that a local learning rule can learn iterative algorithms that exploit test-time computation: (1) On Sudoku, solve time tracks deduction depth, which is a task impossible to tackle with the standard, feedforward models; (2) we can train recursive transformer models, largely outperforming previous art.
To make the comparison explicit, we have trained a feedforward 4-layer MLP mixer on sudoku-hard using PC. It reaches 1.2% board accuracy, due to the limited depth, against our model that reaches 66%, despite it being much shallower.
Thank you for the pointer. We have added an analysis using a larger number of layers in the supplementary material (Fig.6). In the original draft, we had decided to avoid 1-layer recursive blocks for two reasons:
For every non-trivial layer (MLPmixer, transformer), having one node only would make every computation strictly non-local, as every relaxation step would require backpropagating through the whole block. This is in contrast with the core of bio-plausible algorithms: the central point of this work is how to enable test-time computing in an algorithm with local computations, removing the local computations would make all of this meaningless. The only case where this would be possible is when layers can be expressed as a single tensor (MLP, Conv), but for this we refer to the following point.
With only one value node, minimizing the energy is equivalent to finding a fixed point. Hence, a CPC model with one node would be a deep equilibrium model in all senses. The performance then, would be that of a deep equilibrium model trained on a sub-optimal solver.
However, we agree that having multiple nodes can be an interesting ablation. Because of this, we have run two more experiments: (1) In one case, we have replicated the MLP experiment of Figure 2 for the case of 1,2,3,4 hidden layers, for the MNIST, FashionMNIST, and sudoku datasets. You can find it in the supplementary material: the results are comparable to the ones of two layers, even tho a deeper recursive block is less faithful to the gradient due to numerical instabilities, and the fact that gradient descent does not perfectly reach a critical point; (2) we have trained a CPC convolutional model with 1,2,3,4 convolutional layers in the residual block, with some hyperparameter tuning. The test accuracies are as follows: 91.35/91.80/90.31/88.23, respectively, mean of 3 seeds.
Universal Transformers, looped transformers, recurrent-depth models and Deep Thinking networks share weight-tying and test-time compute with CPC, but are trained with (truncated) backprop through the iterations. CPC is trained at equilibrium with local updates to make it more suitable for different kinds of neuromorphic computing. We have added an extensive related work on the matter in the supplementary material.