acceptodds
Under review as a conference paper at ICLR 2027

Circular Predictive Coding Models Unlock Test-Time Computation

Abstract

Predictive coding and equilibrium propagation (EP) perform credit assignment through an energy minimization process. Despite their ability to train models with arbitrary network topologies, they have mostly been used to train feedforward architectures to match the performance of backprop. Here, we use these methods to enable iterative algorithmic reasoning through energy minimization. By closing a feedforward model into a loop, done by identifying its first and last hidden layers, effective recurrent computation emerges from the continuous relaxation of the energy landscape. We show that in predictive coding networks with this circular structure, the bi-level optimization framework of EP provably approximates the gradient update of an implicit layer, allowing it to train a model whose computational depth can increase at test time by relaxing for a longer time for more complex problems. The same construction achieves image classification performance comparable to standard predictive coding models, with substantially fewer parameters, and performs well on medium-scale NLP tasks. This construction has direct implications for physical neural networks, where inference and learning are carried out by the dynamics of the substrate itself, and where computational depth could be traded for recursive operations on a fixed-size circuit.

Primary area:
applications to neuroscience & cognitive science
Keywords:
predictive coding, bio-inspired computing

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_j38AOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Combines well-known concepts of PC trained via EP and recurrence as used in DEQs, with a theoretical connection via IFT. The authors show that circular PC equilibria match DEQ fixed points and EP gradients converge to the implicit loss gradient under stated assumptions, with empirical comparisons to BPTT and IFT. Additional runtime improves task performance and enables length generalization. Their model stays behind DEQ on TinyStories, accompanied by training instabilities whose causes are not fully explained.

Critical strengths
  • The quality of the scientific approach and writeup is high.
  • Theoretical derivation that circular PC matches DEQ equilibria and PC+EP matches the IFT gradient under stated assumptions.
  • Quantitative studies ranging from simple to complex examples, including detailed investigations of gradients.
Critical weaknesses
  • Limited novelty: the method combines established PC and EP techniques with DEQ-like recurrence. Beyond the theoretical connection, the additional methodological contribution or demonstrated practical advantage remains limited.
  • Performance as a function of the number of state nodes in the loop is not covered. Especially looping a single state node would have been interesting, since this aligns with the standard DEQ formulation.
  • No discussion of looped transformers as related work on shared-weight recurrence and test-time computation.
Questions for authors

Questions

  • Could you clarify the motivation for the network architectures and methods in Table 1, and the conclusions drawn?
  • How are the models in Fig. 2 trained, and at which training checkpoints are the gradients compared?
  • How is the DEQ model trained in detail? Does it follow the original DEQ implementation?
  • Which components of the NLP training procedure are necessary, and how does the Hessian-based linear-response solve relate to the claimed locality?

Nits

  • Fix the citep formatting.
  • Clarify “global linear solve” and “solve iteration”.
  • Define IFT at first use.
  • Add panel labels (a–d) to Fig. 2.
  • Define “median commitment iteration”.
  • Use consistent names for the proposed method and comparison methods.
8 comments

W1: Limited Novelty

We kindly disagree on novelty and on the limited practical advantage:

While you are correct that each ingredient (PC, EP, DEQs) already exists, what is novel is the connection among them that never appeared in the literature before: it is non trivial, as we needed two theoretical results to establish it. What is also not novel is the fact that this enables the PC energy to turn any feedforward block, including non-symmetric and normalized ones, into an energy able to perform test-time computations.

Limited demonstrated advantage:

With this change we outperform state of the art predictive coding algorithms on image classification tasks, NLP, and constrained reasoning tasks, despite the use of a much smaller number of parameters -- a 1.5M-parameter, 5-layer model matches PC ResNet-18s, about 11M parameters, without modifying the learning rule. We also provide the very first evidence that a local learning rule can learn iterative algorithms that exploit test-time computation: (1) On Sudoku, solve time tracks deduction depth, which is a task impossible to tackle with the standard, feedforward models; (2) we can train recursive transformer models, largely outperforming previous art.

To make the comparison explicit, we have trained a feedforward 4-layer MLP mixer on sudoku-hard using PC. It reaches 1.2% board accuracy, due to the limited depth, against our model that reaches 66%, despite it being much shallower.

W2: ablation using one node

Thank you for the pointer. We have added an analysis using a larger number of layers in the supplementary material (Fig.6). In the original draft, we had decided to avoid 1-layer recursive blocks for two reasons:

  1. For every non-trivial layer (MLPmixer, transformer), having one node only would make every computation strictly non-local, as every relaxation step would require backpropagating through the whole block. This is in contrast with the core of bio-plausible algorithms: the central point of this work is how to enable test-time computing in an algorithm with local computations, removing the local computations would make all of this meaningless. The only case where this would be possible is when layers can be expressed as a single tensor (MLP, Conv), but for this we refer to the following point.

  2. With only one value node, minimizing the energy is equivalent to finding a fixed point. Hence, a CPC model with one node would be a deep equilibrium model in all senses. The performance then, would be that of a deep equilibrium model trained on a sub-optimal solver.

However, we agree that having multiple nodes can be an interesting ablation. Because of this, we have run two more experiments: (1) In one case, we have replicated the MLP experiment of Figure 2 for the case of 1,2,3,4 hidden layers, for the MNIST, FashionMNIST, and sudoku datasets. You can find it in the supplementary material: the results are comparable to the ones of two layers, even tho a deeper recursive block is less faithful to the gradient due to numerical instabilities, and the fact that gradient descent does not perfectly reach a critical point; (2) we have trained a CPC convolutional model with 1,2,3,4 convolutional layers in the residual block, with some hyperparameter tuning. The test accuracies are as follows: 91.35/91.80/90.31/88.23, respectively, mean of 3 seeds.

Looped Transformers

Universal Transformers, looped transformers, recurrent-depth models and Deep Thinking networks share weight-tying and test-time compute with CPC, but are trained with (truncated) backprop through the iterations. CPC is trained at equilibrium with local updates to make it more suitable for different kinds of neuromorphic computing. We have added an extensive related work on the matter in the supplementary material.

Q1: Motivation for the network architectures and methods in Table 1

We agree with the reviewer that the section/table could have been better explained. We have addressed it now, and specify here: In the literature of PC and Eqprop, most of the works compare their performance on VGG-like models (L convolutional blocks, and a feedforward classifier at the end). To this end, we have compared our circular model against a VGG5 model with the same number of parameters. In short, Why VGG5 as baseline? It is the golden standard in the field.

The motivation is that the table tests whether depth obtained by relaxation can replace depth obtained by stacking, under PC training. Reading the table: The left block compares CPC with feedforward PC models of the same size (VGG5), trained with the standard PC variants of Pinchetti et al. (2025) and the EP formulation of Kerjan et al. (2026). The right block gives the best published deep-PC results (ResNet-18, about 11M parameters, Qi et al. 2025), which reach that accuracy by modifying the learning rule. Conclusion: at equal size, closing the loop beats every feedforward PC variant, and it matches models about 7× larger, (and 4x deeper), without changing the learning rule.

Q2: How are the models in Fig. 2 trained, and at which training checkpoints are the gradients compared?

  • Unroll length: 60 sweeps for 1–2 layers, 90 for 3 layers, 120 for 4 layers.
  • Optimiser: Adam, learning rate 1e-3, batch 128.
  • Duration: 3 epochs on MNIST and FashionMNIST, 200 epochs on 4x4 Sudoku with symmetry augmentation.
  • Seeds: 4 per configuration.
  • Where the gradients are compared: at two points only, always on the same 128 test samples.
  1. Initialization: untrained weights. Panels (a) and (b) of the main figure use this point, as in the published Fig 1.
  2. Final checkpoint: the end of training. Panels (a) and (b) of the figure use this point, and so do panels (c) switch time and (d) relaxation in both figures.

Q3: How is the DEQ model trained in detail? Does it follow the original DEQ implementation?

Same architecture, caps, normalization, optimizer and schedule as CPC. The forward pass uses the same solver and budget as the CPC free phase. The backward pass solves (I−J_Tᵀ)w = ∇_z C. We have implemented everything using the official repo (TorchDEQ).

Q4: Which components of the NLP training procedure are necessary, and how does the Hessian-based linear-response solve relate to the claimed locality?

We empirically find that QK normalization, LayerNorm and linear response nudging are necessary for the full-scale experiment to maintain the network contractive during training and keep the update aligned with backprop gradient. Without QK normalization cosine similarity of the gradient update drifts during training and without LayerNorm the network becomes heavily non-contractive. Using the linear response to set momentum and step size by estimating the extreme values of the Hessian allows for faster convergence of the nudged phase.

The Hessian based solver does not remove locality, as the Hessian is actually never directly computed: the solver needs only Hessian vector products, that in PC energy reduce to quantities that require the same computation each node performs in the free phase, namely its own prediction error and the errors sent by the neighbours. To this end, the method is fully faithful to the local computations of predictive coding. As stated in the paper, momentum and step size are set by two global scalars that bound the eigenvalues of the Hessian that are estimated once per training step.

Thank you again for the time spent on our work. As the deadline for the discussion period is one day away, we kindly ask the reviewer to acknowledge and comment our rebuttal, and to ask any other clarification if needed.

The authors

Reviewer_CRTLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work is quite interesting: it identifies the first and last hidden layers of hierarchical predictive coding (PC) as the same layer, yielding circular predictive coding (CPC); every map around the loop is still defined by the local predictions and squared-residual energy of a PC graph, while the object iterated as a whole is a DEQ-style implicit layer (Bai et al., 2019). There are two theoretical results: when every map in the loop is continuously differentiable and shares a common Lipschitz modulus q<1q<1, the energy has a single stationary point, whose value is zero and which is the global minimum, and whose main value node is the unique fixed point of TT; and when both phases are solved exactly, the EP contrastive update converges as β→0\beta\to 0 to the implicit gradient at that equilibrium (Theorem 2 in the main text; the dependence is stated as the relation among BP through time (BPTT), EP and the implicit function theorem: Zucchet & Sacramento, 2022; Scellier & Bengio, 2017). The experiments cover: Sudoku (the solve rate decreases with dependency depth and the required relaxation iterations increase with depth); prefix-sum (OOD to sequences longer than those seen in training); CIFAR-10/100 (DualCPC is comparable to the PC-variant columns on ResNet-18); and TinyStories against a same-size DEQ (on the EP side a collapse occurs mid-training, ρ(J)\rho(J) drops sharply and the loop degenerates into an almost feedforward network, whereas the DEQ side shows no such behaviour).

Critical strengths
  1. The authors define inference as the energy relaxation of the free phase (pp. 2–3); the nudged phase is introduced only at learning time (pp. 3–4). Theorem 1 shows that this definition is well posed: the main value node of the unique zero-energy stationary point of the free phase is the unique fixed point of the equivalent DEQ (p. 5); on the inference side the output is read from the output vertex (in the loop model it is given by the main value node through the decoder dθd_\theta). This makes the inference side describable entirely by the dynamics of the free phase, without invoking a second equilibrium.
  2. Theorem 1 maps the main value node of the free-phase stationary point onto the fixed point of the equivalent DEQ. Under A.1, with every map sharing the same q<1q<1, Lemmas 1–3 and Theorem 3 establish the unique stationary point with zero energy, with the main value node as the unique fixed point of TT (pp. 13–15); Remark 2 explains why the assumption must be placed on every map inside the loop rather than on the composite map TT. In Related Works the paper states that, for PC graphs with loop/feedback connections, "besides very small scale tasks and a few theoretical results, no real use has been found for it" (p. 2), and it examines test-time computation directly on two reasoning tasks, Sudoku and Prefix-sum (the second contribution bullet); CIFAR/TinyStories serve as standard ML tasks, with Figs. 4/5 additionally reporting effective depth. This gives the circular structure a provable DEQ correspondence and a direct test on two reasoning tasks.
Critical weaknesses
  1. The quantities reported by the experimental protocol are insufficient to verify the premise of A.1. A.1 requires every map in the loop to share the same Lipschitz modulus q<1q<1 (Remark 2 states that this is strictly stronger than contractivity of the composite map TT, p. 15), while the main text invokes Theorem 1 directly through the trained models' "spectral norm ρ(J)<1\rho(J)<1" (p. 6), without stating which map JJ belongs to; Sudoku reports only a per-map spectral-norm cap, and for prefix-sum the per-map cap is 1.05 (>1>1) with an additional downward clamp of 0.8 on the reported ρ(J)\rho(J); the paper does not establish that these caps bound the Lipschitz constant of the full map including GELU/LayerNorm. TinyStories and CIFAR report no cap values.
  2. The experimental readings and the objects of the theorem are not the same quantity. Test-time computation is located at the moment at which the reading is first correct and unchanged thereafter (Figures 2/4), whereas Theorem 1 characterizes the equilibrium and not this moment (for how Remark 3 bears on the solver used in each task, see the next question); the two reasoning tasks report a single model with no DEQ or source-model baseline, so the contribution of the circular structure over an equivalent DEQ cannot be separated.
Questions for authors
  1. The paper places test-time computation at the moment at which the reading is first correct and never changes again (the Figure 2 caption "The sweep after which the loop's prediction is correct and never changes again", and "permanently classified correctly" in the §5.1 description of Figure 4), whereas Theorem 1 characterizes the equilibrium and Remark 3 confines the correspondence to terminal states and excludes accelerated fixed-point solvers and discretizations that require a step size adapted to the curvature of the energy; the Anderson free phase of Sudoku falls in this class, and the TinyStories heavy-ball is the training-time nudged linear-response solve, not the test-time sweep of Figure 5. How do these two levels correspond?
  2. Can the authors give, separately for the 4×44\times 4 Sudoku model of §4.2 (ρ(J)=0.85\rho(J)=0.85) and the 9×99\times 9 model of Appendix B.1 (301,952 parameters, cap 0.70), the definition and measured value of ρ(J)\rho(J), as well as the Lipschitz modulus of each map in the loop? §4.2 invokes Theorem 1 directly through ρ(J)<1\rho(J)<1 — what is its relation to the common qq of A.1?
  3. The prefix-sum section states "ungated: we run truncated EP with no convergence-tolerance gate", whereas the premise of Appendix Theorem 4 (the formal version of Theorem 2 in the main text) is "If both phases are solved exactly"; are this experiment and the theorem on the same footing? How does this formal version relate to the truncated setting of the experiment?
  4. Given that CPC and DEQ share the same solution, which comparison in the paper supports the necessity of the circular structure for these two reasoning tasks?
  5. What is the relation of this work to the line of "trading iterations for depth": adaptive computation time (ACT; Graves, 2016), the Universal Transformer (Dehghani et al., 2019), looped transformers, and trading recurrent depth for test-time computation (Geiping et al., 2025)?
  6. The two ResNet-18 columns of Table 1 are taken from Qi et al. (2025); is their training protocol (number of layers, optimizer and number of epochs, numerical precision) comparable to the columns trained in this paper? Which parameter count does the parameter matching of the VGG5 baseline use?
  7. Which should be taken as authoritative: the readings and parameter counts in the text versus Table 2/Appendix B.3, and the effective-depth formulas of the Figure 4/5 top axes versus the text? The caption/text of Figure 4 and its top axis disagree; the Figure 5 top axis is labelled by effective depth, but the text gives no formula for that figure.
3 comments

We thank the reviewer for the very careful review. About some inaccuracies noted, we’d like to state that we have tried to follow, and be faithful to, the theoretical foundations we laid. However, while we agree that there were some cases in which the models we have used did not respect the premises of the theorems, we have opted for them anyway, if their empirical performance (so, test accuracy) was better than the faithful alternative, as long as it was faithful to the predictive coding formulations.

Besides the above disclaimer, thank you for the pointers: we have dig into them and this has allowed us to reduce this gap between theory and well performing experiments substantially, by doing only a small amount of very minor changes. Here is how:

the premise of A.1 is not verified

Thank you for the pointer, we have slightly reformulated the theorem accordingly: Theorem 1’s per-map assumption is sufficient but not necessary, and our trained models do not satisfy it: once the loop contains normalization layers, weight caps do not bound the Lipschitz constants of the maps. First, the zero-energy states of the circular model are exactly the fixed points of the DEQ map T (Lemma 3). Second, if 1 is not an eigenvalue of J_T at the equilibrium reached (for example, if ρ(J_T) < 1), that equilibrium is isolated and exponentially stable, and EP converges to the DEQ’s implicit gradient as β → 0 (Theorem 2, Prop. 2). This is exactly the condition under which the DEQ’s implicit gradient exists. Unlike A.1, it can be checked on a trained model, and it is the quantity we now report (§4.2, Fig. 5). Spectral caps are now described as a knob that controls ρ, not as a guarantee of contractivity. For prefix-sum, the 0.8 bound is a constraint applied during training, not a modification of the reported value.

W2 / Q1 (the readings and the theorems concern different objects; solvers)

They are different objects, and the paper now says so. The theorems concern terminal states. EP uses only the endpoints of its two phases, so any solver that reaches them (Anderson, heavy-ball or plain sweeps) gives the same estimator. When a solver stops early, Theorem 2 (formally, Theorem 4(iii)) bounds the additional error. That is where we use accelerated solvers: in training. On TinyStories, heavy-ball solves the training-time linear-response system Hv = −∇C, and its residual enters the gradient error through the last clause of Theorem 4(iii). The time-resolved readings (Figs. 2c, 3, 4, 5) are made with plain sweeps from zero. K sweeps are, by construction, a forward pass through a K-fold weight-tied network. “The sweep after which the prediction is correct and never changes” is therefore an exact property of the trained map T and needs no theorem. No accelerated trajectory is ever read, so Remark 3’s restriction does not apply to these figures. On the missing baselines, see Q4.

Q2 (‖J_f‖, q, and per-map moduli)

The quantity we now report is ρ(J_T) at the equilibrium reached. Its relation to q is ρ(J_T) ≤ ‖J_T‖ ≤ q^L. So a measured value below 1 can never establish A.1, and invoking Theorem 1 through it was a typo: it should have invoked Theorem 2, whose premise follows directly from ρ < 1. This is now addressed. In neither model are the per-map Lipschitz moduli bounded below 1. A weight cap c bounds a map containing GELU and LayerNorm only by c · sup GELU′ · Lip(LN). Lip(LN) grows as the inverse standard deviation of its input, which no weight cap controls. A.1 therefore holds for neither model, and we no longer claim it. The paper reports ρ = 0.85 for the 4×4 model of §4.2.

Q3 (ungated, truncated EP versus exact phases)

The theorem and the experiment play different roles. Theorem 4(i) characterises the exact estimator: both phases at equilibrium and β → 0. Any implementation, ours included, approximates this limit with a finite number of local updates, as a physical system would by settling for a finite time. The theorem says what these updates approximate, not that every run reaches the limit. This is the usual relation between theory and implementation in EP (Scellier & Bengio, 2017; Laborieux et al., 2021) and in DEQs (Bai et al., 2019), whose solvers are likewise stopped after a finite number of iterations. Theorem 4(iii) makes the approximation explicit: the deviation from the exact estimator is first order in the residuals at termination, so it vanishes as the phases converge. “Ungated” only means that we do not discard samples whose phases have not fully converged. The prefix-sum results show that the resulting approximate update is sufficient for the task.

Q4 (is the circular structure necessary?)

Note that the necessity of a circular structure must be considered only within the predictive coding literature, and we believe that (1) the results confirm this: We reach results better or comparable to state of the art in the field, despite being much more parameter efficient in computer vision, and allow the training of transformer and reasoning models, something that was out of reach without the circular structure, due to the difficulty of training deep models using this learning algorithm; (2) an acyclic structure cannot, by definition, be used for test-time compute, as its inference is equivalent to a forward pass.

Q5 (Connection to looped transformers)

Like ACT, the Universal Transformer, looped transformers, recurrent-depth models (Geiping et al., 2025) and Deep Thinking networks (Schwarzschild et al., 2021; Bansal et al., 2022), CPC reuses one block to trade iterations for depth. It differs in two ways. Its output is defined as an equilibrium, as in DEQs (Anil et al., 2022), rather than the state after a fixed or learned number of steps, so no halting mechanism is learned. And it is trained by a local contrast between two equilibria, rather than by backpropagating through (truncated) unrolled iterations. We do not claim higher accuracy than these models. We show that this regime can be reached with a local, energy-based learning rule. More broadly, you can see our work as a step towards moving the bio-plausible/neuromorphic ML field towards the direction of looped architectures, popular in the field nowadays.

Q6 (Training protocol of ResNets vs our models)

The training protocols are nearly identical: same dataset, same train/validation split, same data augmentation, and we have used the same library (pcx). The models are not: Our circular model is much smaller and parameter efficient: CPC reaches the same accuracy range with about seven times fewer parameters than a ResNet-18, and using 5 layers only instead of 18. For the VGG5 baselines, Table 1 now gives the total parameter counts (encoder, loop and decoder) for both models, and the text describes them as of comparable size rather than identical.

Q7 (Typos in the numbers)

The figure axes are the correct ones, and we corrected the typo in text to match them (the numbers were a residual of an older setup, for which we apologize). Effective depth counts the layers on the longest path from input to the last readout before convergence. For the CIFAR model this is 3 + 2t: three encoder convolutions and the readout, plus two loop convolutions for every sweep after the first. On the first sweep, the loop convolutions act on the zero-initialised state, not on the input. Figure 5’s caption now gives its own formula under the same convention. All readings and parameter counts in the text now match Table 2 and Appendix B.3.

Thank you again for the time spent on our work. As the deadline for the discussion period is one day away, we kindly ask the reviewer to acknowledge and comment our rebuttal, ask any other clarification if needed, or adjust the vote if you believe our rebuttal has addressed your concerns.

The authors

Reviewer_911bOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper demonstrates that looped Predictive Coding (PC) networks may be interpreted as the energy-based formulation of Deep Equilibrium Models (DEQs). Such PC-DEQs can then be trained with Equilibrium Propagation (EP) instead of the DEQ's usual implicit gradients / backprop-through-time, a combination that seems especially promising for a physical implementation, where additional compute is cheap but chip area is limited. The theory lays out the equivalence between looped PC networks and DEQs as well as sufficient conditions for convergence guarantees, and the experiments demonstrate that the well-established properties of DEQs (wrt test-time computation) are preserved under this energy-based formulation.

Critical strengths
  • I like how this paper ties together DEQs, PC and EP. The insight of connecting looped PC networks with DEQs is original, well-explained, and only obvious in hindsight.
  • This analog version of DEQs, trainable with EP, seems very relevant for physical hardware implementations. Increasing effective depth by temporal unrolling is much easier to do than spatial unrolling on the chip.
  • I appreciate how the paper is very well presented, with clear writing style and helpful figures.
Critical weaknesses

I like this paper, and with its combined strengths, it is worthy of acceptance, in my opinion.

However, one aspect absolutely needs to be addressed before publication: the paper has a tendency of overclaiming and underciting. I made a list:

  • Despite its name and the first sentence of the abstract, CPC is not a PC credit assignment algorithm, but rather a network architecture. This is especially misleading in Table 1, where CPC is compared to other credit assignment algorithms. While the authors are generally careful to write "CPC models", I'd simply prefer to only see the term CPCN (circular PC network), instead of CPC, to indicate that this is an architecture, not a new type of PC.
  • Theorem 2 (& 4) are presented as original work. However, once it's clear that CPC is an architecture, these theorems become simple restatements from established EP literature. This should be made much clearer.
  • The paper is about test-time compute: solving more difficult tasks by giving the network more iterations. However, Fig 3a clearly shows that the difficult puzzles are not more easily solved by the network, even with additional time steps. Additionally, Fig 3b shows that there exist both easy and hard puzzles that take the same nr of iterations to be solved, contrary to the general hypothesis that "more complex deductions require more computation". To be fair, I realize that these two points cannot be fully explained, but I expect them to be pointed out and discussed.
  • These references are noticeably missing from the paper:
    • Ernoult et al. (2019) "Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input" NeurIPS: proves how EP approximates BPTT on RNNs, directly relevant to the paper and at the very least Fig 2a
    • Scellier & Bengio (2019) "Equivalence of equilibrium propagation and recurrent backpropagation" Neural computation: proves that EP = rBP, basically Theorem 2
    • Millidge et al. (2024) "Predictive coding networks for temporal prediction" PLOS Computational Biology: first paper to use PC for temporal learning (ie, mimicking BPTT)
    • Potter et al. (2026) "Learning Long-Range Dependencies with Temporal Predictive Coding" arXiv: follow-up work to the paper above, showing tPC can equal BPTT
    • Du et al. (2022) "Learning Iterative Reasoning through Energy Minimization" ICML: performs iterative algorithmic reasoning through energy minimization (albeit non-local)
    • Goemaere et al. (2026) "ePC: Fast and Deep Predictive Coding in Digital Simulation" ICML: a formal solution to PC's depth scaling issue, scaling to CIFAR10/100 (albeit non-local)
    • Bai et al. (2021) "Stabilizing Equilibrium Models by Jacobian Regularization" ICML: a principled approach to controlling the spectral norm of DEQs, as opposed to hard capping
    • Basically any looped transformer literature (like Dehghani et al. (2019) "Universal transformers" ICLR or Geiping et al. (2025) "Scaling up test-time compute with latent reasoning: A recurrent depth approach" NeurIPS)
    • Directly used assets in the paper, including VGG, MNIST, FashionMNIST, CIFAR, GELU and LayerNorm

I invite the authors to push back on suggested references they deem unnecessary.

If this list is not addressed (either through a modification of the manuscript or a clear argumentation of why a reference is left out), I will lower my score.

Questions for authors
  • What is the role of the encoder and decoder in the setup? How powerful do they have to be to make everything work? Can they be trained locally?
  • Fig 3.a shows that not all puzzles can be solved by the network, not even with additional iterations. What's the total accuracy? How does this compare to the original baseline of Palm et al. (2018)?
  • If CPCN+EP gives the same gradients as DEQ+IFT, why does Table 2 show a noticeably different perplexity?
4 comments

We thank the reviewer for the nice words, and detailed feedback. We have implemented all of the changes suggested/related works. They are now in the first page of the supplementary material, and will be moved to the main body when the page count will be extended (in case of acceptance).

I'd simply prefer to only see the term CPCN (circular PC network), instead of CPC

Addressed everywhere in the manuscript, thank you for the pointer!

the paper has a tendency of overclaiming

We apologize about the overclaiming: it was not our intention. In fact, we stated clearly in the list of contributions in the intro that:

  • “.... We use the result of Zucchet & Sacramento (2022) to show that when performing an update of the parameters, EP approximates the implicit gradient of the corresponding deep equilibrium model;”*

referring to Theorem 2. However, we agree that this should be also re-stated before the theorem. We have done so, now right before the theorem statement, the paper reads

“....More formally, we state the following result, adapted from existing work \citep{zucchet2022beyond,scellier17}:...”

Citations and Looped Transformers

Thank you for the many pointers. They are now all cited in a much more extensive related works section.

Q1: What is the role of the encoder and decoder in the setup? How powerful do they have to be to make everything work? Can they be trained locally?

The role of the encoder and the locality concern only applies to the experiment on convolutional models. In all other experiments, the encoder is a single layer (mostly, a kind of token embedding), which preserves the locality of operations. More generally, in our vision the models follow a “hybrid Eqprop” formulation (Nest, Ernoult, NeurIPS 2024), where in case of need we can use an encoder with non-local computations, and perform the expensive recurrent ones locally.

However, nothing prevents us from training everything locally, and we have done so on CIFAR10 and CIFAR100 experiments now. The results are slightly worse than the reported ones: 90.012 for CIFAR10, and 69.30 for CIFAR100 (avg of 3 seeds). We are running a deeper hyperparameter search, as these are early results, but nonetheless they already show that the performance stays almost as strong with a fully local model.

Fig 3.a shows that not all puzzles can be solved by the network, not even with additional iterations. What's the total accuracy? How does this compare to the original baseline of Palm et al. (2018)?

We have improved the results on the Sudoku dataset as follows:

We have found a new configuration of parameters that improved the old results by more than 10%, where we observe a stronger case of test-time computations: the results keep improving even after 500 iterations, despite the model was trained for much less. Furthermore, we use test-time augmentation to better find the solution: given a test point x, we first produce 64 augmentations of x using the symmetry group of the sudoku (rotations, flips, digit swaps, etc.), and then run inference over the 64 variations of the test point in parallel. At convergence, we select the label with the lowest self-cost, i.e., the cross-entropy between the output and its own most likely digits. This selection uses neither labels nor the rules of Sudoku, and uses local computations only. It allows us to scale to 82% test accuracy, with signs that we could scale up more with more test-time compute, as it can be observed in the plot (c).

About the comparison with Palm et al., they reach ~98/99% accuracy, but embed a lot of the structure of the sudoku inside the architecture, and are hence not directly comparable: They use (1) an equivariant model, that contains all the symmetries of the sudoku, and (2) also a graph structure that connects cells in the same row/column/block to each other. Our model, on the other hand, has none of these and must learn the structure of the task in his weights.


Q3 If CPCN+EP gives the same gradients as DEQ+IFT, why does Table 2 show a noticeably different perplexity?

EP and implicit differentiation give the same gradient at a given set of parameters, in the limit of small β and exactly reached equilibria. In practice, fully converging the nudged equilibrium is expensive, but an approximate solution already gives high fidelity to the DEQ+IFT gradient: Figure 5 shows the two gradients remaining close throughout training. The small per-step differences nevertheless accumulate into a different trajectory, and hence a different final perplexity.

We thank the reviewer for the kind words about our work. If you still believe the work is worth acceptance, would it be possible to reflect this in the score?

Fig 3a clearly shows that the difficult puzzles are not more easily solved by the network, even with additional time steps.

We thank the reviewer. The table below shows how many puzzles are solved after a given number of iterations.

Chain depth1025507510050010001500
3–4183100100100100100100
5–6026859394969696
7–901315463747475
10–140031119363838
≥ 1500024161718
All018455459686868

Table R1. Puzzles solved (%) vs. number of iterations.

(i) Harder puzzles need more computation, but iterations alone stop helping. Easy puzzles are solved within 25–50 iterations. The hardest ones (depth ≥ 15) are almost never solved before ~100 iterations, and most of their progress comes between 100 and 500 iterations. After that they plateau: 16% at 500 iterations and 18% at 1500. The reviewer is right that more iterations alone do not solve them, because the network settles into a wrong but stable state. The ceiling depends on the starting point, not on running time. If we keep the lowest-energy run among 1024 digit relabellings (no labels used), the hardest group goes from 18% to 87%, and the overall rate from 68% to 97%.

(ii) Easy and hard puzzles can take the same number of iterations. This is true, and we will say so explicitly. The trend is strong: the median solve iteration grows from 16 at depth 3 to ~145 at depth 15–16, with Spearman ρ = 0.85. The spread within each depth is wide, however, which is likely because chain depth reflects one solver's deduction order while the network updates all cells in parallel. We will therefore phrase the claim as "on average, deeper deductions require more computation.". This table, and a similar one depicting the results for 1024 views, are now in the supplementary material (Table 3)

Thank you again for the time spent on our work. As the deadline for the discussion period is one day away, we kindly ask the reviewer to acknowledge and comment our rebuttal, and to ask any other clarification if needed.

The authors

I'm glad to see the authors engage with my suggestions and questions. The modified manuscript now makes it very clear that CPCN is an architecture and that Theorem 2 is a restatement of EP theory, applied to CPCN. The test-time compute story is still a difficult one, but the clarification of the authors and the addition of appendix C.2 are sufficient for me. The expanded related works section reads well and will make a nice addition to the main body of the paper. All my questions have been answered to satisfaction, for which I'd like to thank the authors.

I'm sure the authors accidentally skimmed over my final bullet, but I'm still really missing proper attribution for the many assets used in the paper (incl. ImageNet now, also missing its citation). I realize these are well-established datasets and architectural components, so the citation is not intended as a reference for the reader, but rather as credit attribution to the original work. I'm happy to already raise the soundness score, and to raise my final score to 4, but I am expecting the additional attribution in the final version. I think this is a good paper and I hope to see it get accepted.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Edits to the manuscript

We thank the reviewers for the time and effort devoted to evaluating our work. We have uploaded a revised version of the manuscript, which we believe substantially improves upon the original submission. These improvements include changes motivated by the reviewers’ feedback, as well as additional experiments conducted to further strengthen our results. While we address each reviewer’s specific comments in the individual rebuttals below, we first summarize the main improvements introduced in the revised manuscript.

  1. Computer Vision: Scaling to ImageNet32

We have substantially extended our computer vision experiments, scaling from CIFAR-100 to ImageNet32, a considerably more challenging dataset with 10 times as many classes and approximately 20 times as many training examples. To tackle this setting, we use a model with 7.2 million parameters. The results are encouraging: our method outperforms all previously reported predictive coding baselines in the literature. In particular, we compare against the VGG15 architecture presented in [Qi et al., 2025], which our method surpasses despite using substantially fewer parameters. These results provide strong evidence that our approach scales to challenging computer vision tasks, extending its applicability beyond the relatively small-scale benchmarks typically considered in equilibrium propagation, predictive coding, and biologically plausible learning.

  1. Sudoku: Improved Performance and Test-Time Scaling

We have significantly improved our Sudoku results through two complementary developments. First, we identified a new hyperparameter configuration that improves our previous results by more than 10 percentage points. Importantly, this configuration exhibits even stronger test-time scaling: performance continues to improve beyond 500 inference iterations, despite the model having been trained with substantially fewer iterations.

Second, we introduce a test-time augmentation strategy that further improves solution accuracy. Given a test input (x), we generate 1,024 equivalent instances using the symmetry group of Sudoku, including rotations, reflections, and digit permutations, and perform inference on all instances in parallel. At convergence, we select the prediction with the lowest self-cost, defined as the cross-entropy between the model’s output distribution and its own most likely digit assignments. Crucially, this selection procedure requires neither ground-truth labels nor explicit knowledge of Sudoku rules, and relies exclusively on local computations. This approach achieves 97% test accuracy, with further improvements potentially achievable through additional test-time computation, as illustrated in Figure 3(c) of the revised manuscript.

  1. Theoretical and Empirical Analysis of Recursive Block Depth

Following suggestions from two reviewers, we have added a study investigating how our theoretical results and empirical performance extend as we increase the number of nodes and layers within the recursive block. We evaluate this question on MNIST, Fashion-MNIST, and a smaller Sudoku benchmark, providing both theoretical analysis and experimental evidence of the behavior of our method as the recursive architecture becomes deeper.

  1. Additional Analysis of Test-Time Computation

We have expanded the supplementary material with detailed tables examining the relationship between the number of inference iterations and Sudoku performance, particularly on more challenging boards. These experiments provide a more systematic characterization of test-time scaling and its impact on solving increasingly difficult instances. The results are presented in the Sudoku section of the supplementary material.

  1. Expanded Literature Review

Finally, we have substantially expanded the related work section to provide a more comprehensive overview of the relevant literature. This includes the papers suggested by the reviewers, as well as several additional works that help clarify the relationship between our approach and existing methods.

We thank the reviewers again for their constructive feedback, which has helped us strengthen both the experimental evaluation and the presentation of our work.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.