- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work focuses on stabilizing the training of Predictive Coding (PC) networks, addressing the specific challenge that such networks often fail to generate meaningful prediction errors within a limited number of inference steps. The authors propose a two-phase approach to PC learning. During the inference phase, instead of making the update of hidden states dependent on a Jacobian matrix recalculated based on the current state, the network first performs a forward pass to obtain reference activations . The hidden states are then expressed as deviations , such that the inter-layer prediction error is defined as . During the learning phase, after the forward pass, parameter gradients are computed according to local rules and then preconditioned using the inverse of the second moment of activations. This step mitigates the distortion caused by uneven gains across different activation directions.
Critical strengths
The mechanism behind the reduction in cost of inference learning is quite intuitive in principle, and this cost saving has also been confirmed in practice.
Critical weaknesses
- The core idea of using a feedforward reference as an anchor for weight updates has already appeared in Qi et al. (Towards the Training of Deeper Predictive Coding Neural Networks, arXiv:2506.23800) as Forward Update. The paper neither cites nor discusses that work.
- The analysis in Appendix A indicates that the condition number of the coupling matrix is ; this implies that the number of inference steps required to propagate task signals along the network chain grows quadratically with depth. This phenomenon is already clearly observable in Figure 2c: in the ViT-8 model (using FP64 precision), the update increments for layers 1 through 4 fall below the FP64 threshold, meaning that, over the course of inference steps, the task signal fails to effectively propagate to the first half of the network. Nevertheless, the paper claims to achieve "stable latent-state inference" without addressing this limitation. Future research should incorporate experiments at varying depths (e.g., ) and provide a more rigorous and qualified characterization of what is meant by "stability."
- In the experimental section, the shared hyperparameters are overly tuned to favor the proposed method, which ends up suppressing the performance of the BP baseline. In the publicly available code repository, simply adjusting the learning rate (lr) for the BP baseline yields an 11% performance gain.
Questions for authors
See the weaknesses above
4 comments
Finite-step stability, realized updates, and BP tuning fairness
We separately evaluated finite-step inference stability and the effect of CoPC's local signals on parameter learning. Checkpoints from three training methods and three initialization-seed labels formed nine training histories. With 27 checkpoints (epochs 1/10/20), two training batches each, and three state-step settings, we obtained 162 paired repeated inference conditions per method, rather than 162 independently trained models.
| Transport | Inference-objective decrease after 8 steps | Common task-CE decrease after 8 steps |
|---|---|---|
| CoPC fixed coupling | 162/162 | 162/162 |
| Reference Jacobian | 137/162 | 150/162 |
| Gain-normalized Jacobian | 162/162 | 162/162 |
For reference-Jacobian transport, the objective decreased in 42/54, 53/54, and 42/54 cases by seed label; gain normalization raised each to 54/54. At the fixed state step , the counts were 54/54 for CoPC, 45/54 for reference Jacobian, and 54/54 for normalized Jacobian. These paired checkpoint/batch/step-size measurements directly demonstrate how Jacobian-gain control affects finite-step objective reduction. We report each method's inference objective separately from the common task CE.
We then tested realized parameter updates from identical ViT-8 checkpoints and saved AdamW states. Including (ON) or removing (OFF) the supervised task force during inference, we updated the body and classifier once and evaluated CE on separate training batches.
| Seed | ON ΔCE | OFF ΔCE | ON−OFF |
|---|---|---|---|
| 200 | −0.020147 | +0.065869 | −0.086016 |
| 201 | +0.001249 | +0.050801 | −0.049552 |
| 202 | +0.000319 | +0.201131 | −0.200812 |
| Mean±sample SD | −0.006193±0.012093 | +0.105934±0.082787 | −0.112126±0.078938 |
ON−OFF was negative for all three seeds, while the head-only difference was approximately . This directly establishes the contribution of task-dependent inference signals to local parameter updates in the network body. The one-step comparison incorporates the saved AdamW moments.
We also varied inference steps at the same checkpoints. Mean post-update CE changes at K=8, 16, and 32 were −0.00619, −0.02936, and +0.1565, respectively, although each setting reduced its inference objective. Some intermediate-layer signals grew as K increased, while blocks 1–3 remained numerically zero in the measured finite-precision runs. The inference budget therefore affects not only the depth reached by the signals but also the regime in which parameter updates are effective. At K=8, training only the final block and classifier yielded 27.16±0.30%, versus 27.35±0.77% when all blocks and the classifier were trained (paired full-minus-top +0.19±0.50 pp). Corresponding one-step CE changes were −0.006191 and −0.006193. These results characterize where learning is concentrated at the present inference budget.
We provided BP, Meta-PCN, Standard PC, μPC, and ePC-T with the same outer-LR search budget. On CIFAR-100/ViT-8, each method evaluated five LR candidates for 20 epochs on the first seed, rechecked the top two for 20 epochs on a second seed (140 tuning epochs per method), and reported three independent 20-epoch runs using validation-selected LRs.
| Method | Shared-LR test (%) | Selected LR | Selected test (%) | Observed Δ (pp) |
|---|---|---|---|---|
| BP | 32.11±0.44 | 0.0001 | 33.80±0.42 | +1.69 |
| Meta-PCN | 23.87±0.47 | 0.0009 | 31.90±0.93 | +8.04 |
| Standard PC | 2.67±1.94 | 0.0009 | 15.98±0.96 | +13.31 |
| μPC | 22.80±0.67 | 0.0027 | 22.80±0.67 | 0.00 |
| ePC-T | 18.77±15.39 | 0.0027 | 18.77±15.39 | 0.00 |
| CoPC† | 27.12±1.01 | 0.0081 | 27.35±0.77 | +0.22 |
Mean±sample SD (n=3). †CoPC is the final proposed method evaluated under a separately selected configuration.
After matched-budget tuning, BP improved from 32.11±0.44% to 33.80±0.42% (+1.69 pp), Meta-PCN by 8.04 pp, and Standard PC by 13.31 pp. BP's selected LR=0.0001 was the lower search boundary. These results directly address baseline tuning fairness while quantifying CoPC's fixed-transport design, computational savings, and learning behavior.
In depth experiments, CoPC achieved 27.12±1.01%, 26.86±0.46%, and 27.33±0.55% on CIFAR-100 ViT-8/12/16, with GH200 iteration times of 126.08/181.57/239.44 ms. It also reached 79.93±0.70% (n=3) on a non-residual Fashion-MNIST eight-block MLP. CoPC thus links a distinct fixed inference-transport mechanism to explicit task-dependent local gradients, measured finite-step behavior, realized parameter updates, and controlled baseline comparisons.
Dear Reviewer 6BGA,
As the discussion period draws to a close, we would like to emphasize the mechanism and empirical evidence behind CoPC's stable inference. Its fixed quadratic prediction-error structure avoids state-dependent Jacobian amplification, achieving consistent finite-step objective reduction in 162/162 conditions, versus 137/162 for reference-Jacobian transport. We distinguish this stability from uniform credit assignment, explicitly quantify signal attenuation at K=8, and demonstrate consistent accuracy across ViT-8/12/16.
We hope these findings address your concerns and can inform your final assessment.
Thank you for your time and consideration.
- “not only in the subsequent weight-update signal” confirmed my initial review.
- “both the local signals and gradients of blocks 1–3 were numerically zero in all 12/12 observations.” indicates that blocks 1–3 of the network were not trained at all, retaining only their initial random weights. Consequently, the advantage of this 12-block network compared to a standard 9-block network is highly questionable.
- Since the LR=0.0001 meets the lower boundary, further decreasing the LR for BP might continue to improve BP’s performance (the 11% improvement I got for BP was on CIFAR-10 using the provided code). Considering this large (and potentially larger) gap, coupled with the inconsistent performance of the proposed model across different seeds (as noted in the response to other reviewers), I don’t think the result can fully support the claim of the paper.
Considering all of the above, I will keep my current score.
Distinction from Qi et al., task-to-gradient learning, and layerwise signals
CoPC's algorithmic contribution is to use feedforward references to construct fixed difference coupling for inter-layer latent-state inference and to connect the resulting task-dependent local learning signals to blockwise parameter updates. Instead of repeatedly evaluating state- and weight-dependent learned-Jacobian transport, the reference-based differences govern inter-layer propagation. Current-model information enters through task gradients along the inference trajectory and block-specific parameter derivatives.
The Forward Update of Qi et al., Towards the Training of Deeper Predictive Coding Neural Networks (ICML 2026) uses the difference ϵl=hl(K)−al between final inferred states and initial feedforward predictions as a reference for local parameter learning. CoPC instead uses the feedforward reference to define the error structure during inference itself: dl=hl−al and el=dl−dl−1. Its final local signal ql(K)=κel(K) then enters each block's local gradient. The key difference is that CoPC uses the reference in the inference operator, not only in the subsequent weight-update signal.
We adapted Qi et al.'s core Forward Update rule to the same ViT-8 architecture and compared CIFAR-100 performance after 20 epochs with K=8.
Accuracy: official test with validation-selected settings, mean±sample SD over three independent seeds; runtime/memory: 100 timed iterations on RTX 5090, batch 256. This is a ViT adaptation of the core Forward Update rule; spiking precision, auxiliary neurons, and BatchNorm freezing in the original paper are separate experimental components.
Under the compared conditioned settings, CoPC achieved 2.44× shorter iterations and 49.4% lower peak memory. Isolated inference took 123.06 ms for Forward Update versus 20.97 ms for CoPC; local-gradient computation (15.34/15.38 ms) and additional parameter-gradient processing (24.11/24.53 ms) were similar. The main runtime difference arose during latent-state inference itself. Forward Update's high seed-wise variance includes one run whose late-epoch validation accuracy fell from 28.58% to 8.22%.
Why CoPC can learn is made explicit by the task-to-gradient connection across all layers. Combining Eq. (3) with the synchronous updates in Appendix B.2 yields
gt=∇hLLtask(hL(t)),ql(K)=−B∑t=0K−1wl,tgt,
wl,t=c[(I−cDD⊤)K−1−t]lL,c=ηhκ,
Gθlraw=Jθlfl(al−1)⊤∑t=0K−1wl,tgt.
This requires fixed references/parameters during inference, d(0)=0, a shared state step with synchronous updates, minibatch-mean task CE, and h(t+1)=h(t)−ηhB∇hF. The gt are task gradients evaluated along the actual inferred trajectory; Graw is the detached-signal local gradient before parameter updates. The task loss supplies the signal, fixed transport forms depth- and time-dependent local signals, and each block's parameter derivative maps them into local learning gradients.
For the fixed difference matrix D, the prediction-error quadratic term has Hessian HPE=(κ/B)D⊤D and condition number Θ(L2): 113.50/249.65/437.70 at depths 8/12/16. In exact arithmetic, task information from the terminal layer can first reach block l at K≥L−l+1; we also measured its finite-precision magnitude.
Using three selected ViT-8 epoch-20 checkpoints and four fixed training batches per checkpoint, the inference objective decreased in all 12/12 observations at K=8, with mean task-CE change −0.000296±0.000131.
Block 4 had numerically distinguishable signals in 5/12 observations, whereas both the local signals and gradients of blocks 1–3 were numerically zero in all 12/12 observations. The small coefficient c=ηhκ≈3.00×10−5 and FP64 rounding/cancellation contribute to this attenuation, consistent with the limited effective lower-layer directions in Fig. 2c. Fixed coupling defines a learning path for every block, while the measured K=8 signals are concentrated in output-proximal blocks.