acceptodds
Under review as a conference paper at ICLR 2027

Coherent Predictive Coding: Stabilizing Latent-State Inference and Local Parameter Learning

Abstract

Predictive coding (PC) is a biologically inspired alternative to backpropagation that enables local and parallelizable learning by iteratively inferring latent states and learning parameters from layerwise prediction errors. However, forming useful prediction errors within a limited inference budget remains challenging. We address this problem as two connected stages: latent-state inference, which forms prediction errors, and local parameter learning, which translates them into parameter updates. In Standard PC, latent-state updates depend on learned Jacobians evaluated at the current latent states. After inference, anisotropic activation statistics can distort how local parameter gradients are expressed as block-output changes. We propose Coherent Predictive Coding (CoPC), which fixes feed-forward reference activations and represents latent states as deviations , yielding so that intermediate inference uses fixed identity coupling rather than learned block Jacobians evaluated at the evolving latent states. After inference, CoPC computes local parameter gradients from the final prediction errors and preconditions each selected linear projection using its uncentered activation second moment to reduce function-space distortion. Across MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, CoPC achieves competitive predictive-coding performance and reaches test accuracy on CIFAR-100, exceeding the strongest predictive-coding baseline in our comparison by 3.25 percentage points. Controlled analyses further show that CoPC produces more consistent latent-state update directions and that second-moment preconditioning improves the alignment between final prediction errors and the induced block-output changes.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
predictive coding, credit assignment, local learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6BGAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on stabilizing the training of Predictive Coding (PC) networks, addressing the specific challenge that such networks often fail to generate meaningful prediction errors within a limited number of inference steps. The authors propose a two-phase approach to PC learning. During the inference phase, instead of making the update of hidden states dependent on a Jacobian matrix recalculated based on the current state, the network first performs a forward pass to obtain reference activations ala_l. The hidden states are then expressed as deviations dl=hl−ald_l = h_l - a_l, such that the inter-layer prediction error is defined as el=dl−dl−1e_l = d_l - d_{l-1}. During the learning phase, after the forward pass, parameter gradients are computed according to local rules and then preconditioned using the inverse of the second moment of activations. This step mitigates the distortion caused by uneven gains across different activation directions.

Critical strengths

The mechanism behind the reduction in cost of inference learning is quite intuitive in principle, and this cost saving has also been confirmed in practice.

Critical weaknesses
  1. The core idea of ​​using a feedforward reference as an anchor for weight updates has already appeared in Qi et al. (Towards the Training of Deeper Predictive Coding Neural Networks, arXiv:2506.23800) as Forward Update. The paper neither cites nor discusses that work.
  2. The analysis in Appendix A indicates that the condition number of the coupling matrix is ​​Θ(L2)\Theta(L^2); this implies that the number of inference steps required to propagate task signals along the network chain grows quadratically with depth. This phenomenon is already clearly observable in Figure 2c: in the ViT-8 model (using FP64 precision), the update increments for layers 1 through 4 fall below the FP64 threshold, meaning that, over the course of K=8K=8 inference steps, the task signal fails to effectively propagate to the first half of the network. Nevertheless, the paper claims to achieve "stable latent-state inference" without addressing this limitation. Future research should incorporate experiments at varying depths (e.g., L∈{16,32}L \in \{16, 32\}) and provide a more rigorous and qualified characterization of what is meant by "stability."
  3. In the experimental section, the shared hyperparameters are overly tuned to favor the proposed method, which ends up suppressing the performance of the BP baseline. In the publicly available code repository, simply adjusting the learning rate (lr) for the BP baseline yields an 11% performance gain.
Questions for authors

See the weaknesses above

4 comments
AuthorsOpenReview ↗

Distinction from Qi et al., task-to-gradient learning, and layerwise signals

CoPC's algorithmic contribution is to use feedforward references to construct fixed difference coupling for inter-layer latent-state inference and to connect the resulting task-dependent local learning signals to blockwise parameter updates. Instead of repeatedly evaluating state- and weight-dependent learned-Jacobian transport, the reference-based differences govern inter-layer propagation. Current-model information enters through task gradients along the inference trajectory and block-specific parameter derivatives.

The Forward Update of Qi et al., Towards the Training of Deeper Predictive Coding Neural Networks (ICML 2026) uses the difference ϵ~l=hl(K)−al\widetilde{\epsilon}_l=h_l^{(K)}-a_l between final inferred states and initial feedforward predictions as a reference for local parameter learning. CoPC instead uses the feedforward reference to define the error structure during inference itself: dl=hl−ald_l=h_l-a_l and el=dl−dl−1e_l=d_l-d_{l-1}. Its final local signal ql(K)=κel(K)q_l^{(K)}=\kappa e_l^{(K)} then enters each block's local gradient. The key difference is that CoPC uses the reference in the inference operator, not only in the subsequent weight-update signal.

We adapted Qi et al.'s core Forward Update rule to the same ViT-8 architecture and compared CIFAR-100 performance after 20 epochs with K=8.

MethodTest accuracy (%)RTX 5090 ms/iterPeak GiB
Forward Update20.14±10.09172.632.741
CoPC27.35±0.7770.701.386

Accuracy: official test with validation-selected settings, mean±sample SD over three independent seeds; runtime/memory: 100 timed iterations on RTX 5090, batch 256. This is a ViT adaptation of the core Forward Update rule; spiking precision, auxiliary neurons, and BatchNorm freezing in the original paper are separate experimental components.

Under the compared conditioned settings, CoPC achieved 2.44× shorter iterations and 49.4% lower peak memory. Isolated inference took 123.06 ms for Forward Update versus 20.97 ms for CoPC; local-gradient computation (15.34/15.38 ms) and additional parameter-gradient processing (24.11/24.53 ms) were similar. The main runtime difference arose during latent-state inference itself. Forward Update's high seed-wise variance includes one run whose late-epoch validation accuracy fell from 28.58% to 8.22%.

Why CoPC can learn is made explicit by the task-to-gradient connection across all layers. Combining Eq. (3) with the synchronous updates in Appendix B.2 yields

gt=∇hLLtask(hL(t)),ql(K)=−B∑t=0K−1wl,tgt,g_t=\nabla_{h_L}\mathcal L_{\mathrm{task}}(h_L^{(t)}),\qquad q_l^{(K)}=-B\sum_{t=0}^{K-1}w_{l,t}g_t,

wl,t=c[(I−cDD⊤)K−1−t]lL,c=ηhκ,w_{l,t}=c[(I-cDD^\top)^{K-1-t}]_{lL},\qquad c=\eta_h\kappa,

Gθlraw=Jθlfl(al−1)⊤∑t=0K−1wl,tgt.G_{\theta_l}^{\mathrm{raw}}=J_{\theta_l}f_l(a_{l-1})^\top\sum_{t=0}^{K-1}w_{l,t}g_t.

This requires fixed references/parameters during inference, d(0)=0d^{(0)}=0, a shared state step with synchronous updates, minibatch-mean task CE, and h(t+1)=h(t)−ηhB∇hFh^{(t+1)}=h^{(t)}-\eta_h B\nabla_h F. The gtg_t are task gradients evaluated along the actual inferred trajectory; GrawG^{\mathrm{raw}} is the detached-signal local gradient before parameter updates. The task loss supplies the signal, fixed transport forms depth- and time-dependent local signals, and each block's parameter derivative maps them into local learning gradients.

For the fixed difference matrix DD, the prediction-error quadratic term has Hessian HPE=(κ/B)D⊤DH_{\mathrm{PE}}=(\kappa/B)D^\top D and condition number Θ(L2)\Theta(L^2): 113.50/249.65/437.70 at depths 8/12/16. In exact arithmetic, task information from the terminal layer can first reach block ll at K≥L−l+1K\ge L-l+1; we also measured its finite-precision magnitude.

Using three selected ViT-8 epoch-20 checkpoints and four fixed training batches per checkpoint, the inference objective decreased in all 12/12 observations at K=8, with mean task-CE change −0.000296±0.000131.

BlockLocal signal normLocal gradient norm
81.300818×10⁻⁴2.503952×10⁻³
71.368023×10⁻⁸8.831425×10⁻¹⁰
68.221517×10⁻¹³5.317081×10⁻¹⁴
52.716061×10⁻¹⁷1.757179×10⁻¹⁸

Block 4 had numerically distinguishable signals in 5/12 observations, whereas both the local signals and gradients of blocks 1–3 were numerically zero in all 12/12 observations. The small coefficient c=ηhκ≈3.00×10−5c=\eta_h\kappa\approx3.00\times10^{-5} and FP64 rounding/cancellation contribute to this attenuation, consistent with the limited effective lower-layer directions in Fig. 2c. Fixed coupling defines a learning path for every block, while the measured K=8 signals are concentrated in output-proximal blocks.

AuthorsOpenReview ↗

Finite-step stability, realized updates, and BP tuning fairness

We separately evaluated finite-step inference stability and the effect of CoPC's local signals on parameter learning. Checkpoints from three training methods and three initialization-seed labels formed nine training histories. With 27 checkpoints (epochs 1/10/20), two training batches each, and three state-step settings, we obtained 162 paired repeated inference conditions per method, rather than 162 independently trained models.

TransportInference-objective decrease after 8 stepsCommon task-CE decrease after 8 steps
CoPC fixed coupling162/162162/162
Reference Jacobian137/162150/162
Gain-normalized Jacobian162/162162/162

For reference-Jacobian transport, the objective decreased in 42/54, 53/54, and 42/54 cases by seed label; gain normalization raised each to 54/54. At the fixed state step α=1/32\alpha=1/32, the counts were 54/54 for CoPC, 45/54 for reference Jacobian, and 54/54 for normalized Jacobian. These paired checkpoint/batch/step-size measurements directly demonstrate how Jacobian-gain control affects finite-step objective reduction. We report each method's inference objective separately from the common task CE.

We then tested realized parameter updates from identical ViT-8 checkpoints and saved AdamW states. Including (ON) or removing (OFF) the supervised task force during inference, we updated the body and classifier once and evaluated CE on separate training batches.

SeedON ΔCEOFF ΔCEON−OFF
200−0.020147+0.065869−0.086016
201+0.001249+0.050801−0.049552
202+0.000319+0.201131−0.200812
Mean±sample SD−0.006193±0.012093+0.105934±0.082787−0.112126±0.078938

ON−OFF was negative for all three seeds, while the head-only difference was approximately 3×10−83\times10^{-8}. This directly establishes the contribution of task-dependent inference signals to local parameter updates in the network body. The one-step comparison incorporates the saved AdamW moments.

We also varied inference steps at the same checkpoints. Mean post-update CE changes at K=8, 16, and 32 were −0.00619, −0.02936, and +0.1565, respectively, although each setting reduced its inference objective. Some intermediate-layer signals grew as K increased, while blocks 1–3 remained numerically zero in the measured finite-precision runs. The inference budget therefore affects not only the depth reached by the signals but also the regime in which parameter updates are effective. At K=8, training only the final block and classifier yielded 27.16±0.30%, versus 27.35±0.77% when all blocks and the classifier were trained (paired full-minus-top +0.19±0.50 pp). Corresponding one-step CE changes were −0.006191 and −0.006193. These results characterize where learning is concentrated at the present inference budget.

We provided BP, Meta-PCN, Standard PC, μPC, and ePC-T with the same outer-LR search budget. On CIFAR-100/ViT-8, each method evaluated five LR candidates for 20 epochs on the first seed, rechecked the top two for 20 epochs on a second seed (140 tuning epochs per method), and reported three independent 20-epoch runs using validation-selected LRs.

MethodShared-LR test (%)Selected LRSelected test (%)Observed Δ (pp)
BP32.11±0.440.000133.80±0.42+1.69
Meta-PCN23.87±0.470.000931.90±0.93+8.04
Standard PC2.67±1.940.000915.98±0.96+13.31
μPC22.80±0.670.002722.80±0.670.00
ePC-T18.77±15.390.002718.77±15.390.00
CoPC†27.12±1.010.008127.35±0.77+0.22

Mean±sample SD (n=3). †CoPC is the final proposed method evaluated under a separately selected configuration.

After matched-budget tuning, BP improved from 32.11±0.44% to 33.80±0.42% (+1.69 pp), Meta-PCN by 8.04 pp, and Standard PC by 13.31 pp. BP's selected LR=0.0001 was the lower search boundary. These results directly address baseline tuning fairness while quantifying CoPC's fixed-transport design, computational savings, and learning behavior.

In depth experiments, CoPC achieved 27.12±1.01%, 26.86±0.46%, and 27.33±0.55% on CIFAR-100 ViT-8/12/16, with GH200 iteration times of 126.08/181.57/239.44 ms. It also reached 79.93±0.70% (n=3) on a non-residual Fashion-MNIST eight-block MLP. CoPC thus links a distinct fixed inference-transport mechanism to explicit task-dependent local gradients, measured finite-step behavior, realized parameter updates, and controlled baseline comparisons.

Dear Reviewer 6BGA,

As the discussion period draws to a close, we would like to emphasize the mechanism and empirical evidence behind CoPC's stable inference. Its fixed quadratic prediction-error structure avoids state-dependent Jacobian amplification, achieving consistent finite-step objective reduction in 162/162 conditions, versus 137/162 for reference-Jacobian transport. We distinguish this stability from uniform credit assignment, explicitly quantify signal attenuation at K=8, and demonstrate consistent accuracy across ViT-8/12/16.

We hope these findings address your concerns and can inform your final assessment.

Thank you for your time and consideration.

  1. “not only in the subsequent weight-update signal” confirmed my initial review.
  2. “both the local signals and gradients of blocks 1–3 were numerically zero in all 12/12 observations.” indicates that blocks 1–3 of the network were not trained at all, retaining only their initial random weights. Consequently, the advantage of this 12-block network compared to a standard 9-block network is highly questionable.
  3. Since the LR=0.0001 meets the lower boundary, further decreasing the LR for BP might continue to improve BP’s performance (the 11% improvement I got for BP was on CIFAR-10 using the provided code). Considering this large (and potentially larger) gap, coupled with the inconsistent performance of the proposed model across different seeds (as noted in the response to other reviewers), I don’t think the result can fully support the claim of the paper. 

Considering all of the above, I will keep my current score.

Reviewer_gg8TOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes CPC, a new predictive coding strategy to train neural networks. The key is latent-space inference that forms the loss and a way to update the parameters. The results beat prior PC-based training methods.

Critical strengths

Training strategies beyond BP are generally of interest.

Critical weaknesses
  • The baseline accuracy seems to be shockingly low; is that really the state of the art in PC training?
  • What main advantages does this have over BP?
  • What prevents you from experimenting with larger models? I realize that one can't do all the experiments, but some explanation would be helpful. Is it because PC just doesn't scale?
Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Why study PC? CoPC's contribution and contextualized comparisons

Predictive Coding (PC) combines latent-state inference with local parameter learning, offering a framework for studying credit assignment beyond network-wide chain-rule backpropagation. Whereas BP propagates final-loss gradients through the network, PC first adjusts latent activities using prediction errors and then updates each block using its local error. Song et al. (Nature Neuroscience, 2024) examined the principle of prospective configuration, in which neural activities are inferred before weight updates, and its advantages in specific learning settings. Salvatori et al. (NeurIPS, 2022) extended PC to general graphs, studying classification and partial-observation reconstruction under different observation conditions. The ability to investigate local learning together with flexible state-based inference is a distinct motivation for PC. Making repeated latent-state inference computationally practical is consequently important for deep networks, as also emphasized by Pinchetti et al. (ICLR, 2025).

CoPC preserves this local-learning paradigm while replacing learned-Jacobian transport and its gain-control overhead with fixed difference coupling during repeated inference. From feedforward references ala_l, we form deviations dl=hl−ald_l=h_l-a_l and adjacent differences el=dl−dl−1e_l=d_l-d_{l-1}. After K inference steps, the local signal ql(K)=κel(K)q_l^{(K)}=\kappa e_l^{(K)} is combined with each block's parameter derivatives. Thus, fixed transport coefficients coexist with model- and task-dependent local learning signals.

For Fashion-MNIST, we compared methods within identical architectures, using 20 training epochs, K=8 for PC, and three independent seeds:

ArchitectureCoPC test (%)PC baseline test (%)BP test (%)
ViT-882.57±0.57Meta-PCN 81.48±1.2985.23±0.74
Residual MLP-885.23±0.05Standard PC 84.45±0.2086.76±0.16
Non-residual MLP-879.93±0.70Standard PC 79.25±0.5886.37±0.41

CoPC exceeded Meta-PCN on ViT-8 by 1.09 percentage points and Standard PC on residual MLP-8 by 0.78 percentage points. The BP–CoPC gap on residual MLP-8 was 1.53 points. On the non-residual MLP, the three CoPC seed accuracies were 79.19/80.04/80.57%, yielding 79.93±0.70% (n=3); the earlier 79.615±0.601% (n=2) was an intermediate aggregation of the same run.

We also re-evaluated the effect of baseline-specific LR selection under a matched search budget. On CIFAR-100/ViT-8, BP, Meta-PCN, Standard PC, μPC, and ePC-T each received five LR candidates and 140 tuning epochs, followed by evaluation of the selected LR in three independent 20-epoch runs.

MethodShared-LR test (%)Selected LRSelected test (%)Observed Δ (pp)
BP32.11±0.440.000133.80±0.42+1.69
Meta-PCN23.87±0.470.000931.90±0.93+8.04
Standard PC2.67±1.940.000915.98±0.96+13.31
μPC22.80±0.670.002722.80±0.670.00
ePC-T18.77±15.390.002718.77±15.390.00
CoPC†27.12±1.010.008127.35±0.77+0.22

Mean±sample SD, n=3. †CoPC denotes the final proposed method, evaluated under a separately selected configuration.

After method-specific LR selection, Meta-PCN improved from 23.87% to 31.90% and Standard PC from 2.67% to 15.98%. We therefore report CoPC's final performance alongside independently tuned PC baselines, while clearly distinguishing its own evaluation configuration.

AuthorsOpenReview ↗

Fixed-coupling learning, efficiency, and scalability

CoPC's task-dependent learning mechanism is explicit at the level of each block. With fixed references and parameters, d(0)=0d^{(0)}=0, a shared state step, synchronous updates, and minibatch-mean task loss,

q_l^{(K)}=-B\sum_{t=0}^{K-1}w_{l,t}g_t,$$ $$w_{l,t} = c\left[(I-cDD^{\top})^{K-1-t}\right]_{lL}, \qquad c=\eta_h\kappa.$$ $$G_{\theta_l}^{\mathrm{raw}} = J_{\theta_l}f_l(a_{l-1})^\top \sum_{t=0}^{K-1}w_{l,t}g_t.$$ **The task loss provides a time-dependent signal; fixed inter-layer coupling transports it into local learning signals; and each block's parameter derivative converts the signal into a local gradient.** The gradients $g_t$ are evaluated along the actual inference trajectory. These locally computed gradients drive parameter updates, separating the fixed transport structure from task-dependent signal content. We compared transport schemes under **162 paired repeated inference conditions**: three training methods × three seed labels (nine histories), 27 checkpoints at epochs 1/10/20, two training batches per checkpoint, and three state-step settings. After eight inference steps, the **inference objective decreased** in 162/162 CoPC, 137/162 reference-Jacobian, and 162/162 gain-normalized-Jacobian conditions; common task CE decreased in 162/162, 150/162, and 162/162, respectively. Gain normalization recovered the reference-Jacobian method's 25 objective-decrease failures. We also assessed the effect of gain restriction on actual parameter updates using checkpoints trained by Meta-PCN. **These interventions distinguish transport amplification during inference from its effect on subsequent learning, explaining the motivation for CoPC's fixed transport structure.** We further tested the effect of the task signal on body updates. Starting from identical models and saved AdamW states, including versus removing the supervised task force during inference yielded mean one-step CE changes of **−0.00619 and +0.10593** on separate training batches, with a paired difference of **−0.11213±0.07894** across seeds. The seed-wise ON−OFF differences were −0.08602, −0.04955, and −0.20081; the classifier-only difference was approximately $3\times10^{-8}$. **This directly demonstrates that the task signal formed during inference affects parameter updates in the network body.** We measured iteration time and memory on the same device within each comparison. On GH200/ViT-8, CoPC required **126.08 ms/iteration**, versus 317.34 ms for Standard PC and 3063.53 ms for Meta-PCN: **2.52× and 24.3× shorter iterations**, respectively. On RTX 5090, CoPC and gain-normalized Jacobian transport obtained **27.35±0.77% versus 27.40±1.47%** test accuracy, at **70.7 versus 2137.2 ms** per iteration: roughly **30.2× shorter** for CoPC. BP took 40.34 ms per iteration on the GH200 ViT-8 benchmark. These results establish substantial savings over Jacobian-based PC methods while retaining a distinct local-learning route from BP. We also evaluated greater depth using the original fixed settings on CIFAR-100 ViT-8/12/16: | Model (parameters) | CoPC test (%) | CoPC ms/iter | CoPC peak GiB | Meta-PCN test (%) | Meta ms/iter | | --- | --- | --- | --- | --- | --- | | ViT-8 (1.61M) | **27.12±1.01** | 126.08 | 2.15 | 23.87±0.47 | 3063.53 | | ViT-12 (2.41M) | **26.86±0.46** | 181.57 | 2.86 | 4.43±1.01 | 4645.83 | | ViT-16 (3.20M) | **27.33±0.55** | 239.44 | 3.49 | 3.14±0.96 | 6225.25 | *Official test: mean±sample SD from three independent seeds after 20 epochs. Runtime: median of 100 iterations after 20 warm-ups on GH200; memory: peak allocated. Original method-specific outer LR and state-step settings were retained.* **Across ViT-8/12/16, CoPC maintained 26.86–27.33% accuracy while requiring 24.3×, 25.6×, and 26.0× less iteration time than Meta-PCN.** Its peak memory was 2.15/2.86/3.49 GiB, versus 3.63/5.12/6.62 GiB for Meta-PCN. Re-selecting Meta-PCN's state step for ViT-12 improved accuracy from **4.43±1.01% to 20.91±0.80%**. On Fashion-MNIST, CoPC achieved **79.93±0.70%** on an eight-block MLP without residual connections (n=3), versus 79.25±0.58% for Standard PC and 86.37±0.41% for BP. With residual connections, the corresponding accuracies were **85.23±0.05%, 84.45±0.20%, and 86.76±0.16%**. **CoPC therefore supports local learning in both deeper Transformers and non-residual MLPs while reducing the inference cost relative to existing PC methods.**

Dear Reviewer gg8T,

As the discussion period draws to a close, we would like to emphasize that PC provides a framework for studying local learning and credit assignment beyond conventional backpropagation, through latent-state inference and locally computed parameter updates. CoPC's contribution is to make this learning paradigm more computationally practical, achieving 24.3× shorter iterations than Meta-PCN, rather than aiming to outperform BP in accuracy.

We hope these clarifications can inform your final assessment.

Thank you for your time and consideration.

Reviewer_uYhPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Coherent Predictive Coding (CoPC), which modifies both latent-state inference and local parameter learning. During inference, CoPC introduces identity coupling, giving the prediction-error term a fixed quadratic structure independent of current-state learned Jacobians. During parameter learning, it applies activation second-moment preconditioning to local gradients computed from final prediction errors.

CoPC demonstrates relatively stable and efficient training on ViT-8 across four image classification benchmarks. While I acknowledge its practical value, I find the learning mechanism underlying identity coupling insufficiently understood and the baseline comparisons not fully convincing. Therefore, my overall recommendation is Weak Reject.

Critical strengths

1. Stable and efficient training in a challenging setting.

Given the known difficulties of training predictive-coding networks, CoPC's relatively stable performance on ViT is encouraging. It also achieves the lowest per-iteration runtime among the compared PC methods. With further mechanistic understanding, CoPC could become a useful approach to training PC-based Transformers.

Critical weaknesses

1. Insufficient mechanistic understanding of identity coupling.

Replacing learned inter-layer Jacobians with identity mappings fundamentally changes how errors propagate across layers. Although the authors show more consistent latent-state update directions, these metrics do not establish whether the resulting signals support effective credit assignment. It remains unclear why identity coupling works and under what conditions it is justified.

2. Concerns about baseline comparison fairness.

The common outer learning rate was selected based on Standard PC and CoPC rather than independently tuned for each baseline. Moreover, ePC-T and Meta-PCN exhibit substantial variability on MNIST, raising concerns about training stability under the adopted setting. While I appreciate the controlled common-recipe comparison, the limited baseline tuning makes it difficult to assess CoPC's advantage over well-optimized existing methods.

Questions for authors

1. Baseline tuning. Could the authors provide a comparison with more thoroughly tuned baselines, particularly Meta-PCN? Even method-specific outer learning-rate tuning would help establish whether CoPC's advantage persists.

2. Architectural generality and scalability. The experiments are limited to ViT-8. Could the authors examine whether the improvements persist across different depths or architectures, particularly non-residual networks where identity coupling may be less appropriate?

3. Contribution of preconditioning. Since second-moment preconditioning is not specific to CoPC, could the authors apply it to competing PC methods and BP? This would help isolate the contribution of identity coupling. Reporting CoPC without preconditioning under the main benchmark protocol would also be valuable.

5 comments
AuthorsOpenReview ↗

Design contribution, task-dependent learning, and parameter updates

CoPC replaces learned-Jacobian inter-layer transport with fixed difference coupling and connects the resulting task-dependent local learning signals to blockwise parameter updates. In Jacobian-based PC, error transport and amplification depend on learned predictors and latent states. CoPC instead uses deviations dl=hl−ald_l=h_l-a_l from feedforward references ala_l and adjacent differences el=dl−dl−1e_l=d_l-d_{l-1}. The transport operator is fixed, whereas task gradients and block-parameter derivatives retain information about the current task and model.

After inference, ql(K)=κel(K)q_l^{(K)}=\kappa e_l^{(K)} defines each block's local learning signal. Connecting Eq. (3) with Eqs. (15)–(16) of Appendix B.2 yields

gt=∇hLLtask(hL(t)),ql(K)=−B∑t=0K−1wl,tgt,g_t=\nabla_{h_L}\mathcal L_{\mathrm{task}}(h_L^{(t)}),\qquad q_l^{(K)}=-B\sum_{t=0}^{K-1}w_{l,t}g_t,

wl,t=c[(I−cDD⊤)K−1−t]lL,c=ηhκ,w_{l,t}=c[(I-cDD^\top)^{K-1-t}]_{lL},\qquad c=\eta_h\kappa,

Gθlraw=Jθlfl(al−1)⊤∑t=0K−1wl,tgt.G_{\theta_l}^{\mathrm{raw}}=J_{\theta_l}f_l(a_{l-1})^\top\sum_{t=0}^{K-1}w_{l,t}g_t.

The derivation assumes fixed references and parameters during inference, d(0)=0d^{(0)}=0, a shared state step with synchronous updates, minibatch-mean task CE, and h(t+1)=h(t)−ηhB∇hFh^{(t+1)}=h^{(t)}-\eta_h B\nabla_h F. It concerns the local gradient computed with a detached local signal, before conditioning and AdamW. Each gtg_t is evaluated along the actual inference trajectory. First, task loss supplies the learning signal; second, fixed coupling constructs local signals through depth- and time-dependent coefficients wl,tw_{l,t}; third, each block's parameter derivative converts its signal into a local parameter gradient. Thus, every block has a task-to-gradient path; its effective finite-step distribution is assessed separately.

We tested this path through parameter updates from matched initial states. With the same trained ViT-8 and saved AdamW state, we made one body-and-classifier update with (ON) or without (OFF) the supervised force during inference, updating the classifier in both cases. We evaluated CE on separate training batches, disjoint from the update batches.

SeedTask ON ΔCETask OFF ΔCEON−OFF
200−0.020147+0.065869−0.086016
201+0.001249+0.050801−0.049552
202+0.000319+0.201131−0.200812
Mean±sample SD−0.006193±0.012093+0.105934±0.082787−0.112126±0.078938

For all three seeds, retaining the task force produced lower post-update CE than removing it (paired difference −0.11213±0.07894). The head-only ON−OFF difference was about 3×10−83\times10^{-8}. This directly demonstrates the supervised signal's contribution to body-block updates. The 24 repeated observations (3 seeds × 4 update batches × 2 separate training probes) were aggregated by seed. These one-step results include conditioning and saved optimizer moments.

We also tested fixed transport under matched initial conditions. Checkpoints from three training methods and three initialization-seed labels formed nine training histories. Twenty-seven checkpoints (epochs 1/10/20), two batches each, and three state-step settings yielded 162 paired repeated observations per method. The respective inference objective decreased after eight inference steps in 162/162 CoPC, 137/162 reference-Jacobian, and 162/162 gain-normalized-Jacobian cases; common task CE decreased in 162/162, 150/162, and 162/162. Normalization recovered all 25 reference-Jacobian objective-decrease failures. The reference-Jacobian counts by seed label were 42/54, 53/54, and 42/54. These are repeated conditions, not 162 independent training runs.

On Meta-PCN-trained checkpoints, gain restriction changed one-step body-update CE from −0.0018 to +0.0713 without conditioning and +0.0112 to +0.0460 with conditioning. Gain control therefore has distinct effects on inference-objective reduction and local parameter learning. Fixed transport eliminates Jacobian construction and gain-estimation costs. On an RTX 5090, CoPC and gain-normalized Jacobian transport obtained 27.35±0.77% and 27.40±1.47% CIFAR-100 accuracy at 70.7 and 2137.2 ms per iteration, respectively: approximately 30.2× shorter iterations for CoPC.

AuthorsOpenReview ↗

MNIST variability, LR fairness, conditioning, and generality

MNIST's large seed-to-seed variance reflects sharp degradation in individual training trajectories. We verified epoch-wise training/validation logs, six checkpoint hashes, and official fixed-epoch-20 test results.

MethodSeed 200Seed 201Seed 202Mean±sample SD
ePC-T34.4887.4785.7769.24±30.12%
Meta-PCN88.9136.1789.6871.59±30.67%

For ePC-T seed 200, validation accuracy fell from 72.05% (epoch 7) to 9.77% (epoch 10), while training CE increased from 0.770 to 2.307 between epochs 7 and 9. Meta-PCN seed 201 declined from 57.80% (epoch 2) to 27.62% (epoch 4). Other seeds reached roughly 85–90%. The reported SDs therefore reflect observed deterioration in training and validation, not an aggregation artifact. All logged minibatch training CE values were finite. MNIST state-step searches tested five candidates (ePC-T: 0.075–1.2; Meta-PCN: 0.1–1.0), for two epochs each on seed 199, selecting 0.075 and 0.8; outer LR was 0.0027. Crossed experiments could separate initialization, data-order, LR, and state-step effects. In the same MNIST protocol, CoPC achieved 92.44±0.17%, versus 95.77±0.22% for BP.

We quantified method-specific LR selection and gradient conditioning in separate controlled comparisons. On CIFAR-100/ViT-8, BP, Meta-PCN, Standard PC, μPC, ePC-T, and CoPC without conditioning received the same five LR candidates and 140 tuning epochs per configuration. Five LR candidates were trained for 20 epochs on one seed, and the top two on a second; selected LRs were evaluated in three independent 20-epoch runs.

MethodShared-LR test (%)Selected LRSelected test (%)Observed Δ (pp)
BP32.11±0.440.000133.80±0.42+1.69
Meta-PCN23.87±0.470.000931.90±0.93+8.04
Standard PC2.67±1.940.000915.98±0.96+13.31
μPC22.80±0.670.002722.80±0.670.00
ePC-T18.77±15.390.002718.77±15.390.00
CoPC†27.12±1.010.008127.35±0.77+0.22

Official test mean±sample SD across three seeds. †CoPC is the final method with second-moment conditioning.

At LR=0.0081 with matched seeds and data order, CoPC without conditioning achieved 24.18±0.73% versus 27.35±0.77% with conditioning, a paired gain of +3.16±1.50 pp. At shared LR=0.0027, conditioning changed Standard PC from 2.67% to 4.62%, BP from 32.11% to 30.53%, and Meta-PCN from 23.87% to 20.69%. Second-moment preconditioning gives an additional accuracy gain when combined with CoPC's fixed-coupling local gradients.

We evaluated depth with the original outer LR and each method's state-step settings, K=8 for PC, 20 epochs, and three independent seeds.

DepthMethodTest accuracy (%)GH200 ms/iterPeak GiB
ViT-8CoPC27.12±1.01126.082.15
Meta-PCN23.87±0.473063.533.63
Standard PC2.67±1.94317.343.43
BP32.11±0.4440.341.42
ViT-12CoPC26.86±0.46181.572.86
Meta-PCN4.43±1.014645.835.12
Standard PC1.10±0.17484.634.69
BP30.16±2.9360.212.79
ViT-16CoPC27.33±0.55239.443.49
Meta-PCN3.14±0.966225.256.62
Standard PC3.50±3.52664.145.95
BP26.28±9.3581.263.51

Time: median of 100 iterations after 20 warm-ups on the same GH200; memory: peak allocated.

Across the three depths, CoPC maintained 26.86–27.33% accuracy with shorter iteration times and lower peak memory than Standard PC and Meta-PCN. Relative to Meta-PCN, iterations were 24.3×, 25.6×, and 26.0× shorter for ViT-8/12/16. BP was faster. Re-selecting the state step for Meta-PCN ViT-12 improved accuracy from 4.43±1.01% to 20.91±0.80%.

On Fashion-MNIST eight-block MLPs, CoPC obtained 79.93±0.70% without residual connections and 85.23±0.05% with them. Standard PC obtained 79.25±0.58%/84.45±0.20%, and BP 86.37±0.41%/86.76±0.16%. Non-residual CoPC seed accuracies were 79.19/80.04/80.57% (n=3); the earlier 79.615±0.601% (n=2) was an intermediate aggregation of the same run.

We interpret the location of effective learning after establishing the design and its update effect. At K=8, training only the final block and classifier yielded 27.16±0.30%, versus 27.35±0.77% with all blocks and the classifier (paired +0.19±0.50 pp). Together with task-dependent paths defined for all blocks, this indicates a concentration of effective updates in output-proximal blocks under the current finite inference budget.

Thank you for the extensive additional experiments. However, I find the method-specific learning-rate tuning results particularly concerning.

CoPC's superior CIFAR-100 performance over existing PC baselines is presented as a central empirical contribution, even highlighted in the abstract. However, the new results show that CoPC is substantially outperformed by Meta-PCN after proper tuning (27.35% vs. 31.90%).

I consider this a serious concern regarding the original paper's central empirical claim. Considering that the manuscript has not been revised to reflect these contradictory findings, I am lowering my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.