acceptodds
Under review as a conference paper at ICLR 2027

Cached Boundary Transport for Efficient Activity Inference in Deep Predictive Coding

Abstract

Predictive coding (PC) networks infer their hidden activities before every weight update, and in deep networks this inner optimization becomes the bottleneck of training. We show that the difficulty is geometric rather than iterative: the activity landscape contains low-curvature directions that span the whole layer chain, and a roughly 28-fold larger gradient budget leaves the test accuracy of 256-layer networks unchanged. A residual PC chain, however, supplies its own solver for these directions. Linearizing the predictions at the feedforward state yields a batch-shared Gauss–Newton preconditioner whose inverse is applied exactly by one backward and one forward recurrence over the layers, plus a small cached output-space correction. Cached Boundary Transport (CBT) uses this solve for the chain-wide component and a short gradient-descent refinement for the remaining local error; an energy acceptance criterion and a restricted Gauss–Newton fallback handle nonlinear mismatch. From 16 to 256 layers, CBT keeps inference cost nearly constant in gradient evaluations while first-order and Newton-type solvers grow with depth. On trained networks it brings the parameter gradient substantially closer to that of an exact second-order oracle, and in end-to-end training it is more accurate than depth-proportional inference at every tested depth with up to fewer FLOPs. These results point to chain geometry, not the step budget, as the bottleneck of deep residual PC inference.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Predictive Coding, Activity Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dMtvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Cached Boundary Transport (CBT), which can stabilize and accelerate activity inference in deep Predictive Coding networks using a Gauss-Newton-like preconditioner. CBT efficiently applies the inverse of a structured preconditioner without explicitly making or inverting the full matrix, thereby reducing inference energy at a relatively low computational cost.

Critical strengths
  1. The paper is clearly written, and the figures are easy to understand.
  2. The authors propose a structured preconditioner that accelerates activity inference in nonlinear PC networks. To the best of my knowledge, this is among the first approaches to explicitly use curvature information for efficient activity inference in nonlinear PC networks.
Critical weaknesses

Concerns about the model structure

Even with CBT, test accuracy tends to decrease as model depth increases on the tasks in this paper. Although the paper aims to address the training bottleneck in deep PC networks, it does not evaluate CBT in tasks where deeper networks actually provide a performance advantage.More importantly, when deeper networks offer no performance benefit, it is difficult to determine whether CBT successfully optimizes the entire deep network or whether some layers remain poorly optimized, effectively reducing the model to a shallower network. In the latter case, the observed accuracy could be achieved without successfully overcoming the optimization difficulties by depth since skipping training for some layers can improve the performance.

Additionally, Figure 10 shows that increasing network width improves accuracy. Combined with the depth-scaling results, this suggests that the architectures considered in the experiments are too deep and narrow for this task and the model structure is not natural. Figure 10 shows that the wider the width, the smaller the gap between CBT and GD-K. If you make the width large enough, you might not even need the proposed method.

Is the low-curvature component really a bottleneck?

Appendix A.1 use a very simple model (scalar weights and does not have activations) and this only motivates the claim that low-curvature eigenvectors span multiple layers. Thus, it is unclear whether the same low-curvature structure persists in practical training and is really a bottleneck of the training.

Questions for authors
  1. Why is the test accuracy of the MLPs on MNIST relatively low? In Figure 3, the test accuracy remains below 94%, whereas [1] reports over 98% accuracy using a three-layer MLP with a hidden width of 128. It would be helpful to clarify why this difference arises.
  2. Previous studies have shown that, in linear PC networks, increasing the number of inference iterations leads to convergence toward Gauss-Newton-like targets (e.g., Theorem 4.2 in [2] and Theorem 1 in [3]). How does the Gauss-Newton reference matrix defined in Equation 7 relate to these findings in the linear network setting? For clarity, it would be helpful to explicitly discuss the relationship between CBT and the Gauss-Newton-like targets obtained through iterative PC inference in previous work.
  3. Following the previous question, sufficiently many inference iterations in linear PC networks can recover Gauss-Newton-like targets. The paper contrasts geometric and iterative approaches and claims that the difficulty is "geometric rather than iterative." However, I am wondering whether sufficiently many iterations (which might be beyond the inference iterations tested in this paper) can recover the geometry. Could you clarify why just increaseing the inference iterations can not recover the geometry?
  4. Line 293 states that this paper use float64. How sensitive is CBT to numerical precision? Does it remain stable when using float32 or bfloat16?
  5. Why does the paper emphasize low-curvature components as the primary source of difficulty in activity inference? Could the observed benefits of CBT also arise from modifying high-curvature directions? In gauss-newton optimization of neural network training, low-rank approximations often preserve dominant high-curvature directions instead of low-curvature directions and damping removes the low-curvature directions in curvature. It would be helpful to explain why focusing on low-curvature components is particularly important in the PC setting.
  6. In Figure 10, does CBT maintain its advantage over GD when network width is further increased to more commonly used values, such as 1024?
  7. Given that increasing width improves accuracy in the reported experiments, why do the authors focus on increasing depth rather than width? The evaluated models appear to be deeper and narrower than architectures usually used in standard backpropagation-based experiments. It would be helpful to justify these choices of model structure and discuss how they affect the advantages of CBT.
  8. The GD-K baseline uses one inference step per hidden layer. However, this choice is not necessarily optimal compared with sequential inference [4], which can reduce the number of inference iterations required to propagate errors through the network. Does the paper use simultaneous or sequential inference for its GD-K baselines? It would be helpful to clarify this choice and discuss whether CBT's computational advantage persists when compared with sequential inference using an appropriately chosen inference budget.

Minor Questions

  1. Line 318 states that a batch size of eight is used, whereas the training experiments use a batch size of 64. Is this reduction primarily due to the computational cost of the exact-Hessian Newton baseline? Could the batch size affect the observed performance of CBT, given that its preconditioner uses batch-averaged Jacobians?

[1] Pinchetti et al. "Benchmarking Predictive Coding Networks -- Made Simple." ICLR 2024.
[2] Ishikawa et al. "Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation." ICLR 2025.
[3] Innocenti et al. "Only strict saddles in the energy landscape of predictive coding networks?" NeurIPS 2024.
[4] Alonso et al. "Understanding and improving optimization in predictive coding networks." AAAI 2024.

4 comments

We thank the reviewer for finding the paper clearly written and the figures easy to understand, and for recognizing CBT as among the first approaches to use curvature explicitly for efficient activity inference in nonlinear PC networks. Each question below is answered with a new measurement. Unless stated otherwise, the setting is the paper's: MNIST, residual tanh PC networks of width NN and depth KK, Adam 10−310^{-3}, 3 seeds; accuracies are test accuracy in %, differences are in percentage points, and compute is total training compute to the stopping epoch in gradient evaluations (GE).

Weakness 1a: No task on which depth helps. Agreed. In our setting BP is also flat in depth, so the benchmark measures whether a method keeps its accuracy as depth grows. N=64N = 64, test accuracy (%):

MethodK=32K = 32K=256K = 256Change (points)
BP96.1796.33+0.2
CBT93.7892.68−1.1
GD-CM90.9087.64−3.3
Sequential inference, compute-matched94.7789.57−5.2

The revision names a task on which depth helps as the next benchmark (Section 5).

Weakness 1b: Are all layers trained? We checked this directly on the trained networks (N=64N = 64, K=128K = 128, i.e. 129 weight matrices; Appendix R). Moved: the relative change ∥W−W0∥/∥W0∥\|W - W_0\| / \|W_0\| from initialization exceeds 1%. Reset: the first 10% or 50% of the matrices, counted from the input, are set back to their initial values, and we measure the change in validation accuracy (points, mean over seeds).

CBTGD-CMBP
Matrices moved (of 129)12911129
Validation accuracy, trained (%)92.1587.6996.15
Change after resetting the first 10%−78.10.0−43.7
Change after resetting the first 50%−78.60.0−64.5

CBT trains and uses its lower layers, as BP does. GD-CM does not, because its 10 simultaneous GD steps per update move the output error only about one layer per step: the prediction errors, and hence the weight updates, never reach below the top ~11 layers (≈3.4T\approx 3.4\sqrt{T} layers for TT steps, Appendix R). It still reaches 87.7% because every layer has an identity skip, so the input representation passes through the untrained random layers to the trained top layers, and on MNIST a few trained layers on top of it already classify reasonably well. This shallow solution is also why GD-CM stays 4.5 points below CBT and 8.5 below BP at this depth.

Weakness 1c and Question 6: Width. K=128K = 128, test accuracy (%), and GD-CM's total compute relative to CBT's:

NNCBTGD-CMGD-CM compute / CBT compute
6492.3888.192.8×
12893.6492.212.8×
25695.0394.372.3×
51295.1295.422.3×
102494.9396.473.35×

CBT's compute advantage holds at every width. At N=512N = 512 and 1024, GD-CM still moves only its top 9–10 of 129 matrices; on MNIST, a wide random stack with a few trained top layers already classifies well. CBT and BP train all 129. CBT's accuracy advantage therefore belongs to narrow, deep chains, where the lower layers are trained only if inference reaches them. Appendix F reports these widths. With the Adam rate tuned separately for each method (N=64N = 64, K=128K = 128), CBT stays ahead on every seed (mean +0.72 points) and reaches a stable 90% validation accuracy with 0.22× GD-CM's compute.

Question 7: Why depth rather than width. The paper studies how PC inference scales with depth, the obstacle that μPC documented. Narrow, deep chains are where inference is hardest and where solvers differ most. We agree these networks are deeper and narrower than one would choose for MNIST with BP. Section 4.3 now states that CBT's accuracy lead is largest in such narrow, deep chains.

Weakness 2 and Question 5: Is low curvature the bottleneck in practice, and why low rather than high? CBT applies the full reference inverse, so it rescales every direction; low curvature matters because it is what limits first-order inference. GD's stable step is set by the largest curvature, so high-curvature error disappears in a few steps (which is why m=3m = 3 refinements suffice), while low-curvature error decays slowly. Unlike Gauss–Newton in network training, where the stochastic curvature estimate is unreliable in low-curvature directions and is truncated or damped there, the curvature of the linearized PC chain is exact and positive definite, so it can be inverted in full.

On trained networks (N=64N = 64, K=128K = 128, 3 seeds), first-order inference with CBT's compute removes little of the error:

No inference (feedforward)GDHeavy ballCBT
Share of the exact solution's energy reduction0%5.6%12.8%48.7%
Parameter-gradient cosine to the exact solution's0.340.340.350.58

GD, heavy ball and CBT receive the same compute. GD and heavy ball leave the parameter gradient where the feedforward pass, with no inference at all, left it (0.34); CBT raises it to 0.58.

To see whether CBT's gain comes from low or high curvature, we split CBT's direction in each sample's exact-Hessian eigenbasis at λ<0.69\lambda < 0.69, the modes where 3 GD steps remove less than half the error (Appendix Q). On CBT-trained networks, the low-curvature part alone reproduces CBT's energy reduction and gradient cosine, while the high-curvature part alone reduces the energy no more than the 3 GD steps on their own.

Question 2: Relation to Gauss–Newton-like targets. In a linear PC network the Jacobians are constant, so HrefH_{\mathrm{ref}} (Eq. 7) is the exact activity Hessian, and one unit chain step from the feedforward state lands exactly on the inference equilibrium (relative gap ≤8.4×10−19\le 8.4 \times 10^{-19}, Appendix A.2). [2, 3] analyze this equilibrium in deep linear networks: [3] shows that the equilibrated energy is a rescaled MSE, and [2] that the resulting weight gradient lies between first-order and Gauss–Newton-like updates, depending on the parameterization. Iterative inference reaches it only at convergence; CBT computes it in closed form, at a cost constant in depth. In nonlinear networks the same formula is a Gauss–Newton step for the chain linearized at the feedforward state. Section 3.2 now discusses this and cites [2, 3].

Question 3: Can enough iterations recover the geometry? Yes; the geometry decides how many iterations are needed.

Linear residual chains (width 8, 3 seeds; Appendix A.2), cost in gradient evaluations (GE) to reach the minimum (GD and heavy ball: to a 10−610^{-6} relative energy gap, with oracle-tuned steps; CBT: one chain step, exact):

KK163264128256
GD3421,2174,42916,281> 20,000
Heavy ball4585160306592
CBT3.333.413.463.483.49

In training (N=64N = 64, Appendix L), heavy ball needs 128 steps per update, at 9.5× CBT's total compute, to pass CBT at K=64K = 64 (test accuracy 93.28 vs 92.64%). At K=128K = 128, even 256 steps at 17.5× CBT's compute stay below CBT (91.56 vs 92.38%). The revision therefore keeps "geometric" and replaces "rather than iterative" with the cost statement: first-order steps resolve these directions only with a budget that grows with depth.

Question 8: Simultaneous or sequential inference. GD-K, GD-CM and heavy ball use simultaneous inference (all layers updated in parallel); Section 4 now states this. We added sequential inference [4] with CBT's per-update compute (10 sweeps), its step selected on a held-out seed (0.5 from {0.1, 0.3, 0.5, 1.0}). Each cell is test accuracy (%) / total compute to the stopping epoch, in thousands of GE (Appendix K):

NN, KKCBTSequential (10 sweeps)GD-CM
64, 3293.78 / 158k94.77 / 458k90.90 / 607k
64, 6492.64 / 135k91.52 / 453k89.31 / 515k
64, 12892.38 / 170k90.28 / 424k88.19 / 473k
64, 25692.68 / 229k89.57 / 467k87.64 / 517k
256, 12895.03 / 184k95.56 / 399k94.37 / 416k

Sequential inference improves on simultaneous GD, but it still loses 5.2 points from K=32K = 32 to 256, while CBT loses 1.1. From K=64K = 64 on, CBT is 1.1–3.1 points more accurate, and in every setting it uses 2.0–3.4× less compute.

Question 1: Why below 94%? Most of the gap to [1] comes from the architecture. In our exact setting (32–256 residual μPC tanh layers of width 64), BP reaches 96.15–96.36%, against the 98% of [1]'s 3-layer width-128 MLP. At width 512 (K=128K = 128), CBT reaches 95.12% and BP 97.66% test accuracy. Appendix K reports BP next to the PC solvers.

Question 4: float32 and bfloat16. CBT trains in float32 with its solver and its 10−1010^{-10} energy tolerance unchanged. All runs finish with no non-finite state. N=64N = 64, test accuracy (%) (Appendix O):

KKCBT float32CBT float64GD-CM float32
12893.3992.3888.40
25691.7292.6887.56

Every float32 CBT seed is above every float32 GD-CM seed. We did not test bfloat16, and the revision says so.

Question 9: Batch size. (a) Yes. Section 4.1 runs a per-sample exact-Hessian Newton reference to convergence at K=256K = 256, and batch eight is the size whose cost fit our budget. Training uses batch 64. (b) N=64N = 64, K=128K = 128, Adam 10−310^{-3} for both methods (not rescaled with batch size). Test accuracy (%), lead in points, and total compute in thousands of GE (Appendix N):

BatchCBTGD-CMLeadCompute CBT / GD-CM
1692.8590.05+2.80743k / 1935k
6492.3888.19+4.19170k / 473k
25691.4985.97+5.5247k / 142k

The lead grows with batch size, so averaging the Jacobians over a larger batch does not weaken the reference.

We thank the reviewer again for the precise and constructive questions. The new results (width, sequential inference, iteration ladder, precision and batch size) make the scope and the strength of CBT explicit. We hope they address your concerns, and we would be grateful if you would consider raising your score.

Thank you for the additional experiments and detailed comments. Your responses have clarified my concerns about whether all layers are trained and whether more inference iterations can recover the geometry.

However, I remain concerned that CBT's accuracy advantage disappears as width increases, with GD-CM outperforming CBT at larger widths. While the method is interesting, its practicality remains unclear to me.

I therefore maintain my original score.

Reviewer_hZSmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on accelerating the internal optimization process within deep predictive coding (PC) networks. In PC networks, each weight update is after an "inference" phase—an internal optimization problem where hidden states across layers are adjusted to minimize prediction error. As the network depth increases, this internal optimization becomes a training bottleneck. The authors argue that this bottleneck is fundamentally geometric in nature, rather than a result of an insufficient number of iterations. Their proposed solution, CBT, freezes a set of shared Jacobian matrices during the forward pass to construct a Gauss–Newton preconditioner. Crucially, the inverse of this preconditioner can be applied without explicitly forming a large matrix; it requires only one backward pass, one forward pass, and a minor correction based on the output dimensionality. Combined with a few steps of gradient-based refinement and an energy-based acceptance criterion, the inference cost—in terms of gradient equivalence—remains nearly constant regardless of network depth. In end-to-end training, this approach outperforms baselines while requiring up to 17x fewer FLOPs.

Critical strengths
  1. This paper identified that the slow inference in deep predictive coding networks stems from "chained" low-curvature modes and proposed the solution.
  2. It raised a new possibility that PC inference no longer requires iterative updates that scale with network depth.
Critical weaknesses
  1. Limited scale and architecture; claims of universality far exceed the supporting evidence. All evidence supporting the assertion that "chain geometry, not the step budget, is the bottleneck of deep residual PC inference" is derived from MNIST-level experiments (For CIFAR-10, the result of only 40.75% is a performance level far away from any realistic task.)
  2. The core contribution of CBT—"chain-solve"—is evaluated only as an atomic, monolithic entity. The paper lacks any component-level ablation studies to determine which specific parts are actually driving performance. The backward recursion is structurally equivalent to the error signal in BP, which ZIL PC is isomorphic to, The paper fails to acknowledge this connection. Consequently, the most natural baseline—ZIL followed by gradient descent refinement—is omitted from the comparisons. Also, the lack of a comparative perspective against BP makes it difficult to assess the significance of this improvement regarding core challenges in the field.
Questions for authors
  1. How would the block of transfering this method to convolutional or attention layers, and what is the expected additional cost?
  2. Is it possible to conduct an ablation study within the "chain-solve" module and compare the performance of CBT versus BP?
  3. In the paper, the same set of hyperparameters is used for CBT across all experimental settings. Was there a specific reason for this? It is well known that Predictive Coding is quite sensitive to hyperparameters; therefore, different architectures might require distinct hyperparameter configurations. In my view, a fair comparison would necessitate conducting a proper hyperparameter search for each method on each dataset, rather than evaluating all methods using a single, fixed set of hyperparameters. Furthermore, would it be possible to provide an analysis regarding the sensitivity of the model to these hyperparameters?
4 comments

We thank the reviewer for the careful review, and for recognizing that the slow inference of deep predictive coding (PC) networks comes from chained low-curvature modes, and that CBT makes it possible for PC inference to stop needing iterative updates that scale with depth. The revision adds every experiment you asked for: the BP and Z-IL comparisons, a component ablation, a hyperparameter study and a convolutional network. All new runs use the paper's training protocol. Unless stated otherwise: residual tanh PC networks of width NN and depth KK, Adam 10−310^{-3}, 3 seeds; accuracies are test accuracy in % (mean ± s.d.), differences are in percentage points, and compute is total training compute to the stopping epoch in gradient evaluations (GE).

Weakness 1: Scale, architecture and scope of the claims. We agree that the main evidence is at MLP scale. The revision limits every claim to the residual PC networks and datasets we evaluate, and removes the universal phrasing.

Why the CIFAR-10 accuracy is low. The 40.75% comes from the architecture, not from PC or CBT. The network is the deep residual MLP family that μPC uses to scale PC in depth: here 64 fully connected tanh layers of width 128 on the flattened image, with no convolutions, no data augmentation, and plateau stopping. Such a network has no spatial inductive bias, so even BP, trained in exactly this network, reaches only 47.46%.

Why the comparison is still fair. All methods train the identical network, from the same initialization, on the same data split and minibatch order, with the same optimizer and stopping rule; only the inference solver differs. The architecture therefore caps every method equally, and the gap between them measures the solver. CIFAR-10 residual MLP (N=128N = 128, K=64K = 64):

MethodTest accuracy (%)Total compute relative to CBT
CBT40.75 ± 0.381.0×
GD-CM37.57 ± 0.962.68×
BP (reference)47.46 ± 0.27—

Within this fixed architecture, CBT leads GD-CM on every paired seed and closes about a third of GD-CM's gap to BP, with 2.68× less compute. With a convolutional architecture (Question 1), accuracy rises for every method (BP reaches 49.5% in that network).

Question 1: Convolutional and attention layers. Setting. CIFAR-10. A 3×33\times3 stride-2 stem maps the image to 32 channels at 16×1616\times16 (8192 units per layer), followed by 32 residual 3×33\times3 convolutional tanh layers. Training is float32 for 60 epochs, seeds 0–2, and we report the best validation accuracy. CBT's solver constants are unchanged.

Two changes (Section 4.3; Appendix S gives the equations):

  1. Matrix-free solve. The recurrences only need products with JJ and J⊤J^\top. For a convolution, Jv=v+s W∗(aˉ⊙v)J v = v + s\, W \ast (\bar a \odot v) is one convolution and J⊤u=u+s aˉ⊙(W⋆u)J^\top u = u + s\, \bar a \odot (W \star u) is one transposed convolution. No N×NN \times N Jacobian is formed. The result matches the dense solver to 10−1010^{-10}, and the worst-case update costs 17–20 GE. The added cost of each product is one pass of the layer itself.
  2. Branch scale 1/(fan-in L)1/(\sqrt{\text{fan-in}}\, L) instead of μPC's 1/fan-in⋅L1/\sqrt{\text{fan-in} \cdot L}, with LL the number of weight layers. This keeps the chain's Jacobian gain near MLP levels, so the batch-shared reference stays faithful. The scale was chosen on a held-out seed. It does not handicap GD-CM, which reaches 45.23% with it and 44.73% with μPC's scale.

Results. Best validation accuracy (%); GD-CM takes 17 steps per update, matched to CBT's worst-case cost:

Seed 0Seed 1Seed 2Mean
CBT44.7043.5246.2644.83
GD-CM44.8245.6245.2645.23
CBT compute / GD-CM compute to reach 42%0.48×0.64×0.43×

CBT reaches the same final accuracy and needs 36–57% less compute to reach 42% on every seed.

Attention. The Jacobian of softmax attention depends on the activities and couples tokens, so one batch-shared linearization is a weaker reference. Each product would cost one attention forward or backward pass. We have not tested this, and Section 5 names it as the next step.

Weakness 2 and Question 2: Connection to BP and Z-IL, the Z-IL + GD baseline, and the ablation.

(a) The connection. You are right, and Section 3.2 now states it. At the feedforward state only the output error is nonzero, so the backward recurrence q=L−⊤gq = L^{-\top} g (Eq. 8) carries the output error down the chain as BP does, through batch-mean Jacobians. With per-sample Jacobians, qq is exactly the BP error, and zff−qz_{\mathrm{ff}} - q is the state at which Z-IL makes its BP-equal weight updates; unit tests check both identities. CBT then applies the forward recurrence and the output correction. Together they give d=Href−1gd = H_{\mathrm{ref}}^{-1} g, a Gauss–Newton step for the whole chain, rather than the BP error placed at every layer.

(b) Z-IL + GD and BP (Appendix K). MNIST, N=64N = 64, test accuracy (%):

KKCBTGD-CMZ-IL + 3 GDBP (reference)
3293.78 ± 0.2490.90 ± 0.4090.44 ± 1.5696.17 ± 0.33
6492.64 ± 0.4689.31 ± 0.2086.82 ± 1.7496.36 ± 0.19
12892.38 ± 0.4488.19 ± 0.7084.18 ± 2.5096.15 ± 0.17
25692.68 ± 0.4687.64 ± 0.5883.80 ± 1.1496.33 ± 0.24

Z-IL + GD is below GD-CM at every depth and 3.3–8.9 points below CBT, and the gap grows with depth. The reason shows on 12 trained K=128K = 128 checkpoints. After Z-IL + GD the PC energy is higher than at the feedforward state (energy quality −0.15 ± 0.24, where 1 is the exact solution), whereas CBT lowers it (0.48 ± 0.23). The BP sweep approximates BP, but it does not solve PC inference. BP is the accuracy reference. CBT is a solver for the PC inference problem, so the relevant question is how much of PC's gap to BP it closes. At every depth, CBT roughly halves the gap between compute-matched PC and BP (by 47–58%; at K=256K = 256, from 8.7 to 3.7 points). It also stays closest to BP as depth grows: from K=32K = 32 to 256, CBT loses 1.1 points, GD-CM 3.3 and Z-IL + GD 6.6.

(c) Ablation inside the chain solve (Appendix P). End-to-end training on MNIST, N=64N = 64, one component removed at a time; the fallbacks are kept, and each arm is charged its own FLOPs. Test accuracy (%):

ArmK=64K = 64K=128K = 128K=256K = 256
CBT92.64 ± 0.4692.38 ± 0.4492.68 ± 0.46
No output correction91.69 ± 0.7891.99 ± 0.3892.64 ± 0.48
Backward recurrence only (BP-like), scale 0.285.48 ± 1.6183.11 ± 0.6384.21 ± 2.62
Backward recurrence only, unit scale90.76 ± 0.6085.91 ± 1.2882.29 ± 0.98
No GD refinement91.92 ± 0.3892.65 ± 0.5993.02 ± 0.42
GD-CM89.31 ± 0.2088.19 ± 0.7087.64 ± 0.58
  • The forward recurrence carries CBT's accuracy. Keeping only the BP-like backward recurrence costs 6.5–10.4 points at K≥128K \ge 128, and every seed of both backward-only arms is below every CBT seed.
  • The output correction and the refinement are safeguards. Removing either changes accuracy by less than 1 point, but the fallbacks then run more often. Without the correction, for example, the plain-GD fallback fires on 2–36% of updates at K=256K = 256; full CBT never reaches it.
  • The restricted Gauss–Newton fallback protects nonlinear networks (Section 4.4). Removing it lowers SiLU validation accuracy by 26.9 points.

Question 3: One configuration for all settings, and sensitivity. Why one configuration. Our networks use the μPC parameterization (Innocenti et al., 2026), which is designed so that weight and activity learning rates transfer zero-shot across widths and depths. We therefore follow μPC and keep one configuration at every depth for all methods: one Adam rate, one activity step for GD and heavy ball, and one set of CBT constants. For CBT, a fixed configuration is also what makes its cost constant in depth: the number of refinement steps mm does not grow with KK, and tuning it per depth would turn it back into a depth-dependent budget. The constants were chosen on MNIST during development and then frozen, and CIFAR-10 was not used in development. The same constants transfer without retuning to Fashion-MNIST (test accuracy +3.38 points over GD-CM), the CIFAR-10 MLP (+3.18 points), SiLU networks and the convolutional network.

Baselines are tuned. On a held-out seed (MNIST, N=64N = 64, K=128K = 128), we selected the activity step over {0.1, 0.3, 0.5, 1.0} on validation accuracy only. Step 0.3 is best for both GD-CM (88.6% vs 86.5 / 83.2 / 64.0%) and heavy ball. Sequential inference had its step selected the same way.

CBT sensitivity (Appendix M). MNIST, N=64N = 64, K=128K = 128, one constant changed at a time; test accuracy (%), default 92.38:

ConstantLowerDefaultHigher
Chain-step scale α0\alpha_0 (0.1 / 0.2 / 0.4)92.5892.3891.13
Refinement steps mm (1 / 3 / 6)92.3792.3892.29
Refinement step η\eta (0.15 / 0.3 / 0.5)92.7192.3892.62

Five of the six changes stay within 0.33 points of the default.

Outer learning rate, tuned per method. MNIST, N=64N = 64, K=128K = 128, test accuracy (%), mean over 3 seeds:

Adam rate3×10−43\times10^{-4}10−310^{-3}3×10−33\times10^{-3}10−210^{-2}3×10−23\times10^{-2}
CBT91.8592.3892.9293.2992.70
GD-CM85.4088.1991.2292.1992.57

Each method's rate is selected on validation (bold). At these rates CBT is ahead on every seed (mean +0.72 points), and it reaches a stable 90% validation accuracy with 0.22× GD-CM's compute. At GD-CM's selected rate, two of its three seeds collapse after their best epoch; no CBT seed does.

Batch size (Appendix N). MNIST, N=64N = 64, K=128K = 128, Adam 10−310^{-3}: CBT's test accuracy is higher than GD-CM's by 2.80, 4.19 and 5.52 points at batch 16, 64 and 256, with 2.6–3.0× less total compute at each.

We thank the reviewer again for the constructive questions. They led to the BP and Z-IL comparisons, the ablation and the convolutional result, which together make the paper's case clearer. We hope these results address your concerns, and we would be grateful if you would consider raising your score.

Thank you for the authors' response, which partially addresses my concerns. Although there remains a performance gap compared to BP, the proposed solution itself is novel. I hope this approach can provide valuable inspiration to other researchers in the field. Therefore, I will increase my score to 3.

Reviewer_aPg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a biologically-inspired (predictive coding) alternative to gradient descent and the Newtonian alternative / using curvature.

Critical strengths
  1. Improves over GD and HB-CM
  2. Detailed results
Critical weaknesses
  1. The paper was really hard to read. My background in this area is limited, but several things could have been done to present it more clearly-- comparing to one of the cited works by Innocenti, which appear easier to understand.
  2. Predictive coding is not explained very well, nor its biological background or other intuition presented.
  3. The narrative doesn't explain many key concepts, and there many claims and not many explanations. Examples: L44 (what makes this geometric), L52 (is this drop considered large), Fig 1 (what is "reference-conditioned", what is "resolves the slow mode iteratively"), what are dense-width factorizations, what is "spends this solve", what is "depth-coherent component"
  4. In the results, no citations are given for specific baselines, e.g. HB-CM
  5. In the legend for Fig 2 (b), the legend markers don't align visually with the markers on the actual plot (they seem a bit different). Also, what should we expect from this figure? What is considered good performance for the proposed method?
  6. Evaluated on old/simple datasets
Questions for authors

See clarification requests above

2 comments

We sincerely thank the reviewer for the time and care spent reading our paper closely, even though predictive coding is not their main area, and for noting that CBT improves over GD and heavy ball and that the results are detailed. Your comments showed us exactly where the paper needed more explanation for readers beyond the predictive coding community, and the revision adds it.

Weakness 1 and Weakness 2: Readability, and explaining predictive coding. Section 2 now opens with a self-contained formulation of predictive coding (PC), in the style of the Innocenti et al. papers you mention:

  • Prediction and energy. Each layer predicts the activity of the next. The prediction error is the mismatch, and the network's energy is the sum of the squared errors.
  • Inference. Starting from the feedforward pass, the activities move downhill on the energy with the weights fixed. The section writes out the gradient-descent update. Each activity balances its own error against the error it causes in the layer above, so one step moves the output error down by only one layer. This is why deep networks are slow to infer.
  • Learning. Each weight is updated from the error and the activity on the two sides of its connection only. At test time the network predicts with the feedforward pass.
  • Biology. Cortex is thought to compare predictions with incoming signals (Rao & Ballard, 1999). Under suitable conditions this local scheme approximates backpropagation (Whittington & Bogacz, 2017).

Weakness 3: Undefined terms. Each term is now defined where it first appears in plain words, and Appendix J adds a glossary.

TermMeaning
"Geometric" (L44)The difficulty comes from the shape of the energy landscape: some directions are nearly flat and involve all layers at once, so first-order steps resolve them only with a budget that grows with depth.
The "drop" (L52)Test accuracy of a 256-layer, width-64 network trained with 9 vs 256 GD inference steps per update: 87.64% vs 87.49%, within seed variation (s.d. 0.58 and 0.88). The point is that 28× more inference steps left accuracy unchanged; the sentence now says so.
"Reference-preconditioned" (Fig. 1)The step uses one fixed curvature matrix, built once per minibatch and reused for every example in it.
"Resolves the slow mode iteratively" (Fig. 1)GD removes the slow, chain-wide error only a little per step.
"Dense-width factorizations"Exact Newton must factorize a new N×NN \times N matrix (NN = layer width) for each layer, sample and step.
"Spends this solve"Uses this computation for.
"Depth-coherent component"The part of the activity error in which all layers must change together.

Weakness 4: Citations for the baselines. Section 4 now gives them:

BaselineMeaningSource
HB-CMHeavy ball (momentum) inferencePolyak (1964)
GD-KOne inference step per layerμPC depth experiments (Innocenti et al., 2026)
"CM"Compute-matched: the iteration count gives at least CBT's compute per updatethis paper, Appendix B
Sequential inferenceLayers updated one at a time, output to inputAlonso et al. (2024)

The revision also reports backpropagation as the accuracy reference (Appendix K).

Weakness 5: Figure 2(b). The legend markers now match the plotted markers. Each curve shows the compute a method needs, in one inference call, to reach the same energy as CBT as the network gets deeper; compute is measured in gradient evaluations (GE), the cost of one energy-gradient evaluation of the same network. The caption now says what to look for: a flat curve means the cost does not grow with depth.

Untrained residual networks, width 64, mean over 5 seeds:

MethodCompute to reach CBT's energy at 256 layersFrom 16 to 256 layers
CBT7.2 GEnearly flat (6.3 → 7.2)
GD2703 GEgrows
Heavy ball277 GEgrows
Exact Newton365 GEgrows

Weakness 6: Datasets. MNIST, Fashion-MNIST and CIFAR-10 remain the standard benchmarks for training deep PC networks (Pinchetti et al., 2025), because deep PC is still hard to train at larger scale. The revision limits its claims to the residual networks and datasets we evaluate. It also adds a convolutional network on CIFAR-10 (8192 units per layer, 32 layers). There, CBT matches compute-matched GD's best validation accuracy (44.83 vs 45.23%, mean over 3 seeds) and reaches 42% validation accuracy with 36–57% less compute on every seed.

We thank the reviewer again. Your comments made the paper considerably easier to read for a broader audience. We hope the revision addresses your concerns, and we would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.