- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Cached Boundary Transport (CBT), which can stabilize and accelerate activity inference in deep Predictive Coding networks using a Gauss-Newton-like preconditioner. CBT efficiently applies the inverse of a structured preconditioner without explicitly making or inverting the full matrix, thereby reducing inference energy at a relatively low computational cost.
Critical strengths
- The paper is clearly written, and the figures are easy to understand.
- The authors propose a structured preconditioner that accelerates activity inference in nonlinear PC networks. To the best of my knowledge, this is among the first approaches to explicitly use curvature information for efficient activity inference in nonlinear PC networks.
Critical weaknesses
Concerns about the model structure
Even with CBT, test accuracy tends to decrease as model depth increases on the tasks in this paper. Although the paper aims to address the training bottleneck in deep PC networks, it does not evaluate CBT in tasks where deeper networks actually provide a performance advantage.More importantly, when deeper networks offer no performance benefit, it is difficult to determine whether CBT successfully optimizes the entire deep network or whether some layers remain poorly optimized, effectively reducing the model to a shallower network. In the latter case, the observed accuracy could be achieved without successfully overcoming the optimization difficulties by depth since skipping training for some layers can improve the performance.
Additionally, Figure 10 shows that increasing network width improves accuracy. Combined with the depth-scaling results, this suggests that the architectures considered in the experiments are too deep and narrow for this task and the model structure is not natural. Figure 10 shows that the wider the width, the smaller the gap between CBT and GD-K. If you make the width large enough, you might not even need the proposed method.
Is the low-curvature component really a bottleneck?
Appendix A.1 use a very simple model (scalar weights and does not have activations) and this only motivates the claim that low-curvature eigenvectors span multiple layers. Thus, it is unclear whether the same low-curvature structure persists in practical training and is really a bottleneck of the training.
Questions for authors
- Why is the test accuracy of the MLPs on MNIST relatively low? In Figure 3, the test accuracy remains below 94%, whereas [1] reports over 98% accuracy using a three-layer MLP with a hidden width of 128. It would be helpful to clarify why this difference arises.
- Previous studies have shown that, in linear PC networks, increasing the number of inference iterations leads to convergence toward Gauss-Newton-like targets (e.g., Theorem 4.2 in [2] and Theorem 1 in [3]). How does the Gauss-Newton reference matrix defined in Equation 7 relate to these findings in the linear network setting? For clarity, it would be helpful to explicitly discuss the relationship between CBT and the Gauss-Newton-like targets obtained through iterative PC inference in previous work.
- Following the previous question, sufficiently many inference iterations in linear PC networks can recover Gauss-Newton-like targets. The paper contrasts geometric and iterative approaches and claims that the difficulty is "geometric rather than iterative." However, I am wondering whether sufficiently many iterations (which might be beyond the inference iterations tested in this paper) can recover the geometry. Could you clarify why just increaseing the inference iterations can not recover the geometry?
- Line 293 states that this paper use float64. How sensitive is CBT to numerical precision? Does it remain stable when using float32 or bfloat16?
- Why does the paper emphasize low-curvature components as the primary source of difficulty in activity inference? Could the observed benefits of CBT also arise from modifying high-curvature directions? In gauss-newton optimization of neural network training, low-rank approximations often preserve dominant high-curvature directions instead of low-curvature directions and damping removes the low-curvature directions in curvature. It would be helpful to explain why focusing on low-curvature components is particularly important in the PC setting.
- In Figure 10, does CBT maintain its advantage over GD when network width is further increased to more commonly used values, such as 1024?
- Given that increasing width improves accuracy in the reported experiments, why do the authors focus on increasing depth rather than width? The evaluated models appear to be deeper and narrower than architectures usually used in standard backpropagation-based experiments. It would be helpful to justify these choices of model structure and discuss how they affect the advantages of CBT.
- The GD-K baseline uses one inference step per hidden layer. However, this choice is not necessarily optimal compared with sequential inference [4], which can reduce the number of inference iterations required to propagate errors through the network. Does the paper use simultaneous or sequential inference for its GD-K baselines? It would be helpful to clarify this choice and discuss whether CBT's computational advantage persists when compared with sequential inference using an appropriately chosen inference budget.
Minor Questions
- Line 318 states that a batch size of eight is used, whereas the training experiments use a batch size of 64. Is this reduction primarily due to the computational cost of the exact-Hessian Newton baseline? Could the batch size affect the observed performance of CBT, given that its preconditioner uses batch-averaged Jacobians?
[1] Pinchetti et al. "Benchmarking Predictive Coding Networks -- Made Simple." ICLR 2024.
[2] Ishikawa et al. "Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation." ICLR 2025.
[3] Innocenti et al. "Only strict saddles in the energy landscape of predictive coding networks?" NeurIPS 2024.
[4] Alonso et al. "Understanding and improving optimization in predictive coding networks." AAAI 2024.
4 comments
Weakness 2 and Question 5: Is low curvature the bottleneck in practice, and why low rather than high? CBT applies the full reference inverse, so it rescales every direction; low curvature matters because it is what limits first-order inference. GD's stable step is set by the largest curvature, so high-curvature error disappears in a few steps (which is why refinements suffice), while low-curvature error decays slowly. Unlike Gauss–Newton in network training, where the stochastic curvature estimate is unreliable in low-curvature directions and is truncated or damped there, the curvature of the linearized PC chain is exact and positive definite, so it can be inverted in full.
On trained networks (, , 3 seeds), first-order inference with CBT's compute removes little of the error:
| No inference (feedforward) | GD | Heavy ball | CBT | |
|---|---|---|---|---|
| Share of the exact solution's energy reduction | 0% | 5.6% | 12.8% | 48.7% |
| Parameter-gradient cosine to the exact solution's | 0.34 | 0.34 | 0.35 | 0.58 |
GD, heavy ball and CBT receive the same compute. GD and heavy ball leave the parameter gradient where the feedforward pass, with no inference at all, left it (0.34); CBT raises it to 0.58.
To see whether CBT's gain comes from low or high curvature, we split CBT's direction in each sample's exact-Hessian eigenbasis at , the modes where 3 GD steps remove less than half the error (Appendix Q). On CBT-trained networks, the low-curvature part alone reproduces CBT's energy reduction and gradient cosine, while the high-curvature part alone reduces the energy no more than the 3 GD steps on their own.
Question 2: Relation to Gauss–Newton-like targets. In a linear PC network the Jacobians are constant, so (Eq. 7) is the exact activity Hessian, and one unit chain step from the feedforward state lands exactly on the inference equilibrium (relative gap , Appendix A.2). [2, 3] analyze this equilibrium in deep linear networks: [3] shows that the equilibrated energy is a rescaled MSE, and [2] that the resulting weight gradient lies between first-order and Gauss–Newton-like updates, depending on the parameterization. Iterative inference reaches it only at convergence; CBT computes it in closed form, at a cost constant in depth. In nonlinear networks the same formula is a Gauss–Newton step for the chain linearized at the feedforward state. Section 3.2 now discusses this and cites [2, 3].
Question 3: Can enough iterations recover the geometry? Yes; the geometry decides how many iterations are needed.
Linear residual chains (width 8, 3 seeds; Appendix A.2), cost in gradient evaluations (GE) to reach the minimum (GD and heavy ball: to a relative energy gap, with oracle-tuned steps; CBT: one chain step, exact):
| 16 | 32 | 64 | 128 | 256 | |
|---|---|---|---|---|---|
| GD | 342 | 1,217 | 4,429 | 16,281 | > 20,000 |
| Heavy ball | 45 | 85 | 160 | 306 | 592 |
| CBT | 3.33 | 3.41 | 3.46 | 3.48 | 3.49 |
In training (, Appendix L), heavy ball needs 128 steps per update, at 9.5× CBT's total compute, to pass CBT at (test accuracy 93.28 vs 92.64%). At , even 256 steps at 17.5× CBT's compute stay below CBT (91.56 vs 92.38%). The revision therefore keeps "geometric" and replaces "rather than iterative" with the cost statement: first-order steps resolve these directions only with a budget that grows with depth.
Question 8: Simultaneous or sequential inference. GD-K, GD-CM and heavy ball use simultaneous inference (all layers updated in parallel); Section 4 now states this. We added sequential inference [4] with CBT's per-update compute (10 sweeps), its step selected on a held-out seed (0.5 from {0.1, 0.3, 0.5, 1.0}). Each cell is test accuracy (%) / total compute to the stopping epoch, in thousands of GE (Appendix K):
| , | CBT | Sequential (10 sweeps) | GD-CM |
|---|---|---|---|
| 64, 32 | 93.78 / 158k | 94.77 / 458k | 90.90 / 607k |
| 64, 64 | 92.64 / 135k | 91.52 / 453k | 89.31 / 515k |
| 64, 128 | 92.38 / 170k | 90.28 / 424k | 88.19 / 473k |
| 64, 256 | 92.68 / 229k | 89.57 / 467k | 87.64 / 517k |
| 256, 128 | 95.03 / 184k | 95.56 / 399k | 94.37 / 416k |
Sequential inference improves on simultaneous GD, but it still loses 5.2 points from to 256, while CBT loses 1.1. From on, CBT is 1.1–3.1 points more accurate, and in every setting it uses 2.0–3.4× less compute.
Question 1: Why below 94%? Most of the gap to [1] comes from the architecture. In our exact setting (32–256 residual μPC tanh layers of width 64), BP reaches 96.15–96.36%, against the 98% of [1]'s 3-layer width-128 MLP. At width 512 (), CBT reaches 95.12% and BP 97.66% test accuracy. Appendix K reports BP next to the PC solvers.
Question 4: float32 and bfloat16. CBT trains in float32 with its solver and its energy tolerance unchanged. All runs finish with no non-finite state. , test accuracy (%) (Appendix O):
| CBT float32 | CBT float64 | GD-CM float32 | |
|---|---|---|---|
| 128 | 93.39 | 92.38 | 88.40 |
| 256 | 91.72 | 92.68 | 87.56 |
Every float32 CBT seed is above every float32 GD-CM seed. We did not test bfloat16, and the revision says so.
Question 9: Batch size. (a) Yes. Section 4.1 runs a per-sample exact-Hessian Newton reference to convergence at , and batch eight is the size whose cost fit our budget. Training uses batch 64. (b) , , Adam for both methods (not rescaled with batch size). Test accuracy (%), lead in points, and total compute in thousands of GE (Appendix N):
| Batch | CBT | GD-CM | Lead | Compute CBT / GD-CM |
|---|---|---|---|---|
| 16 | 92.85 | 90.05 | +2.80 | 743k / 1935k |
| 64 | 92.38 | 88.19 | +4.19 | 170k / 473k |
| 256 | 91.49 | 85.97 | +5.52 | 47k / 142k |
The lead grows with batch size, so averaging the Jacobians over a larger batch does not weaken the reference.
We thank the reviewer again for the precise and constructive questions. The new results (width, sequential inference, iteration ladder, precision and batch size) make the scope and the strength of CBT explicit. We hope they address your concerns, and we would be grateful if you would consider raising your score.
Thank you for the additional experiments and detailed comments. Your responses have clarified my concerns about whether all layers are trained and whether more inference iterations can recover the geometry.
However, I remain concerned that CBT's accuracy advantage disappears as width increases, with GD-CM outperforming CBT at larger widths. While the method is interesting, its practicality remains unclear to me.
I therefore maintain my original score.
We thank the reviewer for finding the paper clearly written and the figures easy to understand, and for recognizing CBT as among the first approaches to use curvature explicitly for efficient activity inference in nonlinear PC networks. Each question below is answered with a new measurement. Unless stated otherwise, the setting is the paper's: MNIST, residual tanh PC networks of width N and depth K, Adam 10−3, 3 seeds; accuracies are test accuracy in %, differences are in percentage points, and compute is total training compute to the stopping epoch in gradient evaluations (GE).
Weakness 1a: No task on which depth helps. Agreed. In our setting BP is also flat in depth, so the benchmark measures whether a method keeps its accuracy as depth grows. N=64, test accuracy (%):
The revision names a task on which depth helps as the next benchmark (Section 5).
Weakness 1b: Are all layers trained? We checked this directly on the trained networks (N=64, K=128, i.e. 129 weight matrices; Appendix R). Moved: the relative change ∥W−W0∥/∥W0∥ from initialization exceeds 1%. Reset: the first 10% or 50% of the matrices, counted from the input, are set back to their initial values, and we measure the change in validation accuracy (points, mean over seeds).
CBT trains and uses its lower layers, as BP does. GD-CM does not, because its 10 simultaneous GD steps per update move the output error only about one layer per step: the prediction errors, and hence the weight updates, never reach below the top ~11 layers (≈3.4T layers for T steps, Appendix R). It still reaches 87.7% because every layer has an identity skip, so the input representation passes through the untrained random layers to the trained top layers, and on MNIST a few trained layers on top of it already classify reasonably well. This shallow solution is also why GD-CM stays 4.5 points below CBT and 8.5 below BP at this depth.
Weakness 1c and Question 6: Width. K=128, test accuracy (%), and GD-CM's total compute relative to CBT's:
CBT's compute advantage holds at every width. At N=512 and 1024, GD-CM still moves only its top 9–10 of 129 matrices; on MNIST, a wide random stack with a few trained top layers already classifies well. CBT and BP train all 129. CBT's accuracy advantage therefore belongs to narrow, deep chains, where the lower layers are trained only if inference reaches them. Appendix F reports these widths. With the Adam rate tuned separately for each method (N=64, K=128), CBT stays ahead on every seed (mean +0.72 points) and reaches a stable 90% validation accuracy with 0.22× GD-CM's compute.
Question 7: Why depth rather than width. The paper studies how PC inference scales with depth, the obstacle that μPC documented. Narrow, deep chains are where inference is hardest and where solvers differ most. We agree these networks are deeper and narrower than one would choose for MNIST with BP. Section 4.3 now states that CBT's accuracy lead is largest in such narrow, deep chains.