- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper revisits the connection between target propagation and Gauss-Newton optimization. It formulates TP as a trust-region problem, derives layer-local curvature-aware subproblems, and proposes CGTP, which solves them using conjugate gradients. The paper also shows that a batch-averaged approximation recovers a KFAC-like update. Experiments on MNIST and Fashion-MNIST MLPs suggest faster outer-iteration convergence than several TP baselines and competitive accuracy with Adam and KFAC.
Critical strengths
- The trust-region interpretation of TP is interesting and provides a more principled view of local target fitting.
- The connection between the proposed curvature-aware local solver and KFAC is conceptually useful.
Critical weaknesses
- The paper substantially over-formalizes a relatively limited core idea. Many lemmas and theorems are standard consequences of constrained quadratic optimization, linear algebra, or majorization, and the conceptual contribution is much smaller than the theoretical presentation suggests.
- The empirical validation is far too limited for the scope of the claims. Experiments are restricted to small MLPs on MNIST/Fashion-MNIST, with no evidence on CNNs, ResNets, CIFAR-scale tasks, or realistic layer-parallel training. Moreover, CGTP has a very high per-step computational cost, while the reported wall-clock advantage is negligible.
Questions for authors
- Can the authors demonstrate the method on a modern architecture and a non-trivial dataset, e.g. ResNet on CIFAR-10/100 or larger?
- What practical advantage does CGTP provide when comparing wall-clock time, FLOPs, and memory rather than outer iterations?
- Can the authors clarify which theoretical results are genuinely new beyond standard KKT, trust-region, and majorization arguments?
- Does the claimed layer-parallel advantage remain meaningful once the cost of the CG inner loop and Jacobian-vector products is included?
- For the alternating minimization result, can the authors provide a complete convergence argument to the global minimum rather than only monotonic decrease and lower boundedness?
4 comments
Remaining questions on theoretical derivations
Thank you for the detailed rebuttal and the additional cost analysis. I acknowledge that the revised convergence argument and computational profiling address some of my original concerns.
However, I have two remaining questions about the theoretical derivations:
In Theorem 2, Equation (17) introduces an additional factor of in the expression involving , which appears inconsistent with the egalitarian metric condition and Equation (49) in Appendix C.1. Could the authors clarify this discrepancy?
In Theorem 8, Equation (22) requires the inverse of , although is defined as an orthogonal projector and is generally rank-deficient. How is this inverse well-defined, given the full-row-rank assumption required by Lemma 16?
Clarifying these issues would materially affect my assessment of the paper's theoretical soundness.
Corrections to the derivations
We thank the reviewer for their thorough inspection. Both points are errors in our presentation, and we have corrected them in the revision. Neither affect the results.
Theorem 2, Equation 17. We apologise for this oversight, which is a remnant of a previous draft. The reviewer is correct: eq. 49 is the general form, and under the egalitarian condition it reduces to The factor is already contained in the metric (eq. 51 and 52), which is the egalitarian norm that recovers classical TP; the earlier statement applied it twice. Theorem 2 now states the general form, followed by this choice.
Theorem 8, Equation 22. The reviewer is correct that the Schur complement is not invertible, because is rank-deficient. The solution is to decompose , with an orthonormal basis of the range of . The feasibility constraint is then equivalent to , and the standard result (Lemma 16) may be applied (since has full row rank). Supp. C.5 contains the updated derivation, and in the main text, eq 22 has been updated to use a pseudoinverse in place of the inverse. The solution itself is unchanged - our algorithm never needed the inverse, as we used conjugate gradients on the weights (which parameterise the null space of ).
We hope this resolves your remaining concerns. We would happily clarify anything further.
Thank you for the clarifications and corrections.
I agree that both mathematical issues have been appropriately addressed, and I appreciate the authors' careful revisions. These corrections, together with the improved convergence argument, alleviate my main concerns about theoretical soundness.
However, I still have reservations about the practical significance of the method, particularly its substantial computational and memory overhead and the lack of validation on more realistic architectures.
Given the improvements in theoretical soundness, I am willing to raise my rating from 1to 2, while maintaining my concerns about the paper's overall significance and practical applicability.
Response to Reviewer itk1
We thank the reviewer for their considered review. For the Reviewer's question on modern architectures, we direct them to our response to all reviewers. Instead, we focus on the Reviewer's third question first, because it affects how the rest of the paper should be read.
Q3: Novelty We agree the individual tools (KKT conditions, trust regions, majorisation) are standard in optimisation toolkits. We emphasise that our contribution is not a new derivation of Gauss-Newton. It is an account of how an existing backpropagation alternative (TP) approximates Gauss-Newton, and the extent to which that approximation can be tightened.
Trust-region Gauss-Newton has a large design space, as one must choose an output norm, a trust-region metric across layers, how curvature is shared between layers, and the metric in which the weights realise the targets. Our first contribution is to locate classical TP in this space - namely, it is the low-trust limit under an egalitarian layer metric (Theorem 2, Remark 3), with targets realised by a single Euclidean gradient step (Remark 6). This gives us a more rigorous understanding of what TP's targets are, and therefore, when we should trust them by taking larger steps.
The other major contribution concerns the measurement of the step sizes, which ends up being a critical choice in fitting the targets. As soon as targets are not exactly realisable (which is any batch size above one) fitting them in the Euclidean metric (the natural extension of classical TP) is not a Gauss-Newton step. However, fitting them in the curvature metric is (Theorem 8). Our EucTP ablation demonstrates this gap.
The trust-region formalism also enables us to close another gap in the literature. Benzing (2022) observes the resemblance between KFAC and updates on neurons, and Ishikawa et al. (2025) note that the Gauss-Newton-target gradient is a special case of KFAC with preconditioning on the backward signals only. To the best of our knowledge, we are the first to make this correspondence exact, recovering the factored damping of KFAC (Theorem 10, Corollary 12), while exposing potential extensions (e.g. the adoption of non-uniform simplex schemes).
Q5: Convergence of alternating minimisation. The reviewer is correct: our original proof established only monotone decrease and convergence of the objective value. We have restated the claim: Theorem 9 now says that the alternating scheme minimises the regularised majoriser, not the subproblem itself. Lemma 23 (Supp. C.6) now proves that, from any interior starting point, the scheme decreases the majoriser monotonically and converges to its unique minimiser. The proof covers the case the original argument could not, namely, when the majoriser is not differentiable (ie where the α is zero), and the minimiser can lie there. In practice, since we only take a single α-step (Supp. D.3), our algorithm only needs the majorisation property.
Q2 / Q4: Cost of the inner loop. We have run a profile for each method on Fashion-MNIST at the hyperparameters used for Table 2 (Supp. D.2). Per step, CGTP uses about 1,000x the FLOPs of KFAC, at 100x the batch size (about 10x per sample processed). To the milestone it uses about 30x the total FLOPs of Adam and about 2.5x those of KFAC, with about 4x fewer epochs of data, and 6.67 synchronisation steps against 25,667 for Adam and 2,700 for KFAC. This extra compute per step is a setting, not a fixed overhead. The number of CG iters T places known methods on one path: with T=0 the update is the Euclidean fit to the targets (EucTP); a single preconditioned step from zero is along the KFAC direction; and as T grows the update converges to the exact layer-local Gauss-Newton step. CG decreases the layer-local model monotonically, so each iteration buys a tighter step. As a data point, halving the budget from 100 to 50 reaches the same milestone in the same number of outer iterations at about 60% of the FLOPs per step (see also Figure 6). We have added this interpolation as Remark 13.
We acknowledge that parallelism does not remove the CG cost. The solves are independent across layers but sequential within one, so it reduces their wall-clock time by at most a factor of the depth. Our implementation is sequential. Even so, on our hardware CGTP reaches the milestone in 26 s, against 86 s for Adam and 205 s for KFAC; this reflects the per-step overhead of the baselines at their small tuned batch sizes, and we do not claim it as a general speed advantage.