- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes direct feedback propagation (DFP), which learns feedback matrices for projecting the output error directly to hidden layers. Local feedback losses encourage each matrix to match the product of a transposed forward weight matrix and the next layer’s feedback matrix. The paper shows that, under bounded dynamics and sufficient timescale separation, the feedback matrices track the weight-product mapping used by linear backpropagation. Orthogonality regularization conditions this mapping. Experiments compare gradient alignment and classification accuracy across MLP, VGG, and ResNet architectures on MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet.
Critical strengths
The local losses induce a joint equilibrium corresponding to the chain of transposed forward weights. Connecting this equilibrium to BP and analyzing its tracking under timescale separation provides a stronger explanation than simply showing that learned feedback improves classification.
Critical weaknesses
-
The practical advance over existing blockwise and local-learning methods is not sufficiently established. Removing backward locking and enabling independent parameter updates are already central motivations of these papers[1-3]. These methods should be cited and discussed, with a clear explanation of what DFP adds beyond the established benefits of local credit assignment and parallel optimization. Although DFP’s learned feedback pathway differs from their local objectives, comparison only with direct-feedback variants does not establish its advantage within the broader local-learning literature.
-
The contribution of the proposed feedback dynamics also needs to be separated from additional conditioning. Table 3 shows that removing orthogonality regularization reduces VGG-9 accuracy from approximately 61% to 49.22%, close to GrAPE’s 49.36%. Appendix A.2 additionally states that all DFP classification experiments use orthogonal forward initialization and feedback initialized exactly at the proposed equilibrium. Without corresponding controls, it is difficult to determine how much of the improvement comes specifically from learning the feedback connections.
-
The experiments do not establish ImageNet-scale performance or a convincing accuracy–runtime advantage on representative vision workloads. The largest dataset evaluated is Tiny ImageNet; there is no full ImageNet-1K experiment. This leaves the scalability claim insufficiently tested, particularly because the method still has substantial difficulties with residual networks.
[1] Kappel, David, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam, Christian Mayr, and Anand Subramoney. "A variational framework for local learning with probabilistic latent representations." In 5th Workshop on practical ML for limited/low resource settings.
[2] Zhang, Aozhong, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, and Penghang Yin. "Comq: A backpropagation-free algorithm for post-training quantization." IEEE Access (2025).
[3] Cheng, Anzhe, Heng Ping, Zhenkun Wang, Xiongye Xiao, Chenzhong Yin, Shahin Nazarian, Mingxi Cheng, and Paul Bogdan. "Unlocking deep learning: A bp-free approach for parallel block-wise training of neural networks." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4235-4239. IEEE, 2024.
Questions for authors
Please see the weakness above.
2 comments
Follow-up on our responses
We kindly follow up on our previous responses regarding new results and experiments we added to the paper, requested by reviewers WqwS and iFLS, which we think might also be of interest to reviewer vFiZ, based on the latter's comments.
We have now extended our benchmarks to include ResNet-56 results, showing that our DFP method outperforms the state of the art (GrAPE) on both the CIFAR-100 and Tiny ImageNet datasets there too (Results section and Appendix A.4.1, Table 8). Furthermore, in Figure 7 (Appendix A.4.1) we show the accuracy versus FLOPs of our DFP method compared against the baseline (DFA) and GrAPE, demonstrating a superior compute-performance trade-off on both VGG and ResNet architectures.
We thank the reviewer for the constructive criticism and for taking the time to review our work. We will reply to each of the weaknesses and questions below, one by one.
First weakness: We thank the reviewer for the suggested references, which are now cited and discussed in the related work section as requested (Section 2.3, lines 177-182). Regarding what DFP adds beyond the established benefits of local credit assignment and parallel optimization, the revision makes this explicit. While [1] and [3] still backpropagate errors within each block, achieving locality and parallelism at block granularity only, DFP removes the backward pass entirely, is local at the level of individual layers, and provably tracks the BP gradients of the global loss, now with the exact error with respect to BP in the nonlinear regime and an explicit bound on its norm (Equations 17-18, Appendix A.2.1), guarantees not present in the suggested references. Regarding establishing the advantage within the broader local-learning literature, we note that these methods rely on within-block backpropagation and different evaluation protocols, placing them in a different locality class than the fully BP-free methods we compare against. Finally, since [2] addresses post-training quantization rather than training, we cite it as a complementary direction applicable on top of DFP-trained networks.
Second weakness: We thank the reviewer for this comment. We first note that GrAPE's 49.36% is obtained with its own regularization of the feedback matrices, so unregularized DFP was matching the regularized state of the art. Turning GrAPE's regularization off drops it to approximately 46.5%, as now clarified in the main text (lines 464-466). The requested controls are in the new Table 4 (Appendix A.2.2), where DFA and GrAPE are given DFP's initialization, alone and combined with the orthogonality loss, each configuration with its own hyperparameter search. Neither method benefits as DFP does, and DFA even degrades far below its native performance. The new Figure 4 (Appendix A.2.2) addresses the initialization control from the opposite direction, as DFP with random feedback initialization matches the exactly initialized runs at convergence, while freezing the feedback updates is catastrophic even with the exact initialization. Finally, the new analysis in Appendix A.2.1 derives both components from our nonlinear gradient error bound, making them principled requirements of DFP's dynamics. These results attribute the improvement specifically to DFP's feedback dynamics and their required settings. We refer the reviewer to our reply to Reviewer WqwS's third weakness for further details.
Third weakness: We thank the reviewer for this comment, and we agree that larger-scale evaluations would be valuable. However, a full ImageNet-1K experiment is not feasible within the rebuttal timeframe, especially in light of the other experiments requested by multiple reviewers. We also note that no pure BP-free method in the direct feedback and forward-only literature we discuss reports ImageNet-1K results. In particular, GrAPE, the current state of the art (ICLR 2026), explicitly adopts Tiny ImageNet in place of such compute-intensive scenarios, "using it as a compute-efficient proxy that preserves ImageNet-like statistics, mirroring full-ImageNet trends (Shleifer & Prokop, 2019)," so our dataset choice follows the established evaluation protocol of the state of the art we compare against, and that of the broader field of pure BP-free algorithms. Regarding residual networks, we note that the remaining gap to BP is shared by the entire direct feedback and broader BP-free field, within which DFP already surpasses the state of the art on ResNet-20 on both datasets.