acceptodds
Under review as a conference paper at ICLR 2027

Learning to Backpropagate via Coupled Direct Feedback Dynamics

Abstract

Direct feedback alignment (DFA) has been proposed as a local-learning alternative to backpropagation (BP), projecting errors directly to each layer via fixed, randomly initialized feedback connections, thereby overcoming the backward locking imposed by BP's sequential backward pass. However, DFA exhibits limited performance compared to BP, primarily because its feedback connections remain fixed and cannot adapt to the evolving forward connections, thus reducing credit-assignment accuracy in deeper networks. In this work, we present direct feedback propagation (DFP), which addresses this limitation by introducing local losses on the feedback connections that induce learning dynamics whose time-varying equilibrium coincides with the chain of transposed forward weights. We theoretically and empirically demonstrate that, under a separation of timescales between fast feedback dynamics and slower forward dynamics, the feedback connections track this equilibrium throughout training. This yields exact BP gradients in the linear regime and accurate approximations in the nonlinear one, resulting in improved gradient alignment compared to DFA. Furthermore, unlike recent feedback-learning rules, DFP's feedback update is independent of the forward pass and can run fully in parallel with it. We show that DFP achieves the best performance among direct feedback methods across all evaluated settings, surpassing the best method by margins ranging from 4.9% absolute accuracy on ResNet-20 to more than 9.8% on VGG-9 and VGG-11 networks, while achieving update locality and enabling training speedups compared to BP.

Primary area:
optimization
Keywords:
Feedback Alignment, Local Learning, Neural Networks, Direct Feedback

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_vFiZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes direct feedback propagation (DFP), which learns feedback matrices for projecting the output error directly to hidden layers. Local feedback losses encourage each matrix to match the product of a transposed forward weight matrix and the next layer’s feedback matrix. The paper shows that, under bounded dynamics and sufficient timescale separation, the feedback matrices track the weight-product mapping used by linear backpropagation. Orthogonality regularization conditions this mapping. Experiments compare gradient alignment and classification accuracy across MLP, VGG, and ResNet architectures on MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet.

Critical strengths

The local losses induce a joint equilibrium corresponding to the chain of transposed forward weights. Connecting this equilibrium to BP and analyzing its tracking under timescale separation provides a stronger explanation than simply showing that learned feedback improves classification.

Critical weaknesses
  1. The practical advance over existing blockwise and local-learning methods is not sufficiently established. Removing backward locking and enabling independent parameter updates are already central motivations of these papers[1-3]. These methods should be cited and discussed, with a clear explanation of what DFP adds beyond the established benefits of local credit assignment and parallel optimization. Although DFP’s learned feedback pathway differs from their local objectives, comparison only with direct-feedback variants does not establish its advantage within the broader local-learning literature.

  2. The contribution of the proposed feedback dynamics also needs to be separated from additional conditioning. Table 3 shows that removing orthogonality regularization reduces VGG-9 accuracy from approximately 61% to 49.22%, close to GrAPE’s 49.36%. Appendix A.2 additionally states that all DFP classification experiments use orthogonal forward initialization and feedback initialized exactly at the proposed equilibrium. Without corresponding controls, it is difficult to determine how much of the improvement comes specifically from learning the feedback connections.

  3. The experiments do not establish ImageNet-scale performance or a convincing accuracy–runtime advantage on representative vision workloads. The largest dataset evaluated is Tiny ImageNet; there is no full ImageNet-1K experiment. This leaves the scalability claim insufficiently tested, particularly because the method still has substantial difficulties with residual networks.

[1] Kappel, David, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam, Christian Mayr, and Anand Subramoney. "A variational framework for local learning with probabilistic latent representations." In 5th Workshop on practical ML for limited/low resource settings.

[2] Zhang, Aozhong, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, and Penghang Yin. "Comq: A backpropagation-free algorithm for post-training quantization." IEEE Access (2025).

[3] Cheng, Anzhe, Heng Ping, Zhenkun Wang, Xiongye Xiao, Chenzhong Yin, Shahin Nazarian, Mingxi Cheng, and Paul Bogdan. "Unlocking deep learning: A bp-free approach for parallel block-wise training of neural networks." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4235-4239. IEEE, 2024.

Questions for authors

Please see the weakness above.

2 comments

We thank the reviewer for the constructive criticism and for taking the time to review our work. We will reply to each of the weaknesses and questions below, one by one.

First weakness: We thank the reviewer for the suggested references, which are now cited and discussed in the related work section as requested (Section 2.3, lines 177-182). Regarding what DFP adds beyond the established benefits of local credit assignment and parallel optimization, the revision makes this explicit. While [1] and [3] still backpropagate errors within each block, achieving locality and parallelism at block granularity only, DFP removes the backward pass entirely, is local at the level of individual layers, and provably tracks the BP gradients of the global loss, now with the exact error with respect to BP in the nonlinear regime and an explicit bound on its norm (Equations 17-18, Appendix A.2.1), guarantees not present in the suggested references. Regarding establishing the advantage within the broader local-learning literature, we note that these methods rely on within-block backpropagation and different evaluation protocols, placing them in a different locality class than the fully BP-free methods we compare against. Finally, since [2] addresses post-training quantization rather than training, we cite it as a complementary direction applicable on top of DFP-trained networks.

Second weakness: We thank the reviewer for this comment. We first note that GrAPE's 49.36% is obtained with its own regularization of the feedback matrices, so unregularized DFP was matching the regularized state of the art. Turning GrAPE's regularization off drops it to approximately 46.5%, as now clarified in the main text (lines 464-466). The requested controls are in the new Table 4 (Appendix A.2.2), where DFA and GrAPE are given DFP's initialization, alone and combined with the orthogonality loss, each configuration with its own hyperparameter search. Neither method benefits as DFP does, and DFA even degrades far below its native performance. The new Figure 4 (Appendix A.2.2) addresses the initialization control from the opposite direction, as DFP with random feedback initialization matches the exactly initialized runs at convergence, while freezing the feedback updates is catastrophic even with the exact initialization. Finally, the new analysis in Appendix A.2.1 derives both components from our nonlinear gradient error bound, making them principled requirements of DFP's dynamics. These results attribute the improvement specifically to DFP's feedback dynamics and their required settings. We refer the reviewer to our reply to Reviewer WqwS's third weakness for further details.

Third weakness: We thank the reviewer for this comment, and we agree that larger-scale evaluations would be valuable. However, a full ImageNet-1K experiment is not feasible within the rebuttal timeframe, especially in light of the other experiments requested by multiple reviewers. We also note that no pure BP-free method in the direct feedback and forward-only literature we discuss reports ImageNet-1K results. In particular, GrAPE, the current state of the art (ICLR 2026), explicitly adopts Tiny ImageNet in place of such compute-intensive scenarios, "using it as a compute-efficient proxy that preserves ImageNet-like statistics, mirroring full-ImageNet trends (Shleifer & Prokop, 2019)," so our dataset choice follows the established evaluation protocol of the state of the art we compare against, and that of the broader field of pure BP-free algorithms. Regarding residual networks, we note that the remaining gap to BP is shared by the entire direct feedback and broader BP-free field, within which DFP already surpasses the state of the art on ResNet-20 on both datasets.

AuthorsOpenReview ↗

Follow-up on our responses

We kindly follow up on our previous responses regarding new results and experiments we added to the paper, requested by reviewers WqwS and iFLS, which we think might also be of interest to reviewer vFiZ, based on the latter's comments.

We have now extended our benchmarks to include ResNet-56 results, showing that our DFP method outperforms the state of the art (GrAPE) on both the CIFAR-100 and Tiny ImageNet datasets there too (Results section and Appendix A.4.1, Table 8). Furthermore, in Figure 7 (Appendix A.4.1) we show the accuracy versus FLOPs of our DFP method compared against the baseline (DFA) and GrAPE, demonstrating a superior compute-performance trade-off on both VGG and ResNet architectures.

Reviewer_iFLSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce Direct Feedback Propagation (DFP), a method that learns feedback matrices through local losses designed to track products of transposed forward weights. The authors prove asymptotic recovery of the linear BP mapping under timescale separation and introduce orthogonality regularization to stabilize training. Experiments show improvements over existing direct-feedback methods on several image classification benchmarks, along with multi-GPU training speedups on MLPs.

Critical strengths
  • The results on VGG architectures are promising, particularly on Tiny ImageNet, where DFP approaches BP performance while substantially outperforming previous direct-feedback methods.

  • The feedback update is also simple and independent of neural activities, enabling parallel execution. The multi-GPU experiments provide encouraging evidence of potential computational benefits, which is rarely the case in this literature.

Critical weaknesses
  1. Limited novelty and missing relevant related work.

Learning direct feedback connections toward BP-like mappings has already been investigated in DKP and GrAPE. While DFP introduces a different feedback update, its equilibrium directly corresponds to the classical linear BP mapping, and the convergence result largely follows from standard tracking arguments.

Furthermore, Forward Only Learning for Orthogonal Neural Networks of Any Depth (Caillon et al., ECAI 2025) is surprisingly not discussed. FOTON already investigates BP gradient recovery in orthogonal linear networks and the importance of orthogonality for alternative credit-assignment methods. Although the algorithms differ, this connection seems particularly relevant given the central role of orthogonality in DFP.

  1. Insufficient empirical validation and unclear computational trade-offs.
  2. The orthogonality ablation is quite concerning. On CIFAR-100/VGG-9, DFP achieves 49.22% without orthogonality regularization, essentially matching GrAPE's 49.36%. With regularization, DFP reaches approximately 61%. This makes it difficult to determine how much of the improvement comes from the proposed feedback dynamics rather than orthogonality. Moreover, all classification experiments initialize the feedback matrices at the exact linear BP equilibrium, introducing another potential confound.
  3. I also find the architectural evaluation somewhat limited. Most of the strongest results concern VGG-7/9/11 variants, whereas GrAPE has already been evaluated on VGG-16, ResNet-56, and Transformers. On the more representative ResNet-20 architecture, DFP remains far behind BP despite improving over BP-free GrAPE. This makes the claimed scalability difficult to assess.
  4. Finally, while the authors exclude BP-calibrated GrAPE to preserve strict locality, this comparison would still be relevant for evaluating practical efficiency. GrAPE uses only occasional BP calibration, whereas DFP requires computationally expensive orthogonality regularization to achieve its best results. It is therefore unclear which method offers a better accuracy–compute trade-off.
  5. On a less critical side, the reported DFP speedups are restricted to MNIST MLPs.

I would be happy to revise my grade should those issues (and the questions below) be adequately answered.

Questions for authors
  • How does DFP compare against GrAPE with equivalent orthogonality regularization? This seems essential to isolate the contribution of the proposed feedback dynamics.
  • How much of the improvement comes from learning the feedback matrices rather than initializing them at the exact linear BP equilibrium? - Would frozen feedback matrices with the same initialization provide similar results?
  • Could the authors provide a comparison against BP-calibrated GrAPE under matched total training FLOPs, accounting for orthogonality regularization, feedback updates, JVPs, and occasional BP calibration? This would help clarify the practical accuracy–compute trade-off without requiring matched hardware implementations.
3 comments
AuthorsOpenReview ↗

Response to Reviewer iFLS (1/2)

We thank the reviewer for the constructive criticism and the extensive review of our work, as well as for the nice comments. We will reply to each of the weaknesses and questions below, one by one.

First weakness: We thank the reviewer for pointing us to FOTON, now discussed in the related work section (Section 2.3, lines 188-192). We agree that its findings on the importance of orthogonality are particularly relevant to DFP, and we believe the two works resonate on this point. Regarding novelty, we would like to offer our perspective on what distinguishes DFP within this family. While the tracking argument itself follows a well-known (Tikhonov) theorem, identifying a convex local objective whose moving equilibrium is exactly the linear BP chain, and showing that the coupled dynamics provably track it, is precisely where we see our theoretical contribution. DFP's update rule is also new, as was the case for DKP and GrAPE (ICLR 2026) in their respective works, and it occupies a different point in the design space, being linear-exact where DKP comes without a convergence proof (with later work showing, only in the linear case, convergence toward a BP-distorted chain) and deterministic where GrAPE's unbiased estimator converges in expectation with a variance growing with the number of parameters. Furthermore, unlike both DKP and GrAPE, DFP's feedback update involves no activity or error information, fully decoupling it from the forward pass and allowing the two to run in parallel. Building on the reviewer's comment, we have further extended our analysis with the exact gradient error in the nonlinear regime and an explicit bound on its norm (Equations 17-18, Appendix A.2.1). To the best of our knowledge, no such combined analysis and guarantees exist in the direct feedback and related literature (PEPITA, DFA, DKP, DRTP, GrAPE). For instance, FOTON itself leaves it as future work, with its authors stating that "a formal non-linear theory of gradient alignment also remains a clear direction for future work," highlighting the challenges and the importance of such an analysis. We sincerely hope this clarification and our latest extension will be appreciated and strengthen the novelty of our work from the reviewer's perspective.

Second weakness: Please see our answers to the fourth and fifth weaknesses.

Third weakness: We thank the reviewer for this comment, which echoes similar comments raised by other reviewers. Regarding the concern on the orthogonality ablation, we emphasize that GrAPE's 49.36% is obtained with its own regularization of the feedback matrices, so unregularized DFP was matching the state of the art still using it. Turning GrAPE's regularization off drops it to approximately 46.5%, below DFP's 49.2%, as we have now clarified in the main text (lines 464-466). To determine how much of the improvement comes from the proposed feedback dynamics rather than orthogonality, the new Table 4 (Appendix A.2.2) gives DFA and GrAPE the same initialization as DFP's, alone and combined with the orthogonality loss. Neither benefits as DFP does, and DFA even degrades far below its native performance, as without feedback dynamics it is constrained to deliver gradients through the network's own initialization. This shows these components are effective only in combination with DFP's dynamics. Regarding the potential confound of the exact initialization, Figure 4 (Appendix A.2.2) resolves it from the opposite direction, as DFP with random feedback initialization recovers within a few epochs and matches the exactly initialized runs, while freezing the feedback updates is catastrophic even with the exact initialization. Finally, the newly added theoretical analysis in Appendix A.2.1 derives both components from our nonlinear gradient error bound, making them principled requirements within DFP dynamics, alongside the projected-gradient-norm motivation of Appendix A.3.3. Taken together, these results attribute the improvement specifically to the proposed feedback dynamics.

Due to the character limit, our replies to the remaining weaknesses and to the questions follow in part (2/2).

AuthorsOpenReview ↗

Response to Reviewer iFLS (2/2)

Fourth weakness: We thank the reviewer for this comment, and we agree that a more extensive architectural benchmark would be beneficial. Evaluating VGG-16, ResNet-56, and Transformers all within the rebuttal time is unfortunately not realistic, also in light of the other experiments requested by multiple reviewers. Nevertheless, since the VGG family already shows a consistent trend across depths and widths, we will do our best to extend the evaluation to ResNet-56 before the rebuttal deadline, completing the trend in residual architectures as well. We chose this architecture also because our new nonlinear analysis identifies depth as the primary factor controlling DFP's gradient error, making the deepest evaluated network the setting that tests DFP the most. Regarding the remaining gap to BP on residual architectures, we note that it is shared by the entire BP-free field, and DFP surpasses the state of the art (GrAPE, ICLR 2026) on every reported benchmark, currently narrowing this gap the most.

Fifth weakness: We thank the reviewer for this comment. Regarding the concern that DFP requires computationally expensive orthogonality regularization to achieve its best results, we kindly note that Table 3 ablates exactly this, showing that it can be evaluated sporadically, with less than 3% wall-clock overhead in that setting and no performance degradation. Regarding GrAPE's cost, our comparison uses the variant specified by the original pseudocode (Algorithm 1), with one JVP per layer, which carries the least estimator noise, as opposed to compounding all perturbations in a single forward trace. Therefore, it upper-bounds GrAPE's attainable accuracy, as reflected by our results, as our GrAPE baseline matches or exceeds the originally reported accuracies in all comparable settings (e.g., our 35.85% vs the reported 24.28% on ResNet-20 with CIFAR-100). Any cheaper variant could therefore only reduce the accuracy of the baseline we compare against, making our accuracy comparisons the most favourable possible for GrAPE. We have now made these implementation details explicit in Appendix A.3.1. If the reviewer still believes that in our settings, a FLOPs-wise or wall-clock comparison, including the BP-calibrated variant, would be valuable, we will do our best to include it in the appendix before the rebuttal deadline.

Sixth weakness: We thank the reviewer for this comment. The MLP experiments serve as an empirical validation of our parallelizability claims, demonstrating that DFP's update locality translates into concrete wall-clock gains. Extending the timing evaluation to the convolutional and residual architectures requires custom CUDA kernel implementations, an effort that is not feasible within the rebuttal timeline, which is also why we indicated this kind of experiment as future work. We further thank the reviewer for acknowledging that our multi-GPU experiments provide encouraging evidence of potential computational benefits, which, as the reviewer also notes, is rarely the case in this literature.

First question: We thank the reviewer for the question. This comparison is included in the new Table 4 (Appendix A.2.2), where GrAPE is given both DFP's initialization and orthogonality loss, with a dedicated hyperparameter search. GrAPE does not benefit from these components as DFP does, remaining well below DFP in the identical setting, which isolates the contribution of the proposed feedback dynamics. We refer the reviewer to our reply to the third weakness for the complete discussion.

Second question: We thank the reviewer for the suggested ablations. The new Figure 4 (Appendix A.2.2) answers exactly this. Freezing the feedback matrices at the exact linear BP equilibrium is catastrophic, collapsing accuracy from approximately 61% to 25% on VGG-9 with CIFAR-100, as the frozen pathway delivers gradients referring to the initial network, while DFP with randomly initialized feedback matrices recovers within a few epochs and matches the exactly initialized runs. The improvement therefore comes from learning the feedback matrices, with the exact initialization only accelerating the early phase of training, as described by our new theoretical analysis in Appendix A.2.1. We refer the reviewer to our reply to the third weakness for the complete discussion.

Third question: We thank the reviewer for the question. We refer the reviewer to our reply to the fifth weakness, where we discuss the relative cost of our GrAPE implementation and the practically negligible orthogonality overhead, and where we commit to including a FLOPs-wise comparison in the appendix before the rebuttal deadline, if the reviewer believes it would be appropriate.

AuthorsOpenReview ↗

Follow-up on our responses

We would like to kindly follow up on our previous responses, in particular regarding weaknesses 4 and 5 (and therefore question 3).

We have now extended the benchmark of Table 2 (Section 4.2) with ResNet-56 results on both CIFAR-100 and Tiny ImageNet, where DFP still outperforms both DFA and GrAPE, reaching 38.4138.41% on CIFAR-100 (against 14.5014.50% and 36.8736.87%, respectively) and 23.6923.69% on Tiny ImageNet (against 9.209.20% and 22.3522.35%). Appendix A.4.1 (Table 8) also reports the top-5 accuracy for the same experiments, showing that the results hold for this metric too, with even larger margins.

Regarding the FLOPs comparison, we were unfortunately not able to reproduce the gains of the BP calibration in our own implementation. In any case, we believe that such a calibration could in principle be applied on top of DFP as well, so that comparing a BP-calibrated GrAPE against a fully BP-free DFP would not be on equal terms. To nevertheless address weakness 5 and question 3, we have now included in Figure 7 (Appendix A.4.1) the accuracy versus FLOPs comparison for the BP-free DFP, DFA and GrAPE. There, DFP reaches the highest accuracy for a given FLOPs budget on both the VGG and ResNet architectures.

Furthermore, regarding the reviewer's comment that "DFP requires computationally expensive orthogonality regularization", Figure 7 also allows assessing this cost. On ResNet-20, even with the loss applied at every update, DFP requires only a fraction of additional FLOPs compared to DFA. On VGG-9, where this cost is heavier, applying the loss once every 50 updates drastically reduces it without sacrificing performance (in line with the ablation of Table 3), reaching the same final accuracy for just a fraction of additional FLOPs compared to DFA.

Thanks to the valuable feedback received from all reviewers during this round, we believe that the paper has been substantially improved, and that the weaknesses and questions raised have now been addressed and answered. Should any concern remain, we would be glad to discuss it further. Otherwise, we would be grateful if the reviewer could reflect this in a revised assessment, as kindly offered in their review.

Reviewer_kQ5UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a new training algorithm for feed-forward neural networks that builds on the idea of direct feedback alignment (DFA). In DFA, instead of backpropagating the error through the network, the error is backpropagated direclty to each layer via a dedicated feedback weight matrix. The authors propose Direct Feedback Propagation (DFP), which can be thought of as DFA with feedback matrices trained to track the linear backpropagation mapping. More precisely, each fixed random feedback matrix BℓB_\ell becomes learnable, using

LBℓ=12∥Bℓ−Wℓ⊤Bℓ+1∥F2,BL=I,\mathcal L_{B_\ell}=\tfrac12\|B_\ell-W_\ell^\top B_{\ell+1}\|_F^2,\qquad B_L=I,

with only BℓB_\ell differentiated. The coupled updates drive BℓB_\ell toward the product of downstream transposed weights. Since the update rule for feedback matrices uses only the weights, it can run across layers and alongside the forward pass. They also add an orthogonality penalty on forward weights to prevent the feedback products from becoming poorly conditioned. For sufficiently faster feedback dynamics, the method recovers exact BP in linear networks, but only an approximation for nonlinear networks. Note that unlike DFA, the method therefore requires access to forward-weight transposes. In their experiments, the authors use one feedback update per minibatch, with all layers updated in parallel, so they do not (need to) wait for equilibration of the feedback weights.

Experimentally, the authors find that DFP aligns its weights udpates substantially better with BP than DFA, but the gap in performance to BP remains large.

Critical strengths
  • The proposed algorithm is a simple, well-motivated extension of DFA. The local feedback objective directly targets DFA’s fixed-feedback limitation, while preserving parallel error delivery
  • The authors observe empirical improvements over DFA.
  • The orthogonality ablation and runtime experiments also address practical implementation costs.
Critical weaknesses
  • DFP explicitly trains feedback matrices toward the linear BP mapping, so the improved alignment over fixed-feedback DFA primarily confirms the intended mechanism.

  • The contribution of feedback learning to the accuracy gains is not comletely clear to me. Although the improvements over DFA are substantial, DFP also introduces exact equilibrium initialization and orthogonality regularization of the forward weights. More controlled ablations, including DFA with matched initialization and regularization, would clarify how much improvement comes specifically from the proposed feedback dynamics.

Questions for authors

See Weaknesses: could you perform a more controlled ablation to isolate the effect of the feedback dynamics ?

4 comments

We thank the reviewer for appreciating our work and for the nice comments. We reply to each weakness and the question below, one by one.

First weakness: We thank the reviewer for acknowledging that the improved alignment over fixed-feedback DFA confirms the intended mechanism. Furthermore, our analysis now goes beyond the linear BP mapping, as the revision derives the exact error with respect to BP in the nonlinear regime and an explicit bound on its norm. As the reviewer's comment summarizes the fundamental contribution through which DFP delivers the final improved performance without sacrificing update locality, we are unsure which weakness the reviewer is referring to, if any. We would like to ask the reviewer to clarify by engaging with us in the discussion here, so we can answer this concern.

Second weakness: We thank the reviewer for this comment, similarly raised by all reviewers. We have run the requested controls, reported in the new Table 4 (Appendix A.2.2), where DFA and the state of the art (GrAPE) are given DFP's exact equilibrium initialization, both alone and combined with the orthogonality regularization. Neither method benefits from them as DFP does, and DFA degrades substantially below its original performance. The new Figure 4 (Appendix A.2.2) complements this from the opposite point of view, as DFP with random feedback initialization matches the exactly initialized runs at convergence, while freezing the feedback updates is catastrophic despite the exact initialization. Furthermore, in Appendix A.2.1 we analytically demonstrate why the exact initialization and the orthogonality loss are essential to DFP's performance, which makes them principled requirements of the method. Taken together, these results attribute the accuracy gains specifically to the proposed feedback dynamics and the components they require.

Question: We thank the reviewer for the question. The requested controlled ablations have been performed: please see our answer to the second weakness, together with the new Table 4 and Figure 4 (Appendix A.2.2).

I appreciate the additional effort by the authors to perform ablations. The revision makes a stronger case that the gains depend on DFP’s prooposed feedback dynamics, rather than exact initialization or orthogonality regularization. I do think that the responses’ specific attribution to feedback learning is somewhat stronger than the experiments alone warrant: the controls show the components work differently in combination with DFP, but they do not fully decompose the contribution of each component. so I maintain my score.

We sincerely thank the reviewer for engaging in the discussion, for appreciating our efforts, and for acknowledging the stronger case made by the revision. Regarding the remaining concern, we would like to kindly point out that the revision does ablate each component individually (VGG-9 on CIFAR-100):

  • Without feedback learning. DFA with matched initialization and regularization, the control requested in the review, reaches 24.75% against 60.92% for DFP (Table 4). Freezing DFP's feedback update with everything else unchanged is equally catastrophic, resulting in about 25% (Figure 4).
  • DFP with the exact initialization only (no orthogonality loss) reaches 49.28% (Table 4).
  • DFP with the orthogonality loss only (random feedback initialization) recovers the full accuracy (about 60.9%), with only a slower start (Figure 4).
  • DFP with both reaches 60.92%.

We believe these experiments closely follow the controlled ablations suggested in the review, for which we are grateful, as they improved the quality of the paper.

Moreover, we would like to kindly add that these two components are an integral part of the feedback dynamics rather than additions to them. DFP is defined by the coupled system of Equation 14, which includes the orthogonality term, as motivated in Appendix A.2.1. The deviation from BP is bounded by a term governed by the spectral norm of the forward weights, which can grow exponentially with depth and which the orthogonality loss keeps under control, and by the tracking error, which the exact initialization sets to zero from the first step. The feedback dynamics without both components would therefore lie outside the regime the method is derived for, and each ablation above matches these predictions.

We thank the reviewer again, and we hope this resolves the remaining concern.

As a brief follow-up to our previous response, we would like to kindly add a clarification regarding the reviewer's remark on the strength of the attribution to the feedback learning.

We would like to kindly clarify that our attribution has always been to DFP as a whole, and never to the feedback learning alone. DFP is, by definition, the feedback dynamics coupled with the orthogonal regularization (Equation 14), where the orthogonality loss implicitly conditions the feedback matrices through the forward weights (Appendix A.3.3) and bounds the deviation from BP in the nonlinear regime (Appendix A.2.1). The bare feedback dynamics alone therefore do not constitute the proposed method (DFP).

To further address the reviewer's concern and avoid any ambiguous attribution, we have made this more explicit by highlighting it in the abstract as well, as follows: "The nonlinear regime's error is bounded by the largest singular value of the forward weights, and controlled through DFP's orthogonality loss on them, also implicitly regularizing the feedback pathway."

We thank the reviewer again for the constructive discussion, and we hope this addresses the concern regarding the attribution.

Reviewer_WqwSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Direct Feedback Propagation (DFP), which learns direct feedback matrices through coupled weight dynamics. These matrices track products of transposed forward weights, recovering the backpropagation error mapping in linear networks under suitable timescale assumptions. DFP improves classification accuracy over the evaluated direct-feedback methods, and a multi-GPU implementation demonstrates training-time improvements on MLP workloads. The improvements within the direct-feedback family are encouraging. However, I find the comparison with alternative learning approaches incomplete, the locality and weight-transport claims insufficiently justified, and the experiments inadequate to separate the proposed feedback dynamics from important initialization and regularization choices.

Critical strengths

The feedback update has a clear mathematical formulation, and the analysis provides a useful account of tracking a moving linear feedback equilibrium. The experiments cover multiple architectures and datasets, report variability across five seeds, and include several relevant direct-feedback baselines. Measuring actual wall-clock performance is also valuable.

Critical weaknesses
  1. The discussion of alternatives to backpropagation is narrow. The paper motivates DFP through biological plausibility, weight transport, and backward locking, but primarily evaluates methods within the direct-feedback family. This establishes a relatively narrow performance comparison without adequately positioning DFP among approaches that address these motivations through different learning mechanisms. One family of interesting alternatives is forward-only algorithsm. E.g., o Hinton (2022), “The Forward-Forward Algorithm: Some Preliminary Investigations.” Forward-Forward replaces the conventional forward/backward sequence with positive and negative forward passes and layer-specific goodness objectives. It therefore provides a relevant alternative to learning feedback pathways that approximate backpropagation. o Srinivasan et al. (2024), “Forward learning with top-down feedback: empirical and analytical characterization,” ICLR. This work analyzes forward learning with top-down feedback, develops connections to adaptive feedback alignment, and compares Forward-Forward and PEPITA. It is especially relevant to the present paper’s discussion of learned feedback and relationships to backpropagation. These approaches have different objectives and constraints, so identical experimental treatment is not always appropriate. Nevertheless, the manuscript could benefit from discussing other alternatives to backpropagation.
  2. The weight-transport and locality claims require a more precise justification. Equation 10 explicitly uses the transposed forward weights to update the feedback matrices. The authors distinguish this from directly using those weights to propagate errors, but moving access to transposed weights into an auxiliary learning process does not by itself resolve the underlying implementation problem. Moreover, the feedback update combines forward and feedback weight matrices, while the orthogonality penalty couples multiple weights. Calling these operations “local” at the layer level does not establish locality at individual biological synapses. The paper should specify exactly which quantities each update requires and distinguish layer-level computational independence from biological locality. The discussion of symmetric-weight learning identifies a possible extension, but does not demonstrate that the proposed implementation satisfies those constraints.
  3. Initialization and orthogonality regularization substantially complicate attribution of the gains. Appendix A.2.1 states that all classification experiments initialize DFP with orthogonal forward weights and feedback matrices already at the exact linear equilibrium. Consequently, these experiments primarily test maintenance of an initially correct linear feedback mapping, rather than learning that mapping from independently initialized pathways. Random and perturbed feedback initializations are important controls. Orthogonality regularization also has a large effect. In Table 3, removing it reduces VGG-9 accuracy on CIFAR-100 from approximately 60.7% to 49.2%, close to the reported GrAPE result. This does not negate the value of the complete method, but it makes matched controls essential. Applying comparable initialization and conditioning strategies to competing methods would help determine how much improvement comes specifically from DFP’s feedback dynamics.
  4. The theoretical guarantee is narrower than the practical claims. The theorem concerns continuous-time tracking under boundedness assumptions and limiting timescale separation. Its exact correspondence to backpropagation applies to linear networks. In nonlinear networks, the feedback matrices omit intermediate, input-dependent activation Jacobians; tracking the linear weight product therefore does not establish accurate nonlinear gradients. The MNIST alignment experiment provides useful empirical evidence, but does not establish how approximation error behaves with depth or changing activation patterns. The substantial ResNet-20 gaps—40.78% versus 68.04% for BP on CIFAR-100, and 27.92% versus 52.94% on Tiny ImageNet—underscore this limitation. Gradient-alignment measurements on these harder settings and sensitivity analyses of the realized feedback/forward update rates would better connect the theory to practice.
  5. The efficiency evidence does not establish faster training to comparable performance. The reported speedups concern MLPs on MNIST, whereas the strongest classification claims concern convolutional and residual networks. The timing comparison measures training throughput over a fixed number of epochs rather than time to a common accuracy target. Given the accuracy gaps and the additional cost of orthogonality regularization, these are materially different criteria. A stronger evaluation would report time to matched accuracy, memory consumption, and communication costs on the architectures used for the principal classification results. The current measurements support a promising parallel implementation, but leave its practical advantage on more demanding workloads unresolved.
Questions for authors

Several of the concerns are discussed above. At a high level, the main concern centers around the applicability of the results and how much they push the needle in alternatives to backpropagation in terms of performance in real-world problems (beyond speed).

3 comments
AuthorsOpenReview ↗

Response to Reviewer WqwS (1/2)

We thank the reviewer for the thorough comments and the constructive criticism. We reply to each weakness and the question below, one by one.

First weakness: We thank the reviewer for pointing to these resources, as we agree that positioning DFP among alternatives beyond the direct-feedback family strengthens the paper. We have thus expanded the related work section as requested (Section 2.3, lines 183-195). The revision now discusses forward-only approaches, in particular the Forward-Forward algorithm (Hinton, 2022) and the analysis of forward learning with top-down feedback by Srinivasan et al. (2024), covering also other BP-free approaches such as PEPITA and FOTON (suggested by Reviewer iFLS), and comparing their learning mechanisms with the feedback-learning approach of DFP.

Second weakness: We thank the reviewer for raising this point, which allowed us to state the scope of our claims more precisely. Following this comment, we have rewritten the opening of the Discussion (Section 5, lines 493-500). It now states explicitly that, while DFP does not strictly reintroduce weight transport from the error-propagation perspective, it does rely on access to the transposed forward weights in an absolute sense, as one of its trade-offs for a closer alignment with BP. The same passage now defines our locality claim as layer-level computational locality, the property removing backward locking and enabling model-parallel implementations, rather than locality at the level of biological synapses, better scoping the symmetric-weight learning mechanism as a future extension, on which none of the paper's claims rest. As per the quantities each update requires, these are fully specified by Equation 14, involving only quantities available at the layer itself and the feedback matrix of the adjacent layer, so no update accesses quantities from non-adjacent layers, consistent with the explicit locality definition now provided in the Discussion.

Third weakness: We thank the reviewer for this comment, which allowed us to strengthen the paper on a point raised by several reviewers. The requested random and perturbed initialization controls are now in Figure 4 (Appendix A.2.2). DFP recovers from a random initialization within a few epochs and matches the exactly initialized runs at convergence, as predicted by our analysis, which shows that the initial tracking error fades away during training, whereas freezing the feedback dynamics collapses performance. Regarding the orthogonality regularization, we emphasize that the reported GrAPE baseline keeps its own feedback regularization, so DFP without regularization was competing with the state of the art still using it. If GrAPE's regularization is turned off, its performance drops to approximately 46.5%, below DFP's 49.2%, as we have now clarified in the text. We nevertheless agree that matched controls are essential, and the new Table 4 (Appendix A.2.2) applies DFP's exact initialization, alone and combined with the orthogonality loss, to DFA and GrAPE, with each configuration tuned through its own hyperparameter search. Neither method benefits as DFP does, and both remain below DFP in the identical setting. The new ablations are now also included in the main text, where the orthogonality loss analysis has been expanded into a dedicated subsection (Section 4.3, lines 428-473) commenting on these results and directing the reader to Appendix A.2.

Due to the character limit, our replies to the fourth and fifth weaknesses and to the question follow in part (2/2).

AuthorsOpenReview ↗

Response to Reviewer WqwS (2/2)

Fourth weakness: We thank the reviewer for this comment, which motivated the main additions of our revision and allowed us to close the flagged gap between theory and practice. Our guarantees are no longer restricted to the linear regime. We now derive in closed form the exact deviation of DFP's gradient mapping from BP in the nonlinear regime (Equation 17, line 290), explicitly accounting for the omitted activation Jacobians, together with an upper bound on its norm (Equation 18, line 295, derivations in Appendix A.2.1). The bound grows exponentially with the distance from the output layer and with the largest singular value among the forward weights, explaining the larger gaps in the deeper settings. The orthogonality loss keeps this value close to one, slowing the growth as much as possible and making it and the exact initialization even more principled needs of DFP. Regarding alignment on harder settings and the realized update rates, the learning-rate ratio does not measure the effective timescale separation, as the two parameter groups use different optimizers (Appendix A.3.2). We therefore extended the alignment analysis to a VGG-11 trained on CIFAR-100 under the full training protocol (Figure 5, Appendix A.2.2). DFP aligns almost perfectly at the second-last layer and degrades with depth exactly as the new analysis predicts, remaining far above DFA at every layer. We also note that a growing gap with depth and task complexity is common across BP-free and especially direct feedback methods, and that DFP in fact narrows this gap beyond the current state of the art.

Fifth weakness: We thank the reviewer for this comment. The MLP experiments serve as an empirical validation of our parallelizability claims, demonstrating that DFP's update locality translates into concrete wall-clock gains. While we agree that parallelization experiments on convolutional and residual architectures are relevant, we have left them as future work as they require custom CUDA kernel implementations, which are unrealistic within the rebuttal timeframe in light of other requested experiments. Regarding time to matched accuracy, this criterion is not well defined against BP, whose final accuracy exceeds DFP's in these settings, while against GrAPE, the strongest direct feedback competitor, DFP reaches a higher final accuracy at a lower cost per epoch, in our implementation. However, we would like to emphasize that related works usually only refer to potential parallelizability benefits as part of the discussion, while we demonstrate concrete results with MLPs over multi-GPU deployments. To the best of our knowledge, this is one of the rare cases in which such wall-clock-time gains are reported, as also acknowledged by Reviewer iFLS.

Question: The main concern raised in the question is discussed in our previous answers. We nevertheless thank the reviewer for the opportunity to answer by stating the broader significance of our results. DFP is, to our knowledge, the first direct feedback method with deterministic guarantees in both regimes, converging exactly to BP in the linear case and admitting an exact characterization with an explicit bound of its error in the nonlinear one. Empirically, it surpasses the current state of the art on all reported benchmarks, further narrowing the gap with BP, which we believe is precisely how the needle moves for alternatives to backpropagation on real-world problems.

AuthorsOpenReview ↗

Follow-up on our responses

We kindly follow up on our previous responses, especially regarding the last weakness mentioned by the reviewer. We have leveraged the locality of the update in a model-parallel implementation of DFP. The latter now achieves up to a 3.7× speedup, compared to the previous 2.4×, in the most beneficial case, highlighting the advantages of computational locality. Furthermore, as requested by the reviewer, we have included a thorough analysis of memory and communication costs, against both data-parallel and model-parallel variants of BP. In both cases, DFP not only exhibits training speedups, but also a consistent reduction in communication cost. These changes have been inserted in our contributions (lines 83–86), in the Results section (Figure 3 and lines 477–489), and in Appendix A.4.2 (Figures 8, 9, 10, and 11).

Furthermore, we have also included a FLOPs-wise comparison of DFP against DFA and GrAPE on VGG and ResNet architectures, showing that it performs better than both of them for a given FLOPs budget (Appendix A.4.1, Figure 7). Lastly, we have now included results on ResNet-56, demonstrating that DFP outperforms both DFA and GrAPE there too (Results section and Appendix A.4.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.