Learning to Backpropagate via Coupled Direct Feedback Dynamics

Direct feedback alignment (DFA) has been proposed as a local-learning alternative to backpropagation (BP), projecting errors directly to each layer via fixed, randomly initialized feedback connections, thereby overcoming the backward locking imposed by BP's sequential backward pass. However, DFA exhibits limited performance compared to BP, primarily because its feedback connections remain fixed and cannot adapt to the evolving forward connections, thus reducing credit-assignment accuracy in deeper networks. In this work, we present direct feedback propagation (DFP), which addresses this limitation by introducing local losses on the feedback connections that induce learning dynamics whose time-varying equilibrium coincides with the chain of transposed forward weights. We theoretically and empirically demonstrate that, under a separation of timescales between fast feedback dynamics and slower forward dynamics, the feedback connections track this equilibrium throughout training. This yields exact BP gradients in the linear regime and accurate approximations in the nonlinear one, resulting in improved gradient alignment compared to DFA. Furthermore, unlike recent feedback-learning rules, DFP's feedback update is independent of the forward pass and can run fully in parallel with it. We show that DFP achieves the best performance among direct feedback methods across all evaluated settings, surpassing the best method by margins ranging from 4.9% absolute accuracy on ResNet-20 to more than 9.8% on VGG-9 and VGG-11 networks, while achieving update locality and enabling training speedups compared to BP.