Reassessing the Biological Plausiblity of Backpropagation

Backpropagation is often called into question on the grounds of biological implausibility and incompatibility with analog and neuromorphic hardware. Alternatives to backpropagation range from heuristic modifications to principled energy-based contrastive learning algorithms. However, the criteria for what constitutes biological plausibility vary and derive from a reductive definition of BP, which conflates the choice of algorithm and network topology, exclusively considering BP in the feedforward/DAG setting. In this work, we revisit the method of Lagrange multipliers, which, when applied to credit assignment in a neural network, yields recurrent BP with standard BP and BPTT as special cases. This frequently overlooked Lagrangian formulation of BP, previously explored in LeCun (1988) possesses many of the merits and locality properties favored by the literature on local and bio-inspired learning. We highlight that recurrent BP dynamics are fully local and that a common objection, the distinction between activity states and error states, can be bypassed with a simple change of coordinates. We demonstrate experimentally on networks with asymmetric topology that the gradient estimates obtained in both coordinate systems are equivalent, with parameter-matched models out-performing standard feedforward backpropagation in the asymmetric setting. We additionally derive a variant of dyadic RBP which may be more hardware friendly at the cost of introducing an approximation error term. These results suggest that backprop is substantially closer to biological plausibility than is typically assumed, and may be more compatible with analog and neuromorphic computing paradigms than previously believed.