On the Role of Balanced Initialization in Feedback Alignment

Feedback alignment (FA) and its variants replace exact backward weights with fixed random feedback matrices, and have recently been argued to retain important properties of gradient-based learning. We identify a fundamental distinction between these dynamics through balancedness, a key invariant of gradient flow in deep linear networks. We show that Random Feedback Alignment (RFA) generically fails to preserve balancedness: even from exactly balanced initialization, the balancedness defect departs from zero. We prove, under mild nondegeneracy conditions, that exact balancedness can occur only at isolated times. Surprisingly, balanced structure nevertheless remains beneficial: in nonlinear networks, balanced initialization improves RFA training over Xavier initialization, and this advantage persists after matching the layerwise norms of the Xavier initialization. These results separate the benefits of balanced structure from its exact conservation, and expose a structural difference between feedback alignment and backpropagation.