Isotropic Updates Enable Backprop-Free Training of Modern Vision Architectures

Forward Direct Feedback Alignment (FDFA) trains networks without backpropagation (BP) by delivering the output error directly to each layer through feedback matrices learned with forward-mode automatic differentiation. On modern vision architectures, namely deep non-convolutional models with normalization and residual connections such as ViT, MLP-Mixer, and Swin, however, FDFA and existing backpropagation-free methods fail to train well. We hypothesize that a key contributing factor is spectral concentration of the updates: the errors of all samples pass through the same feedback matrix, and in our diagnostics FDFA updates are skewed toward a few singular directions. Accumulated skewed updates inflate the weights anisotropically and lower the sensitivity of some feedback points, where single-pass feedback estimation then degrades and in turn sustains the skewed updates. We therefore introduce norm-preserving update orthogonalization (NPO), which makes the update of any optimizer isotropic while preserving its column and row spaces and its Frobenius norm, and apply it to FDFA to obtain IsoFDFA. Training diagnostics are consistent with the hypothesis, and NPO suppresses each stage. IsoFDFA trains MLP-Mixer, ViT, and Swin on CIFAR, matching or exceeding a BP baseline trained with the same learning rate on MLP-Mixer and ViT, and trains DeiT-S on ImageNet, where FDFA makes no progress. To our knowledge, this is the first result showing that a vision Transformer can be trained on ImageNet to non-trivial accuracy with learned feedback and a single forward-mode pass per step, without auxiliary local losses or backpropagation of errors across blocks.