Forward Iterative Optimization Network for Image Recognition

Neural networks trained with backpropagation (BP) have achieved remarkable success across a wide range of domains. However, BP contains several limitations like weight transport, backward locking, computational and memory overhead. Although recent BP-free learning methods have explored alternative training mechanisms, they often compromise predictive performance or computational efficiency. To overcome these limitations, we propose Forward Iterative Optimization (FIO) framework for network learning. Different from existing BP-free methods, FIO derives analytical solutions for parameter updates and incorporates them into an iterative optimization strategy, thereby reducing training-time memory overhead and enabling efficient parameter updates. Based on this strategy, we first develop a parameter optimization method for a single-layer network and then extend it to multi-layer and multi-head architectures. Extensive experiments demonstrate that FIO achieves competitive or even superior performance compared with both BP and representative BP-free methods, while requiring fewer parameters and lower computational cost.