acceptodds
Under review as a conference paper at ICLR 2027

Forward Iterative Optimization Network for Image Recognition

Abstract

Neural networks trained with backpropagation (BP) have achieved remarkable success across a wide range of domains. However, BP contains several limitations like weight transport, backward locking, computational and memory overhead. Although recent BP-free learning methods have explored alternative training mechanisms, they often compromise predictive performance or computational efficiency. To overcome these limitations, we propose Forward Iterative Optimization (FIO) framework for network learning. Different from existing BP-free methods, FIO derives analytical solutions for parameter updates and incorporates them into an iterative optimization strategy, thereby reducing training-time memory overhead and enabling efficient parameter updates. Based on this strategy, we first develop a parameter optimization method for a single-layer network and then extend it to multi-layer and multi-head architectures. Extensive experiments demonstrate that FIO achieves competitive or even superior performance compared with both BP and representative BP-free methods, while requiring fewer parameters and lower computational cost.

Primary area:
learning theory
Keywords:
Backpropagation-Free Learning, Analytical Optimization, Image Classification, Efficient Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_NGBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Forward Iterative Optimization (FIO), a framework that trains image classifiers through alternating parameter updates without end-to-end backpropagation. Each block combines feature-wise projections, ReLU activations, and token-wise projections, with multiple heads averaged before passing representations to the next block. Training uses squared-error objectives, analytical updates for several parameter groups, an active-set solver for pooling weights, and column-wise updates based on fixed ReLU activation patterns. Experiments on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 compare classification accuracy, parameter counts, and inference latency against BP and several BP-free methods. Ablations examine depth, head count, hidden dimension, and residual connections.

Critical strengths

The objectives and parameter updates are stated mathematically, and several updates solve regularized least-squares subproblems directly. This makes the procedure relatively transparent and offers a useful direction for studying learning without conventional backward gradient propagation. However, this area has been studied for years.

Critical weaknesses
  1. The dataset is too small. The experiment was only validated on small-scale datasets such as CIFAR and MNIST, which have already been extensively studied, and most weights could easily achieve higher accuracy. This paper did not produce competitive results. Also, no large-scale dataset is tested, such as ImageNet. I feel it is somewhat pointless to develop new architectures on small datasets, given the current evolution of LLMs.

  2. The method looks incremental. From Figure 1, it looks like the method is not so different from traditional DNN. Also, analytical parameter updates and alternating minimization have already been studied in neural network training. Particularly relevant examples are Training Neural Networks Without Gradients: A Scalable ADMM Approach (Taylor et al., ICML 2016), Learning ReLU Networks via Alternating Minimization (Jagatap and Hegde, 2018), and Training a Two-Layer ReLU Network Analytically (Barbu, 2023). The latter two are especially close to the use of activation-pattern estimates and least-squares updates. These works are neither discussed nor compared, despite analytical optimization being presented as the main distinction from existing methods.

  3. The authors also do not mention how BP performed with the same FIO architecture. Will it also improve BP's performance?

Questions for authors

Please see the weakness above.

Reviewer_a95BOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework called Forward Iterative Optimisation (FIO), which trains neural networks without the need for backpropagation. Backpropagation (BP) suffers from issues such as weight transport and backward locking, as well as computational and gpu memory overhead caused by caching intermediate activations. Meanwhile, existing backpropagation-free methods—including bio-inspired, asymmetric feedback, and pure forward propagation approaches—often require compromises in terms of accuracy, efficiency, or scalability. In contrast, FIO does not use gradient updates but instead updates parameters using analytical solutions within an iterative (alternating) optimization framework, thereby eliminating the need to store intermediate states for backward computations. The method is first derived for single-layer networks and then extended to multi-layer and multi-head architectures which multi-head structures are used to generate diverse feature projections, while multi-layer structures are used to enhance the flexibility of representations. The paper claims that FIO outperforms existing BP-free methods, achieving classification accuracy comparable to standard BP while requiring fewer parameters and incurring lower computational and GPU memory costs. Experiments were conducted on image classification tasks using MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, with images sliced into 4×4 patches as tokens and no data augmentation applied. With a two-hidden-layer configuration, FIO was compared with BP and eight BP-free baselines.

Critical strengths
  1. This paper proposes a BP-free framework based on ridge regression. It eliminates the need for backpropagation and enhances interpretability. This is an issue that many BP-free methods aim to address, and this paper attempts to tackle it from this perspective.
  2. The proposed FIO requires less computation time and fewer parameters, resulting in lower resource consumption. It has potential application value in low-resource scenarios.
Critical weaknesses
  1. The authors constructed a network to validate FIO’s performance, but this lacks generalizability and credibility. More results from different backbones should be considered to further demonstrate its effectiveness.
  2. Based on the results, the differences among the four datasets selected in this paper, the various baselines, FIO, and BP are too small to fully attribute the gains to FIO’s BP-free design.
  3. The approach of using ridge regression to circumvent gradient propagation applies to all closed-form solutions or methods, such as extreme learning machines, analytic learning, ADMM-based methods, and block coordinate descent training; the innovation of this approach requires further clarification.
  4. The closed-form solution for the token-mixing matrix is always of rank 1. Therefore, the observed gains may not be attributable to the FIO method itself.
Questions for authors

I have the following questions and suggestions regarding this paper, and I hope the authors will include additional experiments or explanations to clarify these points.

  1. In the equations, I noticed that the token-mixing matrix is always rank-one. If this is the case, it would actually prevent token-level information exchange within the network. Please clarify and explain this issue.
  2. The network architecture currently used is somewhat specific. Can the method proposed by the authors be implemented on a broader range of backbones, such as Vit? If so, given that FIO employs closed-form solutions and greedy optimization, would it actually be more time-consuming and computationally intensive on networks with a larger number of parameters? Please explain.
  3. The authors mention in the introduction that FIO can reduce training-time memory and compute costs compared to BP, but they provide no memory measurements or training time estimates. Please elaborate further.
  4. There has been extensive prior work on closed-form, layer-wise, and block-coordinate methods. How does FIO’s underlying structure and methodology differ from these approaches? More detailed comparisons and contrasts are needed to highlight the true points of innovation.
  5. The paper mentions “backward locking.” Can it be proven that this is a genuine problem, and can the authors’ method account for it?
Reviewer_pLJDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FIO, a BP-free training procedure for a patch-token model. In each block, every head ii applies a per-token feature projection Wℓ,iW_{\ell,i}, a ReLU, and an m×mm \times m token-wise projection Aℓ,iA_{\ell,i}, and the pp head outputs are averaged. Training minimizes the squared loss on one-hot labels by alternating updates over the whole training set:

  • the pooling vector aa (on the simplex) is updated by an active-set QP;
  • the classifier WW and the token projections AA by ridge-regression closed forms;
  • the columns of the feature projections by ridge regression with the ReLU activation pattern held fixed.

Deeper models are built greedily. While block ℓ\ell is trained, the pooling vector aa and the classifier WW form its supervised readout and are re-optimized, and earlier blocks stay frozen. Heads are updated one at a time, and at inference the readout is applied after the last block. On MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 the method reports accuracy comparable to a BP baseline and above several BP-free methods, with fewer parameters and lower single-sample inference time.

Critical strengths
  1. The question is relevant: can closed-form or alternating least-squares updates be a practical alternative to gradient-based training?
  2. The subproblem derivations (Appendices A–D) are explicit and easy to check, and Appendix G gives a complexity analysis.
  3. The paper reports its own limitations: depth does not help consistently, and residual connections did not help (Appendix H).
Critical weaknesses

I read the supplementary code and checked the relevant formulas on small, fully specified inputs. I did not run full training. Points W1 and W2 concern the released code path; I cannot confirm that every reported table was produced with it.

W1. The released evaluation pipeline reports maxima of test accuracy.

  • The training script passes the test set to the training routine as validation data, and the data loading creates no validation split.
  • The returned accuracy is a maximum of test accuracy, taken at three levels:
    • over the four parameter sub-updates within each head step;
    • over all iterations within a block;
    • over all blocks.
  • The code neither saves nor restores the parameter state that produced this maximum, so the reported best score need not correspond to the model kept at the end of training.
  • A related implementation detail: the parameter reset after a NaN is also triggered by the test loss.

Sec. 3.2 states that depth is selected "according to validation performance", but the paper describes no validation split. If Table 1 was produced with this pipeline, its numbers are test-set maxima rather than held-out estimates. Please re-run with a validation split taken from the training set, and report the test accuracy of the selected final model.

W2. The implementation departs from the method described in the paper.

  • The token update differs from Eq. (8) and adds an undisclosed rank-increasing step.
    • The implemented token-projection update computes the minimum-norm solution of rank at most one (including the degenerate zero solution) without the ridge term of Eq. (8).
    • It then adds a random Gaussian component (scale 10−210^{-2}) in the orthogonal complement of aa. The added component leaves aAaA unchanged up to floating-point error, so the current block's prediction is unaffected. It can, however, raise the rank of AA (generically to full rank) and it changes the inputs of later blocks.
    • The step thus introduces token-representation freedom that is absent from the paper's closed form (see W4). It needs to be disclosed and ablated.
  • The aa-update formula does not match the stated objective.
    • Even in the simple case where C=BB⊤C = BB^\top is positive definite, all variables are active and the solution is strictly positive, the equality-constraint correction in the implemented pooling update uses C−21C^{-2}\mathbf{1} instead of C−11C^{-1}\mathbf{1}. That corresponds to minimizing ∥Cx−d∥2\lVert Cx - d \rVert^2 rather than x⊤Cx−2d⊤xx^\top C x - 2 d^\top x subject to 1⊤x=1\mathbf{1}^\top x = 1.
    • Example: C=diag(1,4)C = \mathrm{diag}(1, 4), d=0d = 0. The correct minimizer is [0.8, 0.2][0.8,\ 0.2], whereas the code's formula returns [16/17, 1/17][16/17,\ 1/17].
    • Removed coordinates are never checked for optimality or reactivated.
  • The update order and head combination differ. The implementation re-solves aa and WW before each head's update rather than once per sweep as in Algorithm 3, so different heads can carry different left factors. Head outputs are summed rather than averaged. Summation and averaging may represent the same function class after parameter rescaling, but this does not establish that the training procedures are equivalent, particularly given the different update order and regularization.
  • Regularization.
    • The classifier WW and the single-head feature-column update use no ridge term.
    • The multi-head column update applies a fixed 10−610^{-6} to the increment, which is not the direct ridge update of Eq. (11).
    • λW\lambda_W, λA0\lambda_{A_0} and λW0\lambda_{W_0} are not given in the paper and do not appear as such in the code.
  • Runnability and configuration.
    • The entry script hard-codes a configuration that overrides the command-line arguments: CIFAR-10, residual, e=128e = 128, 8 heads, depth 10.
    • It also enables a debug mode that skips training of the first block.
    • The training routine of the residual block calls itself and recurses without end, so the residual variant of Appendix H cannot run as released.
    • The supplement contains no configuration files or run commands that correspond to the paper's experiments, and no baseline implementations or benchmarking scripts that reproduce the timing results in Table 2, although the reproducibility statement says configuration files are provided.

The authors should state which algorithm, as written or as implemented, produced each reported result.

W3. The W0W_0 update has no descent guarantee on the original ReLU objective.

Paper version. Equation (11) exactly solves a surrogate with the activation mask fixed. This does not guarantee that the original ReLU objective decreases. Here is an example where even the regularized true objective increases:

  • Setup: one token and one hidden neuron, with a=A0=1a = A_0 = 1, X=[1102]X = \begin{bmatrix} 1 & 1 \cr 0 & 2 \end{bmatrix}, Y=I2Y = I_2, W=[1,0]W = [1, 0], uold=[2,−1]⊤u_{\text{old}} = [2, -1]^\top and λW0=0.01\lambda_{W_0} = 0.01.
  • Update: Eq. (11) gives unew=[1,1]⊤/2.01u_{\text{new}} = [1, 1]^\top / 2.01.
  • Effect: the regularized surrogate decreases from 1.051.05 to ≈1.00498\approx 1.00498. The regularized true objective, however, increases from 1.051.05 to ≈1.99505\approx 1.99505, because the second sample becomes active.

Released implementation. The implemented column update is incremental,

u←u+(G⊤G+10−6I)−1G⊤Rv⊤∥v∥2,u \leftarrow u + \frac{(G^\top G + 10^{-6} I)^{-1} G^\top R v^\top}{\lVert v \rVert^2},

where RR is the current residual and vv is the classifier row vector of the hidden unit, as in the manuscript. It can also increase the true loss:

  • Setup: a two-head block with the other head's contribution set to zero and both head coefficients equal to one; one token, one hidden unit, a=A=1a = A = 1.
  • Data: X=[112−1−10]X = \begin{bmatrix} 1 & 1 \cr 2 & -1 \cr -1 & 0 \end{bmatrix}, all three labels in class 2, v=[1,−2]v = [1, -2] and uold=[−1,−2]⊤u_{\text{old}} = [-1, -2]^\top. Only sample 3 is active.
  • Update: unew=uold+[75(1+10−6),0]⊤≈[0.4,−2]⊤u_{\text{new}} = u_{\text{old}} + \left[\frac{7}{5(1+10^{-6})}, 0\right]^\top \approx [0.4, -2]^\top. This deactivates sample 3 and activates sample 2, and the true squared loss rises from 1212 to ≈53.4\approx 53.4.

The code resets parameters on NaNs, but it has no acceptance test for increases in the true objective. Neither example shows that the algorithm fails to converge in practice, and neither contradicts Fig. 2. They show that there is no general per-step descent guarantee. The expectation in Section 4.4 that the training loss decreases within each block therefore needs further justification, together with an empirical report of how often column updates increase the true objective and which safeguards, if any, are used.

W4. The token update stated in the manuscript implies rank-at-most-one representations, whereas the implementation uses a different update.

The closed form in Eq. (8)/(B.7) is

A0=vec(a)vec⊤(y)B0⊤(∥a∥22B0B0⊤+λA0Im)−1,A_0 = \mathrm{vec}(a) \mathrm{vec}^\top(y) B_0^\top \left( \lVert a \rVert_2^2 B_0 B_0^\top + \lambda_{A_0} I_m \right)^{-1},

an outer product. Hence rank(A0)≤1\mathrm{rank}(A_0) \le 1, and its left factor is aa. With λA0>0\lambda_{A_0} > 0, as in the paper, this is a property of the objective rather than of the solver: the loss depends on A0A_0 only through aA0aA_0, so the stationarity condition forces A0=a⊤rA_0 = a^\top r for some row vector rr. Under Algorithm 3, each head therefore satisfies Aℓ,i=a⊤qℓ,iA_{\ell,i} = a^\top q_{\ell,i}, and the block output is

Zℓ+1=a⊤[1p∑iqℓ,iσ(ZℓWℓ,i)].Z_{\ell+1} = a^\top \left[\frac{1}{p} \sum_{i} q_{\ell,i} \sigma(Z_\ell W_{\ell,i})\right].

Each head performs its own position-weighted aggregation (qℓ,iq_{\ell,i}, Wℓ,iW_{\ell,i}), but all heads share the output token factor aa. After a complete head-update sweep, the block output therefore has token-mode rank at most one for each image. As written, the token-wise projections thus act as position-weighted aggregations rather than general token mixing.

For the single-block predictor, only ξ=aA0\xi = aA_0 enters the prediction. Under regularization, aa still matters through the effective penalty λA0∥ξ∥2/∥a∥2\lambda_{A_0} \lVert \xi \rVert^2 / \lVert a \rVert^2, but the roles of the separate parameters aa and A0A_0 should be explained.

This conclusion applies to Eq. (8) and Algorithm 3 as written. It does not automatically apply to the submitted implementation, which adds a random nullspace component and updates the shared pooling weights between heads (W2). The authors should clarify which algorithm produced the reported results.

W5. There is no same-architecture BP control, so architecture and learning rule are confounded.

The baseline architectures are not fully specified. The paper gives depth, width and batch size, but not the input processing, connectivity or most training details. The reported parameter counts are consistent with flattened-input two-hidden-layer MLPs: 784-512-512-10 gives 0.67M, and 3072-1024-1024-10 gives 4.21M (with k=100k = 100, the same MLP would have about 4.30M). FIO instead uses patch tokenization, weight sharing across tokens, multiple heads and learned pooling. The statement that "performance differences primarily reflect the underlying learning and optimization mechanisms" (l.361) therefore does not follow from Table 1.

The most informative missing experiment is the identical FIO architecture (same patches, heads, pooling and squared loss) trained with BP. A greedy layer-wise BP variant with the same readouts would also help. Without these, the gains over BP on Fashion-MNIST and CIFAR-10 cannot be attributed to the learning rule.

W6. The efficiency and parameter claims are insufficiently supported.

  • Inference time.

    • A direct dense implementation of the CIFAR-10 FIO model (12 heads, e=384e = 384, m=64m = 64, two blocks) needs about 165M MACs per sample. The corresponding MLP needs about 4.2M. Table 2 nevertheless reports an 18× lower latency for FIO.
    • MAC counts do not determine single-sample GPU latency. The rank-one simplification applies to the update stated in the manuscript, but it cannot be assumed for the submitted implementation with its random rank-increasing component.
    • Please identify the implementation used for the timing results, and report FLOPs, batch size, hardware and software setting, the warm-up and synchronization protocol, and the exact computation path timed for the baselines.
  • Parameter count. Counting the stated dense structure (two blocks, e=384e = 384), I get:

    • 2,092,864 for CIFAR-10 (12 heads), which matches the reported 2.09M;
    • 2,127,424 (≈2.13M) for CIFAR-100;
    • 954,301 (≈0.95M) for MNIST/F-MNIST (6 heads), not the reported 0.47M.

    Please reconcile.

  • Memory.

    • The abstract claims lower memory overhead, but no peak-memory measurements are reported in the manuscript or the supplementary material.
    • Appendix G analyzes an explicitly constructed HiH_i. The submitted code instead uses compact matrix products and does not materialize this Kronecker matrix, so the ≈7.7 GB that an explicit HiH_i would need for CIFAR-100 does not describe an allocation made by the code.
    • The code does hold dense full-dataset tensors of shape m×n×em \times n \times e, such as block representations and head outputs, which take about 4.9 GB each in FP32 at the stated CIFAR settings.
    • Peak training memory remains unreported.
  • Training time. No wall-clock training time is reported, although efficiency is a central claim. The complexity in (G.30) absorbs the number of outer iterations τ\tau into the constant. Please report τ\tau, the wall-clock time and the peak memory for FIO and for BP on the same architecture, at matched accuracy.

Questions for authors
  1. Were the reported results produced with the released code? If so, how do they change when a held-out validation split is used for all selection and the test set only for final evaluation?
  2. Which update, the manuscript's Eq. (8)/Algorithm 3 or the implemented one, was used for the reported experiments? How do the ranks of the token matrices produced by the submitted implementation compare with those implied by Eq. (8)? Please explain and ablate the random nullspace component, and correct the aa-update formula.
  3. Does the W0W_0 update use any acceptance test, damping or line search? How often does the true objective increase after a column update in practice?
  4. What accuracy does the identical FIO architecture reach when trained with BP under the same squared loss?
  5. Please report FLOPs, peak memory and wall-clock training time for FIO and for BP on the same architecture, and describe how the latencies in Table 2 were measured.
  6. Please reconcile the MNIST parameter count (0.47M) with the stated configuration, and confirm the configuration behind Table 1.
Reviewer_WvSQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Forward Iterative Optimization (FIO), a BP-free image classification framework using alternating optimization and analytical least-squares updates. The method is extended to multilayer and multihead architectures and reports competitive results on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.

Critical strengths

The classification results are promising, particularly on CIFAR-10, and the proposed optimization procedure provides an interesting alternative to BP.

Critical weaknesses

1. Limited novelty and theoretical concerns.

  • Analytical and greedy layer-wise neural-network training are already established, notably in Nguyen et al. (2021), An Analytic Layer-wise Deep Learning Framework with Applications to Robotics, and Wu et al. (2022), Deep Layer-wise Networks Have Closed-Form Weights. Although the proposed optimization rules differ, the paper does not sufficiently clarify what constitutes a significant methodological advance beyond these established approaches.

  • More importantly, Eq. (8) implies that the learned token-mixing matrix has rank at most one. Since all heads share the same aggregation vector, this restriction also applies to the resulting multihead representations. This substantially undermines the claimed expressiveness of the token-mixing architecture.

  • Furthermore, the ReLU updates in Eq. (11) are optimal only for fixed activation masks. Since these masks can change after updating the weights, monotonic improvement of the actual nonlinear objective is not guaranteed.

2. Insufficient experimental validation and questionable comparisons.

  • The reported improvements over BP are difficult to interpret because FIO uses a different architecture. A comparison with BP training of the identical FIO architecture is essential.

  • The baseline results also raise reproducibility questions: several reported FOTON and BP accuracies exactly match configurations from the original FOTON paper that differ from those described here.

  • Moreover, despite strong efficiency claims, neither training wall-clock times nor peak training memory are reported. Inference speedups alone do not establish the efficiency of the proposed optimization algorithm.

  • Finally, some citations are inaccurate. For instance, GrAPE (Caillon et al., 2026) is incorrectly described as a method for fine-tuning BP-pretrained models.

Questions for authors
  • How do the authors reconcile the claimed expressive token-mixing architecture with the rank-one structure implied by Eq. (8)?

  • Can the authors justify convergence despite changes in ReLU activation patterns?

  • Can the authors clarify the baseline configurations and provide matched-architecture BP comparisons, including training time and memory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.