- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes LIFTY, a backpropagation-free neural network training framework based on Fenchel–Young lifting. The method treats intermediate activations as optimization variables and introduces geometry-aware layerwise updates to optimize network weights without conventional backpropagation.
Critical strengths
-
The paper provides a principled formulation of backpropagation-free training using Fenchel–Young gaps, offering a unified perspective on layerwise optimization.
-
The proposed method supports various activation functions and convex weight regularizers, providing flexibility in network training.
-
LIFTY achieves competitive performance against existing lifted training methods while demonstrating improvements in computational efficiency and memory usage in the evaluated settings.
Critical weaknesses
-
Limited methodological novelty. The proposed geometry-aware activation update (Eq. 15) appears mathematically equivalent to a standard mirror-descent step applied to the same Fenchel–Young lifted objective under forward-pass initialization, with an appropriately chosen step size. Therefore, it remains unclear what fundamentally new optimization mechanism LIFTY introduces beyond established mirror-descent and Bregman majorization techniques. The authors should clarify whether the main contribution lies in the update rule itself or in the convergence analysis of the resulting warm-start block-coordinate scheme.
-
Limited applicability of the convergence guarantees. Theorems 1–3 critically depend on Assumption 2 (Forward Pass Decrease), which requires a summable error sequence to control the discrepancy between the lifted and original forward-pass objectives. However, Remark 7 (Appendix B.1) only establishes this condition trivially for single-layer networks and informally discusses its validity for multilayer linear networks under additional restrictions, without providing explicit sufficient conditions or a formal proof. Its validity for general multilayer nonlinear networks remains unestablished. Since such networks are the primary target of LIFTY, this limits the applicability of the main convergence guarantees.
-
Limited experimental validation. The experiments are restricted to relatively simple datasets and small-scale MLP architectures, which are insufficient to demonstrate the effectiveness and scalability of LIFTY for modern deep neural networks. In particular, the evaluation lacks experiments on more challenging datasets and architectures, such as ImageNet-1K and CNNs or Transformers. Additional large-scale experiments would strengthen the empirical evidence for the proposed backpropagation-free training framework.
Questions for authors
See weaknesses