acceptodds
Under review as a conference paper at ICLR 2027

Principled Backpropagation-Free Training via Layerwise Fenchel-Young Lifting

Abstract

Training neural networks is typically performed using backpropagation, which tightly couples layers and requires storing the full computational graph. In this work, we study lifted training approaches, where intermediate activations are treated as optimization variables, and show that their underlying structure admits an exact characterization. Specifically, we prove that the compositional structure of neural networks is equivalent to the vanishing of Fenchel-Young gaps induced by Legendre functions, providing a principled measure of deviation from the forward pass. This perspective reveals that lifted activations act as regularized predictors, inducing an implicit regularization effect that improves generalization in practice. Building on this foundation, we propose Lifty, a memory-efficient backpropagation-free training algorithm with closed-form updates supporting a broad class of monotone activations and convex regularizers.

Primary area:
optimization
Keywords:
Lifted neural networks, Backpropagation-free learning, Fenchel-Young losses, Proximal optimization, Block coordinate descent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hfv2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LIFTY, a backpropagation-free neural network training framework based on Fenchel–Young lifting. The method treats intermediate activations as optimization variables and introduces geometry-aware layerwise updates to optimize network weights without conventional backpropagation.

Critical strengths
  1. The paper provides a principled formulation of backpropagation-free training using Fenchel–Young gaps, offering a unified perspective on layerwise optimization.

  2. The proposed method supports various activation functions and convex weight regularizers, providing flexibility in network training.

  3. LIFTY achieves competitive performance against existing lifted training methods while demonstrating improvements in computational efficiency and memory usage in the evaluated settings.

Critical weaknesses
  1. Limited methodological novelty. The proposed geometry-aware activation update (Eq. 15) appears mathematically equivalent to a standard mirror-descent step applied to the same Fenchel–Young lifted objective under forward-pass initialization, with an appropriately chosen step size. Therefore, it remains unclear what fundamentally new optimization mechanism LIFTY introduces beyond established mirror-descent and Bregman majorization techniques. The authors should clarify whether the main contribution lies in the update rule itself or in the convergence analysis of the resulting warm-start block-coordinate scheme.

  2. Limited applicability of the convergence guarantees. Theorems 1–3 critically depend on Assumption 2 (Forward Pass Decrease), which requires a summable error sequence to control the discrepancy between the lifted and original forward-pass objectives. However, Remark 7 (Appendix B.1) only establishes this condition trivially for single-layer networks and informally discusses its validity for multilayer linear networks under additional restrictions, without providing explicit sufficient conditions or a formal proof. Its validity for general multilayer nonlinear networks remains unestablished. Since such networks are the primary target of LIFTY, this limits the applicability of the main convergence guarantees.

  3. Limited experimental validation. The experiments are restricted to relatively simple datasets and small-scale MLP architectures, which are insufficient to demonstrate the effectiveness and scalability of LIFTY for modern deep neural networks. In particular, the evaluation lacks experiments on more challenging datasets and architectures, such as ImageNet-1K and CNNs or Transformers. Additional large-scale experiments would strengthen the empirical evidence for the proposed backpropagation-free training framework.

Questions for authors

See weaknesses

Reviewer_LbRwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LIFTY, a backpropagation-free training framework based on layerwise Fenchel–Young lifting. The main idea is to introduce hidden activations as optimization variables and use Fenchel–Young gaps to characterize the consistency between adjacent layers, enabling layerwise updates without standard end-to-end backpropagation. The paper provides convergence analysis and studies the resulting optimization and memory properties. Experiments compare LIFTY with other lifting approaches and standard backpropagation on several benchmark datasets, together with analyses of architecture sensitivity, convergence behavior, and compute/model scaling.

Critical strengths
  1. The paper provides a coherent mathematical motivation for backpropagation-free layerwise training. The Fenchel–Young formulation gives a principled connection between the lifted variables and the original layerwise computation, rather than introducing local objectives purely heuristically.
  2. The paper studies optimization and memory behavior in addition to predictive performance. The architecture and scaling experiments provide some evidence that LIFTY can train stably across different network configurations and may offer favorable memory characteristics as depth increases.
Critical weaknesses
  1. The empirical evaluation is too limited relative to the scope of the paper’s claims. Most experiments use small or classical datasets such as MNIST, Fashion-MNIST, Sonar, and Covertype, while a backpropagation-free method would be most compelling on modern deep architectures where backpropagation memory is actually a significant bottleneck. The baseline coverage is also relatively limited for such a broad positioning.
  2. The current experiments do not establish a convincing accuracy–memory–compute trade-off. Standard backpropagation with Adam remains substantially stronger on some benchmarks (e.g., 93.9% vs. 78.3% on Covertype), while the scaling experiments are conducted in regimes where accuracy quickly saturates. Consequently, it is difficult to determine whether the observed scaling behavior reflects genuine efficiency or simply task saturation, and epoch-wise convergence curves alone do not account for differences in computational cost.
Questions for authors
  1. What is the intended practical regime in which LIFTY is preferable to standard backpropagation? Given the substantial accuracy gap on some datasets, it would be important to quantify the joint trade-off among accuracy, peak memory, wall-clock time, and computational cost.
  2. Can the authors validate the claimed scalability on substantially deeper modern architectures and more challenging datasets? The current small-scale experiments make it difficult to assess whether the advantages persist in settings where backpropagation memory is genuinely limiting.
  3. How should the scaling results be interpreted when validation accuracy already saturates over much of the evaluated parameter/compute range? In particular, could the weak scaling of the optimal LIFTY model size with compute reflect limited benefit from additional model capacity rather than improved efficiency?
  4. Could the authors provide more direct empirical validation of the theoretical quantities underlying LIFTY? For example, layerwise Fenchel–Young gaps or stationarity measures across training would be more informative than the current \epsilon_k plot, which approaches zero almost immediately.
Reviewer_zZhtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LIFTY (Lifted training via Fenchel-Young lifting), a backpropagation-free training method based on layer-wise Fenchel–Young lifting. The method combines geometry-aware activation updates, proximal weight updates, and forward-pass warm starts. Then, the paper provides some convergence guarantees connecting the lifted formulation to the original training objective. Finally, some numerical results are reported to validate the theories.

Critical strengths

Impressive motivation: The geometry-aware activation updates are well motivated and admit explicit expressions, requiring only one update per block at each iteration.

Numerical experiment: Comparisons with FY-Adam and FY-SGD help isolate the benefit of the proposed updates. LIFTY also achieves better accuracy than the evaluated lifted baselines and reduces memory usage in deeper MLPs.

Critical weaknesses

The theoretical arguments contain gaps. Proposition 1 overlooks the supervised loss contribution to the gradient with respect to the final hidden activation. Moreover, Theorem 1 infers output-layer stationarity from vanishing weight updates without requiring step sizes to be bounded away from zero.

The novelty of the underlying characterization is overstated. Gu et al. (2020) already establish the equivalence between activation consistency and a vanishing Fenchel–Young gap, while Wang and Benning (2023) develop a closely related Bregman formulation. The paper is supposed to clearly and explicitly claim the scope of its contributions.

The empirical evidence does not fully support the broader claims. Accuracy experiments primarily involve shallow MLPs, and backpropagation performs better on three of the four benchmarks, particularly Covertype. It's unclear if the method still works in larger-scaled systems.

Questions for authors

How should Proposition 1 account for the generally nonzero term WL⊤(∇Φ∗(WLXL)−y)W_L^\top(\nabla\Phi^*(W_LX_L)-y)? Does Theorem 1 require an additional positive lower bound on the weight step sizes?

Can the authors provide some numerical results on deeper networks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.