acceptodds
Withdrawn from ICLR 2027

A PHYSICAL THEORY OF BACKPROPAGATION: EXACT GRADIENTS FROM THE LEAST-ACTION PRINCIPLE

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Backpropagation (BP) computes gradients through successive forward and back- ward passes. We derive the same gradients from an extension of Hamilton’s action principle that describes dissipative systems using two coupled states. Their mean represents network activations, while their difference carries the sensitivities needed to compute gradients. The task loss supplies the output error, and the two states evolve together through interactions between neighboring computational stages. These interactions use the same transposed local derivatives as BP. At equilibrium the gradient is exactly BP; updating all stages together with unit-step Euler reaches this equilibrium within 2Lupdates for a feedforward chain of Lstages. The formu- lation also lets us change how the gradient is reached. Grouping operations into larger blocks reduces the number of update rounds. Adding inertia and choosing its settings from the system’s decay rates speeds up the tested recurrent gradient computations while preserving their equilibrium. The formulation opens the door to using tools from physics to study and design learning dynamics.

Primary area:
learning theory
Keywords:
Backpropagation, least-action principle, variational methods, continuous-time learning dynamics, exact gradient computation, neuromorphic computing.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NW1rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper embeds an LL-layer feedforward network in a dissipative-mechanics framework on a doubled phase space: activations and sensitivities are encoded as the mean m=12(x+z)m=\tfrac{1}{2}(x+z) and the difference s=x−zs=x-z (the stress) of a conjugate pair of global states xx and zz, and a single global Lagrangian is constructed by way of the Bateman–Galley formalism for non-conservative systems. At the unique stable equilibrium the exact BP gradient is given by the stress ss, and under unit-step Euler discretization BP is recovered exactly within 2L2L steps; the algebraic mechanism is the strict lower block-triangularity of the global weight matrix WW, which makes Mˉ=W⊤Dˉ\bar{M}=W^{\top}\bar{D} satisfy MˉL=0\bar{M}^{L}=0. The experiments compare a 9-layer VGG on CIFAR-10 against PyTorch autodiff, with test accuracy on par with BP ($\sim$93%); among other readings, the relative gradient error and 1−cos⁡1-\cos are reported; a block-grouping comparison is run on a reduced CNN (15/7/4 stages); and an inertia comparison on a linear recurrent model.

Critical strengths
  1. The paper writes the BP gradient as the stress at the unique stable equilibrium, and recovers standard BP in 2L2L steps under unit-step Euler (pp. 1–2; Theorem 4.2 and Algorithm 2, pp. 6 and 16). WL=0W^{L}=0 (Lemma 3.1, p. 3) is the algebraic source of this finite-step conclusion; the authors report two practical consequences (p. 1, pp. 8–9): the number of convergence rounds for block grouping is exactly twice the number of stages (30/14/8), and the inertia result comes from a linear model with spectral radius close to 1. This writes the credit assignment of BP as an equilibrium expressed through the mean and difference of the conjugate states and gives an explicit 2L2L-step recovery path, so that the physical formulation and the computational content of BP can be compared directly.
Critical weaknesses
  1. The paper does not connect the "physical formulation" to a usable capability: block grouping and inertia come only with simulation-level readings (pp. 8–9), the claimed analog/neuromorphic/hardware use case exists only as a design (Appendix G), and p. 9 states "hardware runtime has not been measured"; the Appendix G variant likewise reports no gradient or accuracy readings — the authors summarize it with "tested with identical accuracy" (p. 4) and "matches the performance of algorithm 1" (p. 18), but give no readings beyond the equations and Algorithm 3. The paper thus provides no checkable account of what this physicalized formulation of BP adds over existing physical descriptions of BP.
  2. The paper is hard to read and lacks both a concrete illustration of the concept and a positioning against related algorithms: the derivation proceeds through dense formulas, Figures 1–8 are all curves or heatmaps, and no figure maps xx and zz onto network units; beyond the comparison table in Section 6, the paper does not state how the formulation differs in computational content from existing algorithms such as PC/BP/EP.
Questions for authors
  • The structure is already defined algebraically in §3.1 and Eq. (2); can the authors add a figure that maps the derived formulas onto the layer units, and state how the formulation differs from PC/BP/EP architectures (cf. Whittington et al., 2019)? And what does the formulation provide beyond what other algorithms already offer — where does its value lie (biological plausibility, training or inference cost, deployable hardware, or something else)?
  • In §3.4, L\mathcal{L} in Eq. (5) depends only on xx and zz; taking its variation as written in Eq. (6) yields the equilibrium conditions of Eq. (10), while Eq. (7) carries −∂R/∂x˙-\partial R/\partial\dot{x}, corresponding to the modified Euler–Lagrange equation, Eq. (26), in Appendix A. How does the main text get from Eq. (6) to Eq. (7)? Where does the Rayleigh term sit within the variational principle?
  • Read literally, Eqs. (44)(45) and Algorithm 3 take the loss gradient at xLx_L and zLz_L rather than at the midpoint mLm_L; is the equilibrium so obtained different from BP? If "matches" refers to test accuracy, what is the per-layer gradient error of this variant?
Reviewer_mZunOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives standard backpropagation gradients from Hamilton’s least‑action principle for dissipative systems with two coupled states. The mean of two states represents network activations while their difference encodes gradient‑carrying sensitivities. Under unit‑step Euler discretization, exact BP gradients are reached within 2L updates for an L‑layer feedforward network. The formulation enables block grouping and inertia addition to speed gradient computation, opening opportunities to apply physics tools for learning‑dynamics design.

Critical strengths
  1. It gives a novel variational physics‑rooted derivation of exact backpropagation based on the Bateman‑Galley least‑action principle. Theoretical proofs for finite‑time convergence to BP gradients are laid out.
  2. It explores two interesting extensions: block grouping to cut communication rounds and adding inertia to accelerate relaxation for gradient solving.
Critical weaknesses
  1. Empirical validation is severely insufficient. Experiments are limited to one small CNN on CIFAR‑10 and toy synthetic recurrent examples. Missing tests on larger networks, different datasets, or real neuromorphic hardware.
  2. The practical significance is questionable. The method reproduces standard BP gradients and still needs exact weight transpose. It is not clear what tangible benefit this brings over existing backpropagation in real deep‑learning practice. The paper mostly offers a theoretical reinterpretation rather than a practically superior algorithm.
  3. Some assumptions like doubled phase‑space construction are physically motivated but hard to connect to existing real neural hardware without further evidence.
Questions for authors
  1. Are there any hidden constraints or edge cases where your mathematical derivation would break? Could you point out concrete failure scenarios of your dynamical system?
  2. Your current experiments are quite small. Why do you not test on larger networks / more datasets to show practical value?
  3. Since your method produces exactly the same gradients as standard BP and still requires weight‑transpose, what real‑world scenario would make this physical formulation preferable to regular autograd BP?
  4. Can you elaborate how this work could move beyond theoretical reinterpretation to deliver concrete gains for neuromorphic hardware?
Reviewer_ypPtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The standard feedforward (primal) and reverse sweep (adjoint) computational processes of an MLP are replaced with a Physics-based formulation, involving a Hamiltonian. It is shown that, in the appropriate limit, the equations of motion of the new formulation reduce to the standard equations, and moreover the equilibrium state gives the same results when that limit is not taken.

Critical strengths

This is an interesting idea. Connecting the dynamics of neural systems to that of some physical system has led to a number of important insights and methods, like momentum in stochastic gradient descent, neural ODEs, Hopfield networks, Boltzmann Machines, etc.

Critical weaknesses

This particular connection to a physical theory does not seem to have led to anything interesting, at least not yet. It gives a set of new algorithms for calculating the forward (primal) and reverse (adjoint) phases of MLP backpropagation, however none of these appear to be faster than the presently popular methods. Maybe there are some clever applications, but I don't see them.

The system appears formally equivalent to the dynamics of the Pineda (1987) paper, with an added Hamiltonian derivation, along with the restriction to an acyclic W. But that work allows W to be cyclic, and the network to therefore require relaxation for convergence. Which leads to a question (see below).

Questions for authors
  1. Have I missed something of significance?

  2. Does the math go through here if you do not restrict to an acyclic W? If you turn the crank in that case, do the adjoint equations of Pineda (1987) fall out? (Suggestion: in that recurrent network setting, there might be more traction to be had on using this formulation to find new methods, perhaps to accelerate convergence.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.