Constrained Neural Network Training via Landing Dynamics

Equality-constrained optimization of neural networks is a common situation, where the network output is required to satisfy conserved quantities, boundary conditions of an equation, or other identities. Existing approaches, however, either lack sound theoretical properties or do not generalize. We propose Landing with Multiplier Tracking (LandTrack), a method that opens another route, in which the constraint acts on the training dynamics instead. Specifically, each update is split into two orthogonal components, a tangential one that decreases the objective along the constraint set and a normal one that pulls the iterate back toward feasibility. Training with this update drives the network to constrained optima, in the sense that the fixed points of the update are the Karush-Kuhn-Tucker (KKT) points of the sampled constrained problem. Building on the theoretical foundation of the classical landing method, we obtain the two components by solving a linear system in a Lagrange multiplier, whose every inner iteration costs a Jacobian-vector and a vector-Jacobian product (JVP and VJP). We further reduce this overhead with a Richardson iteration that tracks the slowly drifting multiplier across training steps at one correction per step, whose fixed point remains the exact projection. Furthermore, we stabilize the recursion with a warm-started spectral estimate, a trust-region cap, and a damped initialization. In experiments spanning the prediction of physical system dynamics and partial differential equations (PDEs), with constraints ranging from conserved quantities to boundary conditions, LandTrack improves the entire accuracy-conservation trade-off curve over other baselines.