acceptodds
Under review as a conference paper at ICLR 2027

Constrained Neural Network Training via Landing Dynamics

Abstract

Equality-constrained optimization of neural networks is a common situation, where the network output is required to satisfy conserved quantities, boundary conditions of an equation, or other identities. Existing approaches, however, either lack sound theoretical properties or do not generalize. We propose Landing with Multiplier Tracking (LandTrack), a method that opens another route, in which the constraint acts on the training dynamics instead. Specifically, each update is split into two orthogonal components, a tangential one that decreases the objective along the constraint set and a normal one that pulls the iterate back toward feasibility. Training with this update drives the network to constrained optima, in the sense that the fixed points of the update are the Karush-Kuhn-Tucker (KKT) points of the sampled constrained problem. Building on the theoretical foundation of the classical landing method, we obtain the two components by solving a linear system in a Lagrange multiplier, whose every inner iteration costs a Jacobian-vector and a vector-Jacobian product (JVP and VJP). We further reduce this overhead with a Richardson iteration that tracks the slowly drifting multiplier across training steps at one correction per step, whose fixed point remains the exact projection. Furthermore, we stabilize the recursion with a warm-started spectral estimate, a trust-region cap, and a damped initialization. In experiments spanning the prediction of physical system dynamics and partial differential equations (PDEs), with constraints ranging from conserved quantities to boundary conditions, LandTrack improves the entire accuracy-conservation trade-off curve over other baselines.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
optimization, constraint, conservation law, physical system, PDE

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KtPrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The work concerns training neural networks under equality constraints on the network output (conserved quantities, PDE boundary conditions). It adapts the landing method from orthogonal manifolds to general constraints. Each update decomposes into a tangential step that decreases the objective and a normal step that contracts the constraint residual. The tangential step needs a projection, which costs a Krylov solve per step. The authors replace the solve with a multiplier estimate tracked across steps by one Richardson correction per step, and add three safeguards: a spectral refresh, a trust-region cap, and a damped initialization. Experiments on 15-body, 24-body, KdV and reaction-diffusion problems shows the method to be more favorable than a weighted penalty method and an augmented Lagrangian (AL) method.

Critical strengths
  1. The tangential/normal decomposition is a clear idea, it works with any base optimizer, and it needs only JVPs and VJPs. This helps me evaluate the applicability of the method.
  2. The evaluation covers several constraint types (quadratic, linear, nonlinear, boundary). This helps me the applicability of the method to different problems.
Critical weaknesses
  1. Writing. This makes the paper very hard to understand. Now that AI writing is ubiquitous, it is important for paper to be still be well-written so that we are not swamped in slops:
    • The description of the landing method on L167-180 is poorly written and lack coherence with the rest of the paper, which makes it impossible for me to learn how the landing method's update would compare with other constrained optim method.
    • Section 3.2 is also poorly written. It reads like a collection of separate sentences instead of coherent story. This is an important paragraph for a reader to understand the method, so it reflects poorly on the authors to not put time on writing this properly.
    • Figure 1 is hard to understand without an annotation of what Gh,kG_{h, k}, dkd_k, AkA_k are. Ofc it's fine when we go to the notation later, but Figure 1 is mentioned first in the intro when none of these quantities has been introduced.
  2. The guarantee stops at "fixed points are KKT points". There is no result on feasibility during training. This is important for constraint optimization problems.
    • No condition guarantees that the normal step contracts CkC_k. The Richardson condition is only for the projection error for the projeciton step.
  3. The set of baselines is quite small. The intro and related work sections cite adaptive-weight methods (Wang et al.), primal-dual and PID Lagrangian methods (Stooke et al., Cotter et al., Gallego-Posada et al.), and projection or hard-constraint methods (DC3, Négiar et al., Hu et al.), but only a fixed-β\beta penalty and one AL are run.
Questions for authors
  1. Under what conditions does the iterate stay within ∥Ck∥≤ε\|C_k\| \le \varepsilon in training?
  2. Is there an assumption on σmin⁡(Ak)\sigma_{\min}(A_k) (PL-type) behind the contraction of CkC_k? If so, please state it. Is it checked in the experiments?
  3. How sensitive are results to α\alpha, and how was it set?
  4. An ablation is missing to understand the method better: landing with the exact CG projection every step (what tracking approximates) so that we can understand the tracking error.. What happens with the exact projection (CG to convergence) at every step, on both accuracy and constraint error?
  5. The Pareto framing in the experiments is not justified. A Pareto curve needs an inherent trade-off between the two objectives, and here there is none. Is there a reason to use Pareto there?
Reviewer_QgucOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LandTrack for equality-constrained neural-network training. It combines an approximate tangent-space update with a feasibility correction and reduces projection cost by tracking a multiplier across training steps. Experiments on particle dynamics and PDEs report improvements over penalty and augmented Lagrangian methods.

Critical strengths
  • Reusing the projection multiplier addresses a meaningful computational bottleneck without changing the network architecture or inference procedure.

  • The experiments cover several constraint types and include autoregressive rollouts. Some configurations improve prediction accuracy and constraint satisfaction simultaneously, providing encouraging evidence of practical value.

Critical weaknesses
  • The geometric decomposition closely overlaps with ODCGM [1], which already handles general nonlinear equality constraints through projected gradients and orthogonal feasibility corrections. This missing comparison makes the novelty appear overstated; the contribution should be established primarily through the tracking mechanism.

  • That mechanism assumes a slowly changing multiplier, but minibatch resampling changes the sample-specific constraints associated with its coordinates. Small parameter updates do not establish that the previous multiplier remains appropriate, and the paper provides neither a clear correspondence rule nor tracking-error measurements.

  • Proposition 1 analyzes a fixed linear system, leaving the changing minibatches, correction scaling, and coupled training dynamics unaddressed. Moreover, projecting an Adam update does not generally preserve the KKT stationarity condition. The broader convergence claims therefore exceed the supplied analysis.

  • The experiments do not isolate tracking from the feasibility correction and added penalty. Comparisons with accurate projection, limited-budget warm-started CG, and penalty-free LandTrack are missing. Without multi-seed results and complete baseline tuning details, the robustness of smaller improvements also remains uncertain.

ref: [1] Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold, COLT 2023

Questions for authors
  • How does the proposed method differ from ODCGM beyond the approximate projection mechanism? Could the authors clarify which contributions concern the underlying optimization framework and which concern its efficient implementation?

  • Could the authors measure the tracking error against accurately solved projections during training, including after minibatch changes? Reporting projection residuals and the frequency of correction scaling would help establish whether the assumed tracking behavior occurs in practice.

  • Which convergence or stationarity claims hold for Algorithm 1 with Adam and the stabilization mechanisms enabled? If these claims require additional assumptions, could they be stated explicitly and reflected in the abstract and conclusion?

Given these unresolved issues, my initial score is 1, but I'm open to change my rating if my concerns are resolved.

Reviewer_5ojvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LandTrack, a method for equality-constrained neural-network training that combines a feasibility correction with an approximate tangent projection. It amortizes projection computation through one multiplier-tracking update per training step.

Critical strengths
  • The authors present a correct contraction analysis for the explicitly frozen projection system.
  • Evaluation across different constraints types is useful to measure impact.
Critical weaknesses
  • The novelty needs to have clearer positioning. Schechtman et al. COLT 2023 already developed the general orthogonal objective decomposition. So the emphasis on multiplier tracking needs to be more prominent.
  • There seem to be no repeated-seed uncertainty estimates or comparisons against accurate per-step projection and a feasibility-only correction.
Questions for authors
  • How are multiplier coordinates handled across resampled batches, and what are the observed tracking residuals and cap-activation rates?
  • Which convergence or KKT claims apply to the implemented algorithm with Adam and clipping?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.