- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The work concerns training neural networks under equality constraints on the network output (conserved quantities, PDE boundary conditions). It adapts the landing method from orthogonal manifolds to general constraints. Each update decomposes into a tangential step that decreases the objective and a normal step that contracts the constraint residual. The tangential step needs a projection, which costs a Krylov solve per step. The authors replace the solve with a multiplier estimate tracked across steps by one Richardson correction per step, and add three safeguards: a spectral refresh, a trust-region cap, and a damped initialization. Experiments on 15-body, 24-body, KdV and reaction-diffusion problems shows the method to be more favorable than a weighted penalty method and an augmented Lagrangian (AL) method.
Critical strengths
- The tangential/normal decomposition is a clear idea, it works with any base optimizer, and it needs only JVPs and VJPs. This helps me evaluate the applicability of the method.
- The evaluation covers several constraint types (quadratic, linear, nonlinear, boundary). This helps me the applicability of the method to different problems.
Critical weaknesses
- Writing. This makes the paper very hard to understand. Now that AI writing is ubiquitous, it is important for paper to be still be well-written so that we are not swamped in slops:
- The description of the landing method on L167-180 is poorly written and lack coherence with the rest of the paper, which makes it impossible for me to learn how the landing method's update would compare with other constrained optim method.
- Section 3.2 is also poorly written. It reads like a collection of separate sentences instead of coherent story. This is an important paragraph for a reader to understand the method, so it reflects poorly on the authors to not put time on writing this properly.
- Figure 1 is hard to understand without an annotation of what , , are. Ofc it's fine when we go to the notation later, but Figure 1 is mentioned first in the intro when none of these quantities has been introduced.
- The guarantee stops at "fixed points are KKT points". There is no result on feasibility during training. This is important for constraint optimization problems.
- No condition guarantees that the normal step contracts . The Richardson condition is only for the projection error for the projeciton step.
- The set of baselines is quite small. The intro and related work sections cite adaptive-weight methods (Wang et al.), primal-dual and PID Lagrangian methods (Stooke et al., Cotter et al., Gallego-Posada et al.), and projection or hard-constraint methods (DC3, Négiar et al., Hu et al.), but only a fixed- penalty and one AL are run.
Questions for authors
- Under what conditions does the iterate stay within in training?
- Is there an assumption on (PL-type) behind the contraction of ? If so, please state it. Is it checked in the experiments?
- How sensitive are results to , and how was it set?
- An ablation is missing to understand the method better: landing with the exact CG projection every step (what tracking approximates) so that we can understand the tracking error.. What happens with the exact projection (CG to convergence) at every step, on both accuracy and constraint error?
- The Pareto framing in the experiments is not justified. A Pareto curve needs an inherent trade-off between the two objectives, and here there is none. Is there a reason to use Pareto there?